Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

Logos-based lexer that converts source text into tokens.

Implementations

impl<'a> LogosLexer<'a>

pub fn new(source: &'a str) -> Self

Create a new lexer for the given source text with a fresh interner.

Uses the default file ID. For multi-file compilation, use with_file_id.

pub fn with_interner(source: &'a str, interner: ThreadedRodeo) -> Self

Create a new lexer with an existing interner.

pub fn with_file_id(source: &'a str, file_id: FileId) -> Self

Create a new lexer with a specific file ID.

pub fn with_interner_and_file_id( source: &'a str, interner: ThreadedRodeo, file_id: FileId, ) -> Self

Create a new lexer with both an existing interner and a specific file ID.

pub fn tokenize(self) -> CompileResult<(Vec<Token>, ThreadedRodeo)>

Tokenize the entire source, returning all tokens and the interner.