Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

Logos-based lexer that converts source text into tokens.

Implementations

impl<'a> LogosLexer<'a>

pub fn new(source: &'a str) -> LogosLexer<'a>

Create a new lexer for the given source text with a fresh interner.

Uses the default file ID. For multi-file compilation, use with_file_id.

pub fn with_interner(source: &'a str, interner: ThreadedRodeo) -> LogosLexer<'a>

Create a new lexer with an existing interner.

pub fn with_file_id(source: &'a str, file_id: FileId) -> LogosLexer<'a>

Create a new lexer with a specific file ID.

pub fn with_interner_and_file_id( source: &'a str, interner: ThreadedRodeo, file_id: FileId, ) -> LogosLexer<'a>

Create a new lexer with both an existing interner and a specific file ID.

pub fn tokenize( self, ) -> Result<(Vec<Token>, ThreadedRodeo), DiagnosticWrapper<ErrorKind>>

Tokenize the entire source, returning all tokens and the interner.