Struct Lexer

pub struct Lexer<'a> { /* private fields */ }

The Lexer struct - the core of the library - borrows &str and can incrementally produce Tokens.

The encoding of the &str is assumed to be utf-8. Other sources should be re-encoded into utf-8 prior to ingesting into the Lexer.

The Lexer may be configured with additional Features to allow for lexing tokens in ways which diverge from the CSS specification (such as tokenizing comments using //). With no additional features this lexer is fully spec compliant.

Tokens are untyped (there are no super-classes like Ident); but they have a Kind which can be used to determine their type. Tokens do not store the underlying character data, nor do they store their offsets. They just provide “facts” about the underlying data. In order to re-build a string, each Token will need to be wrapped in a Cursor and consult the original &str to get the character data. This design allows Tokens live in the stack, avoiding heap allocation as they are always size_of 8. Likewise Cursors are always a size_of 12.

Limitations

The Lexer has limitations around document sizes and token sizes, in order to keep Token, SourceOffset and Cursor small.

• Documents are limited to ~4gb in size. SourceOffset is a u32 so cannot represent larger offsets. Attempting to lex larger documents is considrered undefined behaviour.

• Tokens are limited to ~4gb in length. A Token’s is a u32 so cannot represent larger lengths. If the lexer encounters a token with larger length this is considered undefined behaviour.

• Number Tokens are limited to 16,777,216 characters in length. For example encountering a number with 17MM 0s is considered undefined behaviour. This is not the same as the number value, which is an f32. (Please note that the CSS spec dictates numbers are f32, CSS does not have larger numbers).

• Dimension Tokens are limited to 4,096 numeric characters in length and 4,096 ident characters in length. For example encountering a dimension with 4,097 0 is considered undefined behaviour.

General usage

A parser can be implemented on top of the Lexer by instantiating a Lexer with Lexer::new() or Lexer::new_with_features() if you wish to opt-into non-spec-compliant features. The Lexer needs to be given a &str which it will reference to produce Tokens.

Repeatedly calling Lexer::advance() will move the Lexer’s internal position one Token forward, and return the newly lexed Token, once the end of &str is reached Lexer::advance() will repeatedly return Token::EOF.

Example

use css_lexer::*;
let mut lexer = Lexer::new("width: 1px");
assert_eq!(lexer.offset(), 0);
{
    let token = lexer.advance();
    assert_eq!(token, Kind::Ident);
    let cursor = token.with_cursor(SourceOffset(0));
    assert_eq!(cursor.str_slice(lexer.source()), "width");
}
{
    let token = lexer.advance();
    assert_eq!(token, Kind::Colon);
    assert_eq!(token, ':');
}
{
    let token = lexer.advance();
    assert_eq!(token, Kind::Whitespace);
}
{
    let token = lexer.advance();
    assert_eq!(token, Kind::Dimension);
    assert_eq!(token.dimension_unit(), DimensionUnit::Px);
}

Implementations

impl<'a> Lexer<'a>

pub fn new(source: &'a str) -> Self

pub fn new_with_features(source: &'a str, features: Feature) -> Self

pub fn source(&self) -> &'a str

pub fn at_end(&self) -> bool

Is the lexer at the last token

pub const fn offset(&self) -> SourceOffset

Current position in file

pub fn checkpoint(&self) -> Cursor

pub fn rewind(&mut self, cursor: Cursor)

Rewinds the lexer back to the given checkpoint

pub fn hop(&mut self, cursor: Cursor)

Advances the lexer to the end of the given token

pub fn advance(&mut self) -> Token

Moves the lexer one token forward, returning that token

Trait Implementations

impl<'a> Clone for Lexer<'a>

fn clone(&self) -> Lexer<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Lexer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Default for Lexer<'a>

fn default() -> Lexer<'a>

Returns the “default value” for a type.

impl<'a> PartialEq for Lexer<'a>

fn eq(&self, other: &Lexer<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> StructuralPartialEq for Lexer<'a>