Enum Kind

pub enum Kind {
    Eof = 0,
    Whitespace = 1,
    Comment = 2,
    CdcOrCdo = 3,
    Number = 4,
    Dimension = 5,
    BadString = 6,
    BadUrl = 7,
    Ident = 8,
    Function = 9,
    AtKeyword = 10,
    Hash = 11,
    String = 12,
    Url = 13,
    Delim = 16,
    Colon = 17,
    Semicolon = 18,
    Comma = 19,
    LeftSquare = 20,
    RightSquare = 21,
    LeftParen = 22,
    RightParen = 23,
    LeftCurly = 24,
    RightCurly = 25,
}

Kind represents the token “Type”, categorised mostly by the token types within the CSS Syntax spec.

Maintaining parity with the spec makes it easier to reason about logica round the parser, despite it being possible to group a bunch of these tokens into a single “delimiter” token.

Importantly, Kind is represented as u8 and must only use the 5 low bits, because the upper 3 bits get used to house details about each kind, that a token would be interested in learning about.

Variants

Eof = 0

Represents the <eof-token> defined in CSS. While CSS stipulates that this token is never produced by a tokenizer, this Lexer will produce <eof-token>s if the underlying source has been fully consumed.

Whitespace = 1

Represents the <whitespace-token> defined in CSS.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<whitespace-token>
 │├─╭─ <whitespace> ─╮─┤│
    ╰────────────────╯

While CSS stipulates that this token represents collapsed whitespace, it is possible for Lexer to produce multiple consecutive Kind::Whitespace tokens if the Feature::SeparateWhitespace runtime feature is enabled. In this case, <whitespace-token> becomes:

<whitespace-token>
 │├──╮─╭─ " " ───────╮─╭──┤│
     │ ╰─────────────╯ │
     ├─╭─ "\t" ──────╮─┤
     │ ╰─────────────╯ │
     ╰─╭─ <newline> ─╮─╯
       ╰─────────────╯

Comment = 2

Represents the <comment> defined in CSS. While CSS stipulates comment tokens are not produced during tokenization, they are for this Lexer as they’re needed in order to preserve them.

<comment>
           ╭──────────────────────────────────────────╮
 │├─ "/*" ─╯-╭─ (anything but "*" followed by "/") ─╮─╰─ "*/" ─┤│
             ╰──────────────────────────────────────╯

It is possible for Lexer to produce Kind::Whitespace tokens that begin // if the Feature::SingleLineComments runtime feature is enabled. In this mode, <comment> becomes:

<comment>
              ╭──────────────────────────────────────────╮
 │├──╮─ "/*" ─╯-╭─ (anything but "*" followed by "/") ─╮─╰─ "*/" ─╭─┤│
     │          ╰──────────────────────────────────────╯          │
     │              ╭───────────────────────────╮                 │
     ╰─ "//" ───────╯-╭─ (anything but "\n") ─╮─╰─ "\n" ──────────╯
                      ╰───────────────────────╯

CdcOrCdo = 3

Represents both the <cdc-token> and <cdo-token>s defined in CSS. While CSS separates these tokens, they’re only useful representations at the top-level stylesheet, anywhere else they represent a parse error, and it’s a little pointless to define two tokens types for what amounts to a parse error.

<cdo-token>
 │├─ "<!--" ─┤│

<cdc-token>
 │├─ "-->" ─┤│

<cdc-or-cdo-token> (Not part of the CSS specification)
 │├──╮─ <cdo-token> ─╭──┤│
     ╰─ <crc-token> ─╯

Number = 4

Represents the <number-token>.

<number-token>
    ╭─ "+" ─╮
 │├─├───────┤───╭─ [digit] ─╮─ "." ─╭─ [digit] ─╮──╭───╮──────────────────────────────────╭──┤│
    ╰─ "-" ─╯ │ ╰───────────╯       ╰───────────╯  │   │         ╭─ "+" ─╮                │
              ├───────── ╭─ [digit] ─╮─────────────┤   ├─ "e" ─╭─├───────┤──╭─ [digit] ─╮─╯
              │          ╰───────────╯             │   ╰─ "E" ─╯ ╰─ "-" ─╯  ╰───────────╯
              ╰──── "." ─╭─ [digit] ─╮─────────────╯
                         ╰───────────╯

Dimension = 5

Represents the <dimension-token>.

Here we deviate from the spec slightly, which has both <dimension-token> and <percentage-token>. <percentage-token> represents a dimension with a % symbol, but having this as a separate token results in more work in the parser for little gain in the Lexer. So instead this lexer does not have a <percentage-token> and instead folds the grammar for it inside of <dimension-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<ident-token>
    ╭───────────────── "--" ─────────────────────╮  ╭───────────────────────────────────────────╮
 │├─╯─╮───────╭─╮─ [a-z, A-Z, "_", non-ASCII] ─╭─╰──╯─╭─╮─ [a-z, A-Z, 0-9, "_", non-ASCII] ─╭─╮─╰──┤│
      ╰─ "-" ─╯ ╰──────── <escape> ────────────╯      │ ╰──────────── <escape> ─────────────╯ │
                                                      ╰───────────────────────────────────────╯

<number-token>
    ╭─ "+" ─╮
 │├─├───────┤─╮─╭─ [digit] ─╮─ "." ─╭─ [digit] ─╮──╭───╮──────────────────────────────────╭──┤│
    ╰─ "-" ─╯ │ ╰───────────╯       ╰───────────╯  │   │         ╭─ "+" ─╮                │
              ├───────── ╭─ [digit] ─╮─────────────┤   ├─ "e" ─╭─├───────┤──╭─ [digit] ─╮─╯
              │          ╰───────────╯             │   ╰─ "E" ─╯ ╰─ "-" ─╯  ╰───────────╯
              ╰──── "." ─╭─ [digit] ─╮─────────────╯
                         ╰───────────╯

<dimension-token>
 │├─ <number-token> ─ <ident-token> ─┤│

<dimension-token> // Refined for this lexer, not true to the standard.
 │├─ <number-token> ─╮─ <ident-token> ─╭──┤│
                     ╰────── "%" ──────╯

BadString = 6

Represents the <bad-string-token>. This token is a failure to fully lex the <string-token>.

BadUrl = 7

Represents the <bad-url-token>. This token is a failure to fully lex the <url-token>.

Ident = 8

Represents the <ident-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ─────╭──┤│
     ├─ "\t" ────┤
     ╰─ newline ─╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<ident-token>
    ╭───────────────── "--" ─────────────────────╮  ╭───────────────────────────────────────────╮
 │├─╯─╮───────╭─╮─ [a-z, A-Z, "_", non-ASCII] ─╭─╰──╯─╭─╮─ [a-z, A-Z, 0-9, "_", non-ASCII] ─╭─╮─╰──┤│
      ╰─ "-" ─╯ ╰──────── <escape> ────────────╯      │ ╰──────────── <escape> ─────────────╯ │
                                                      ╰───────────────────────────────────────╯

Function = 9

Represents the <function-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<ident-token>
    ╭───────────────── "--" ─────────────────────╮  ╭───────────────────────────────────────────╮
 │├─╯─╮───────╭─╮─ [a-z, A-Z, "_", non-ASCII] ─╭─╰──╯─╭─╮─ [a-z, A-Z, 0-9, "_", non-ASCII] ─╭─╮─╰──┤│
      ╰─ "-" ─╯ ╰──────── <escape> ────────────╯      │ ╰──────────── <escape> ─────────────╯ │
                                                      ╰───────────────────────────────────────╯

<function-token>
 │├─ <ident-token> ─ "(" ─┤│

AtKeyword = 10

Represents the <at-keyword-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<ident-token>
    ╭───────────────── "--" ─────────────────────╮  ╭───────────────────────────────────────────╮
 │├─╯─╮───────╭─╮─ [a-z, A-Z, "_", non-ASCII] ─╭─╰──╯─╭─╮─ [a-z, A-Z, 0-9, "_", non-ASCII] ─╭─╮─╰──┤│
      ╰─ "-" ─╯ ╰──────── <escape> ────────────╯      │ ╰──────────── <escape> ─────────────╯ │
                                                      ╰───────────────────────────────────────╯

<at-keyword-token>
 │├─ "@" ─ <ident-token> ─┤│

Hash = 11

Represents the <hash-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<hash-token>
 │├─ "#" ──╭─╮─ [a-z, A-Z, 0-9, "_", "-", non-ASCII] ─╭─╮─┤│
           │ ╰─────────────── <escape> ───────────────╯ │
           ╰────────────────────────────────────────────╯

String = 12

Represents the <string-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<string-token>
            ╭───────────────────────────────────╮
 │├─╮─ """ ─╯─╭─╮─ [not """, "\", newline] ─╭─╮─╰── """ ─╭─┤│
    │         │ ├──────── <escape> ─────────┤ │          │
    │         │ ╰───── "\" ─ <newline> ─────╯ │          │
    │         ╰───────────────────────────────╯          │
    │       ╭───────────────────────────────────╮        │
    ╰─ "'" ─╯─╭─╮─ [not """, "\", newline] ─╭─╮─╰── "'" ─╯
              │ ├──────── <escape> ─────────┤ │
              │ ╰───── "\" ─ <newline> ─────╯ │
              ╰───────────────────────────────╯

Url = 13

Represents the <url-token>.

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<whitespace-token>
 │├─╭─ <whitespace> ─╮─┤│
    ╰────────────────╯

<ws*>
    ╭──────────────────────────╮
 │├─╯─╭─ <whitespace-token> ─╮─╰─┤│
      ╰──────────────────────╯

<hexdigit>
 │├─ [ 0-9, A-F, a-f ] ─┤│


<escape>
 │├─ "\" ─╮───── [not <newline> or <hexdigit>] ───╭─┤│
          ╰─╭── <hexdigit> ─╮──╮────────────────╭─╯
            ╰─ (1-6 times) ─╯  ╰─ <whitespace> ─╯

<ident-token>
    ╭───────────────── "--" ─────────────────────╮  ╭───────────────────────────────────────────╮
 │├─╯─╮───────╭─╮─ [a-z, A-Z, "_", non-ASCII] ─╭─╰──╯─╭─╮─ [a-z, A-Z, 0-9, "_", non-ASCII] ─╭─╮─╰──┤│
      ╰─ "-" ─╯ ╰──────── <escape> ────────────╯      │ ╰──────────── <escape> ─────────────╯ │
                                                      ╰───────────────────────────────────────╯

<url-token>
                                        ╭───────────────────────────────────────────────────────────────────╮
 │├─ <ident-token "url"> ─ "(" ─ <ws*> ─╯─╭─╮─ [not """ "'" "(" ")" "\" <whitespace> or non-printable] ─╭─╮─╰─ <ws*> ─ ")" ─┤│
                                          │ ╰──────────────────────── <escape> ─────────────────────────╯ │
                                          ╰───────────────────────────────────────────────────────────────╯

Delim = 16

Represents the <delim-token>. The <delim-token> has a value composed of a single code point.

<delim-token>
 │├─ [codepoint] ─┤│

Colon = 17

Represents the <colon-token>.

<colon-token>
 │├─ ":" ─┤│

Semicolon = 18

Represents the <semicolon-token>.

<semicolon-token>
 │├─ ";" ─┤│

Comma = 19

Represents the <comma-token>.

<comma-token>
 │├─ "," ─┤│

LeftSquare = 20

Represents the <[-token>.

<[-token>
 │├─ "[" ─┤│

RightSquare = 21

Represents the <]-token>.

<]-token>
 │├─ "]" ─┤│

LeftParen = 22

Represents the <(-token>.

<(-token>
 │├─ "(" ─┤│

RightParen = 23

Represents the <)-token>.

<)-token>
 │├─ ")" ─┤│

LeftCurly = 24

Represents the <{-token>.

<{-token>
 │├─ "{" ─┤│

RightCurly = 25

Represents the <}-token>.

<}-token>
 │├─ "}" ─┤│

Trait Implementations

impl Clone for Kind

fn clone(&self) -> Kind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Kind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Kind

fn default() -> Kind

Returns the “default value” for a type.

impl Display for Kind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Cursor> for Kind

fn from(cursor: Cursor) -> Self

Converts to this type from the input type.

impl From<Token> for Kind

fn from(token: Token) -> Self

Converts to this type from the input type.

impl Hash for Kind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for Kind

fn cmp(&self, other: &Kind) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq<Kind> for &Cursor

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Kind> for Cursor

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Kind> for PairWise

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Kind> for SourceCursor<'_>

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Kind> for Token

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<KindSet> for Kind

fn eq(&self, other: &KindSet) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq for Kind

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Kind

fn partial_cmp(&self, other: &Kind) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Copy for Kind

impl Eq for Kind

impl StructuralPartialEq for Kind