Struct Token

pub struct Token(/* private fields */);

An abstract representation of the chunk of the source text, retaining certain “facts” about the source.

Design

The Token type is an immutable packing of two u32s that represents a unit in the source text, but without the associated offset data that points to its position in the source text. This is important because it means that equivalent Tokens are equal even in different parts of the document. For the most part a Token doesn’t represent data that can be put into a text file because it lacks the underlying character data. It is lossy. For example a Token with Kind::Ident just represents an ident, but it doesn’t retain what the keyword is). Storing raw-character data would require either storing tokens on the heap (and therefore they couldn’t be Sized) or by keeping a reference to &'a str which means larger token sizes and lifetime tracking. By not storing character data we can keep Token Sized and keep it to size_of 8, avoiding the heap, avoiding references/lifetimes, and keeping Token entirely in the stack. For a lot of tokens this is fine because the underlying character data isn’t that useful past a certain point.

A Token retains certain “facts” about the underlying unit of text, though. For example it retains the Kind, how many characters the token consumed, and various other pieces of information, depending on the Kind. In some cases, it’s entirely possible to represent the full token, including character data, into the available bits (for example Kind::Delim stores its char, Kind::Number stores its f32). Taking the time in the tokenizer to gather these facts and values can keep cache-lines hot, which speeds up subsequent checks in the parser.

If you’re familiar with “red green” syntax trees such as Swiftlang’s libsyntax, or Rust-Analyzer’s Rowan or Roslyn this might be a little familiar in some concepts. However Token does not represent a tree, and relies on resorting back to the string data to find out keyword values.

This representation of facts, kind, length, or other metadata can be quite complex - so here’s a full breakdown:

Anatomy of Token

A Token is a struct of (u32, u32). The second u32 is usually the token length (hence keeping them separate). The first u32, however, is split into 3 (sometimes 5) parts. The two u32s can be thought of like so:

  |-----|-------|--------------------------|---------------------------------|
  | TF  | K     | VD                       | Value                           |
0b| 000 | 00000 | 000000000000000000000000 | 0000000000000000000000000000000 |
  |-----|-------|--------------------------|---------------------------------|
  | 3-- | 5---- | 24---------------------- | 32----------------------------- |

TF = Type Flags (or “Token Facts”)

This represents a bit-mask in the upper-most 3 bits. The flags are general purpose and change meaning depending on the Token’s Kind. Each flag generally maps to a method so it’s not necessary to remenber the contents of this table, but it can serve as a useful reference. Note that not all methods return a bool, so footnotes have been added to explain these further.

Kind::	Flag	Description	Method
Kind::Number	001	Floating Point	Token::is_float()
	010	Has a “Sign” (-/+)	Token::has_sign()
	100	(Reserved)	–
Kind::Dimension	001	Floating Point	Token::is_float()
	010	Has a “Sign” (-/+)	Token::has_sign()
	100	Unit is a known dimension	Token::dimension_unit()1
Kind::String	001	Uses Double Quotes	Token::quote_style()2
	010	Has a closing quote	Token::has_close_quote()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::Ident	001	Contains non-lower-ASCII	Token::is_lower_case()
	010	Is a “Dashed Ident”	Token::is_dashed_ident()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::Function	001	Contains non-lower-ASCII	Token::is_lower_case()
	010	Is a “Dashed Ident”	Token::is_dashed_ident()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::AtKeyword	001	Contains non-lower-ASCII	Token::is_lower_case()
	010	Is a “Dashed Ident”	Token::is_dashed_ident()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::Hash	001	Contains non-lower-ASCII	Token::is_lower_case()
	010	First character is ASCII	Token::hash_is_id_like()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::Url	001	Has a closing paren )	Token::url_has_closing_paren()
	010	Contains whitespace after (	Token::url_has_leading_space()
	100	Contains escape characters	Token::contains_escape_chars()
Kind::CdcOrCdo	001	Is CDO (000 would be CDC)	Token::is_cdc()
	010	(Reserved)	–
	100	(Reserved)	–
Kind::Whitespace	---	Whitespace style	Token::whitespace_style()3
Kind::Delim	---	Associate whitespace rules	Token::associated_whitespace()4
Kind::Comment	---	(Special)	Token::comment_style()5

K = Kind Bits

The K value - upper-most bits 4-9 stores the 5-bit Kind.

VD = Value Data

The VD value - the lower-most 24-bits - stores data depending on the Token Kind. For most kinds this data is reserved (just 0s). The value data cannot be interrogated manually, but it packs in additional data about the underlying string to make the string easier to parse without doing the same lookups that the tokenizer already had to - such as determining lengths of the various parts of the token, or packing values so that consulting the string can be avoided (which keeps cache-lines hot).

Below describes the special kinds which use the Value Data to store yet more information about the token…

Value Data for Kind::Number

If the Kind is Kind::Number, Value Data represents the length of that number (this means the parser is restricted from representing numbers longer than 16,777,216 characters which is probably an acceptable limit). Note that this does not affect the value of a number, just the characters in a string. Numbers in CSS are f32. The vast majority of f32s can be represented in 16MM characters, but it’s possible to author a document that contains a set of numeric characters longer than 16MM code points. These scenarios are considered undefined behaviour.

Value Data for Kind::Hash

If the Kind is Kind::Hash, Value Data represents the length of that hash (this means the parser is restricted from representing IDs and hex codes longer than 16,777,216 characters which is probably an acceptable limit). Note that this restriction means that ID selectors have a much tigher limit than other tokens, such as strings or idents, but it’s very unlikely to see a 16million character ID in CSS (String, maybe).

Value Data for Kind::Url

If the Kind is Kind::Url, Value Data represents the “leading length” and “trailing length” of the URL. This means the value data is split into two 12 bit numbers:

|--------------|--------------|
| LL           | TL           |
| 000000000000 | 000000000000 |
|--------------|--------------|
| 12---------- | 12---------- |

The “leading” length represents the url( part of the token. Typically this will be 4, however it’s possible (for legacy compatibility reasons within CSS) to add whitespace between the opening parenthesis and the URL value. It’s also possible to escape the url ident portion. This means \75\52\6c( is also a valid leading section of a URL ident (which has a character length of 13), as is \000075 \000052 \00006c ( (28 characters). 12 bits allows for a maximum character length of 4,096. It is not possible to represent a URL token’s leading section using 4,096 characters so there is some headroom (wasted bytes) here.

The “trailing” length represents the ) part of the token. Typically this will be 1, however it’s possible to add any number of whitespace characters between the end of the URL and the closing parenthesis. If a CSS document contains more than 4095 whitespace characters then this is considered undefined behaviour.

Value Data for Kind::Dimension

If K is a Dimension, then this represents both the number of characters in the numeric portion of the dimension and the length of the ident portion of the dimension… or the dimension unit itself (more on that below). This means the value data is split into two 12 bit numbers:

|--------------|--------------|
| NL           | DUL          |
| 000000000000 | 000000000000 |
|--------------|--------------|
| 12---------- | 12---------- |

The NL portion - the numeric length - represents the length of characters the number contains. This means the numeric portion of a dimension can only be 4,096 characters long. This is dramatically shorter than the 16MM allowed for numbers but it’s still also incredibly generous such that it’s highly unlikely to ever be hit unless someone is intentionally trying to break the parser. The Lexer encountering a dimension with a numeric portion longer than 4,096 characters is considered undefined behaviour.

The DUL portion (if TF & 100 == 0) will represent the length of characters the ident portion of the dimension (aka the dimension unit) contains. This means the ident portion of a dimension can only be 4,096 characters long. For practical purposes CSS has a fixed set of dimensions - the longest of which (at the time of writing) are 5 characters long (e.g. svmax). Through the use of escaping shenanigans it’s possible to create a valid CSS dimension longer than 5 characters though (every ident can be made 8 times longer by using escape characters, e.g. 1svmax at 6 characters can be instead written as 1\000073 \000076 \00006d \000061 \000078 at 40 characters). In addition to these factors, it’s worth pointing out that there is scope for further dimensions and some proposals for “custom” dimensions, and lastly this library is designed for CSS and CSS-alike languages, which may invent their own dimension units. In other words being too restrictive on dimension ident length could be costly in the future, therefore 4,096 characters seems like a reasonable, if generous, trade-off.

There’s a giant caveat here though, and a carve out for parsing CSS as it exists today. If TF & 100 != 0, then the dimension is considered “known” and DUL will be encoded differently. Instead of being the dimension unit length, which requires consulting the underlying &str to get the actual dimension, it will be used to store the DimensionUnit - an enum of known CSS dimensions. In this mode Token::dimension_unit() will return a valid DimensionUnit (excluding DimensionUnit::Unknown). When it comes to reasoning about dimensions from the outside, this won’t make a significant difference but it does provide a nice performance boost in parser implementations without slowing down the Lexer by any significant amount. However, if a dimension unit is escaped in any way it will not be represented as a known DimensionUnit, due to the variability in the length encoding which would otherwise be lost if using the enum variant.

Value

The Value portion of Token represents the length of the token for most token kinds. However, for some tokens their length is already packed into the first u32. So it would make more sense to use this u32 to store more interesting data.

Value for Kind::Delim and single character tokens

Kind::Delim and single-character tokens (i.e. Kind::Colon->Kind::RightCurly) typically have a length of 1 (Kind::Delim can have a varied length for surrogate pairs). Instead of storing the length and wasting a whole u32, this region stores the char. Calling Token::char() will return an Option which will always be Some for Kind::Delim and single-character tokens.

Value for Kind::Hash

The length of a hash is stored in its VD portion, leaving 32bits to storing other data. It just so happens that a 8-character hex code (#ffaabbcc) fits nicely inside of 32-bits. During tokenization we can eagerly parse the hex code and stuff it here, so it can be more easily reasoned about in upstream code (rather than reading the character data).

Value for Kind::Number and Kind::Dimension

As these tokens store their length data in the VD portion, this u32 instead stores the value of the number, stored as f32::to_bits().

Value data for other tokens.

In all other cases, this represents the length of the token as utf-8 bytes. This means the token length is 4,294,967,296 aka ~4GB. This sounds very long but also CSS can host very large image data and browsers will accomodate very large URLs. An mdn article on Data URLs claims that Firefox supports 32mb Data URLs, Chrome supports over 512mb, and Safari over 2gb. The reality is that if someone has such a large data URL in their CSS they probably should split it out, but we have a whole 32 bits to store the length so we may as well use it…

---

1. Dimensions do not have a bool returning method for whether or not the dimension is known, instead Token::dimension_unit() == DimensionUnit::Unknown can be consulted. ↩

2. Strings do not have a bool returning method for whether or not the quote is using double or single quotes, instead the Token::quote_style() method will returning the QuoteStyle enum for better readability. ↩

3. Whitespace tokens to not have a bool returning method, instead Token::whitespace_style() will return the Whitespace enum for improved readability. ↩

4. Delims can be used in interesting ways inside of CSS syntax. At higher levels CSS is sometimes whitespace sensitive, for example the whitespace inside of a CSS selector sometimes represents the descendant combinator, meanwhile delimiters inside calc() are sensitive to whitespace collapse (calc(1px + 1px) is valid while calc(1px+1px) is a parse error). Further to this, introducing whitespace (say through a formatter) might break in interesting ways due to some combinations of Delims & Idents - for example Pseudo Classes like :hover, or CSS like languages such as SASS using $var style syntax. While :hover and $var are comprised of two tokens they’re considered one conceptual unit. Having a way to express these relationships at the token level can be useful for other low level machinery such as formatters/minifiers, rather than introducing complex state at higher levels. For these reasons, Delim tokens have the ability to express their whitespace association. The lexer will always produce a token with empty whitespace rules, but parsers can replace this token with a more complex set of rules. ↩

5. Rather than using the 3 bits as a bit-mask, Comment tokens use the data to store the CommentStyle enum, which is capable of representing 8 discrete comment styles. ↩

Implementations

impl Token

pub const EMPTY: Token

Represents an empty token.

pub const EOF: Token

Represents an EOF token.

pub const CDO: Token

Represents a CDO (<!--) token.

pub const CDC: Token

Represents a CDC (-->) token.

pub const SPACE: Token

Represents a single ’ ’ space token.

pub const TAB: Token

Represents a single Tab token.

pub const NEWLINE: Token

Represents a single \n token.

pub const NUMBER_ZERO: Token

Represents the Number 0. This is not equal to other representations of zero, such as 00, 0e0, 0.0 and so on.

pub const COLON: Token

Represents the : token.

pub const SEMICOLON: Token

Represents the ; token.

pub const COMMA: Token

Represents the , token.

pub const LEFT_SQUARE: Token

Represents the [ token.

pub const RIGHT_SQUARE: Token

Represents the ] token.

pub const LEFT_PAREN: Token

Represents the ( token.

pub const RIGHT_PAREN: Token

Represents the ) token.

pub const LEFT_CURLY: Token

Represents the { token.

pub const RIGHT_CURLY: Token

Represents the } token.

pub const BANG: Token

Represents a ! Kind::Delim token.

pub const HASH: Token

Represents a # Kind::Delim token.

pub const DOLLAR: Token

Represents a $ Kind::Delim token.

pub const PERCENT: Token

Represents a % Kind::Delim token - not to be confused with the % dimension.

pub const AMPERSAND: Token

Represents a & Kind::Delim token.

pub const ASTERISK: Token

Represents a * Kind::Delim token.

pub const PLUS: Token

Represents a + Kind::Delim token.

pub const DASH: Token

Represents a - Kind::Delim token.

pub const PERIOD: Token

Represents a . Kind::Delim token.

pub const SLASH: Token

Represents a / Kind::Delim token.

pub const LESS_THAN: Token

Represents a < Kind::Delim token.

pub const EQUALS: Token

Represents a = Kind::Delim token.

pub const GREATER_THAN: Token

Represents a > Kind::Delim token.

pub const QUESTION: Token

Represents a ? Kind::Delim token.

pub const AT: Token

Represents a @ Kind::Delim token. Not to be confused with the @keyword token.

pub const BACKSLASH: Token

Represents a \\ Kind::Delim token.

pub const CARET: Token

Represents a ^ Kind::Delim token.

pub const UNDERSCORE: Token

Represents a _ Kind::Delim token.

pub const BACKTICK: Token

Represents a ``` Kind::Delim token.

pub const PIPE: Token

Represents a | Kind::Delim token.

pub const TILDE: Token

Represents a ~ Kind::Delim token.

pub const REPLACEMENT_CHARACTER: Token

Represents a replacement character Kind::Delim token.

pub const fn dummy(kind: Kind) -> Self

Creates a “Dummy” token with no additional data, just the Kind.

pub const fn dummy_ident() -> Self

Creates a “Dummy” token with no additional data, just Kind::Ident.

pub const fn kind(&self) -> Kind

Returns the Kind.

pub const fn is_empty(&self) -> bool

The only token with an empty length is EOF, but this method is available for symmetry with len().

pub const fn len(&self) -> u32

Returns the amount of characters (utf-8 code points) this Token represents in the underlying source text.

pub const fn char(&self) -> Option<char>

If the Kind is “Delim Like” (i.e. it is Kind::Delim, Kind::Colon, Kind::Semicolon, Kind::Comma, Kind::LeftSquare, Kind::RightSquare, Kind::LeftParen, Kind::RightParen, Kind::LeftCurly, Kind::RightCurly) then this will return a Some with a char representing the value. For non-delim-like tokens this will return None.

pub const fn is_int(&self) -> bool

The Token is a Kind::Dimension or Kind::Number and is an integer - i.e. it has no ..

pub const fn is_float(&self) -> bool

The Token is a Kind::Dimension or Kind::Number and is a float - i.e. it has decimal places. This will be true even if the decimal places are 0. e.g. 0.0.

pub const fn has_sign(&self) -> bool

The Token is a Kind::Dimension or Kind::Number and the underlying character data included a - or + character. Note that a positive number may not necessarily have a sign, e.g. 3 will return false, while +3 will return true.

pub const fn numeric_len(&self) -> u32

If the Token is a Kind::Dimension or Kind::Number then this returns the amount of characters used to represent this number in the underlying source text. Numbers may be inefficiently encoded in the source text, e.g. 0.0000.

Asserts: the kind() is Kind::Dimension or Kind::Number.

pub fn value(&self) -> f32

If the Token is a Kind::Dimension or Kind::Number then this returns the f32 representation of the number’s value.

Asserts: the kind() is Kind::Dimension or Kind::Number.

pub fn whitespace_style(&self) -> Whitespace

Returns the Whitespace.

If the Token is not a Kind::Whitespace this will return Whitespace::none().

pub fn associated_whitespace(&self) -> AssociatedWhitespaceRules

Returns the AssociatedWhitespaceRules.

If the Kind is not “Delim Like” (i.e. it is not Kind::Delim, Kind::Colon, Kind::Semicolon, Kind::Comma, Kind::LeftSquare, Kind::RightSquare, Kind::LeftParen, Kind::RightParen, Kind::LeftCurly, Kind::RightCurly) then this will always return AssociatedWhitespaceRules::none().

pub fn with_associated_whitespace( &self, rules: AssociatedWhitespaceRules, ) -> Token

Returns a new Token with the AssociatedWhitespaceRules set to the given AssociatedWhitespaceRules, if possible.

If the Kind is not “Delim Like” (i.e. it is not Kind::Delim, Kind::Colon, Kind::Semicolon, Kind::Comma, Kind::LeftSquare, Kind::RightSquare, Kind::LeftParen, Kind::RightParen, Kind::LeftCurly, Kind::RightCurly) then this will return the same Token. If the AssociatedWhitespaceRules is different it will return a new Token.

pub fn comment_style(&self) -> Option<CommentStyle>

Returns the CommentStyle.

If the Token is not a Kind::Comment this will return None.

pub const fn dimension_unit(&self) -> DimensionUnit

Returns the DimensionUnit.

If the Token is not a Kind::Dimension this will return DimensionUnit::Unknown. If the Token is a Kind::Dimension, but the dimension unit is custom (e.g. dashed), has escape characters, or is not a recognised CSS Dimension, this will return DimensionUnit::Unknown.

pub fn quote_style(&self) -> QuoteStyle

Returns the QuoteStyle.

If the Token is not a Kind::String this will return QuoteStyle::None.

pub fn with_quotes(&self, quote_style: QuoteStyle) -> Token

Returns a new Token with the QuoteStyle set to the given QuoteStyle, if possible.

If the Token is not a Kind::String, or the QuoteStyle is already the given QuoteStyle this will return the same Token. If the QuoteStyle is different it will return a new Token. QuoteStyle must not be QuoteStyle::None

pub const fn has_close_quote(&self) -> bool

If the Token is a Kind::String this checks if the string ended in a close quote. It is possible to have a valid String token that does not end in a close quote, by eliding the quote at the end of a file.

Asserts: The Kind is Kind::String.

pub const fn can_escape(&self) -> bool

Checks if it is possible for the Token to contain escape characters. Numbers, for example, cannot. Idents can.

pub const fn contains_escape_chars(&self) -> bool

If the Token can escape, checks if the underlying source text contained escape characters.

Asserts: The token can escape (Token::can_escape()).

pub const fn is_dashed_ident(&self) -> bool

If the Token is Ident like, checks if the first two code points are HYPHEN-MINUS (-).

Asserts: The token is “ident like”, i.e. it is Kind::Ident, Kind::AtKeyword, Kind::Function, Kind::Hash.

pub const fn is_lower_case(&self) -> bool

Checks if the Token is Ident like and none of the characters are ASCII upper-case.

pub const fn is_trivia(&self) -> bool

Checks if the Token is Trivia-like, that is Kind::Comment, Kind::Whitespace, Kind::Eof

pub const fn url_has_leading_space(&self) -> bool

If the Token is Kind::Url, checks if there are leading Whitespace characters before the inner value.

Asserts: The token is Kind::Url.

pub const fn url_has_closing_paren(&self) -> bool

If the Token is Kind::Url, checks if the closing parenthesis is present.

Asserts: The token is Kind::Url.

pub const fn hash_is_id_like(&self) -> bool

If the Token is Kind::Hash, checks if the Hash is “ID-like” (i.e its first character is ASCII).

Asserts: The token is Kind::Hash.

pub const fn is_bad(&self) -> bool

Checks if the Token is Kind::BadString or Kind::BadUrl.

pub const fn is_cdc(&self) -> bool

Checks if the Token is Kind::CdcOrCdo and is the CDC variant of that token.

pub fn leading_len(&self) -> u32

Some tokens may have a “leading” part:

• Kind::AtKeyword always starts with a @,
• Kind::Hash with a #.
• Kind::String with a " or '.
• Kind::Comment with a leading /* (or //).
• Kind::Dimension has a leading numeric portion.
• Kind::Url has the leading url( ident (which may vary in exact representation).

This function returns the length of that, irrespective of the Kind. For other kinds not listed, this will return 0, but for the above kinds it will calculate the leading length. This is useful for parsing out the underlying data which is likely to be of greater use.

pub fn trailing_len(&self) -> u32

Some tokens may have a “trailing” part:

• Kind::Function will always have an opening (.
• Kind::String may have a closing " or '.
• Kind::Comment may have a closing */
• Kind::Url may have a clsoing ).

This function returns the length of that, irrespective of the Kind. For other kinds not listed, this will return 0, but for the above kinds it will calculate the leading length. This is useful for parsing out the underlying data which is likely to be of greater use.

pub fn to_pairwise(&self) -> Option<PairWise>

Certain kinds have a PairWise equivalent:

• Kind::LeftParen has Kind::RightParen
• Kind::LeftCurly has Kind::RightCurly
• Kind::LeftSquare has Kind::RightSquare

This function returns the PairWise enum, if the Token is one of the above listed Kinds. For any other Kind this returns None.

pub fn with_cursor(self, offset: SourceOffset) -> Cursor

A convenience function for Cursor::new(offset, token).

pub fn hex_value(self) -> u32

If the Kind is Kind::Hash then this token may have had the opportunity to be parsed as a <hex-value> (e.g. #fff). When this happens the character data is parsed during tokenization into a u32 which stores the RR,GG,BB,AA values.

pub fn needs_separator_for(&self, second: Token) -> bool

If this Token is preceded by the Token other then a separating token (e.g. a comment) will need to be inserted between these the two tokens during serialization, in order for them to be able to be re-tokenized as the same tokens. For example an Ident (“a”) adjacent to an Ident (“b”), if serialized without whitespace, would create a single Ident (“ab”). The rules for estbalishing whether or not these tokens needs whitespace are quite simple and are effectively defined in the serialization section of the spec. To reproduce the table:

	ident	function	url	bad url	-	number	percentage	dimension	CDC	(	*	%
ident	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
at-keyword	✗	✗	✗	✗	✗	✗	✗	✗	✗
hash	✗	✗	✗	✗	✗	✗	✗	✗	✗
dimension	✗	✗	✗	✗	✗	✗	✗	✗	✗
#	✗	✗	✗	✗	✗	✗	✗	✗	✗
-	✗	✗	✗	✗	✗	✗	✗	✗	✗
number	✗	✗	✗	✗		✗	✗	✗	✗			✗
@	✗	✗	✗	✗	✗				✗
.						✗	✗	✗
+						✗	✗	✗
/											✗

The one exception not in this table is that two consecutive / characters should also be separated by spaces in order to avoid abmiguities with CSS-alike languages that treat two consecutive / characters as a single line comment.

Example

use css_lexer::*;
let mut lexer = Lexer::new("10 %");
let first = lexer.advance();
let _ = lexer.advance(); // Whitespace
let second = lexer.advance();
assert!(first.needs_separator_for(second));

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Token

fn default() -> Self

Returns the “default value” for a type.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Cursor> for Token

fn from(cursor: Cursor) -> Self

Converts to this type from the input type.

impl From<Token> for DimensionUnit

fn from(token: Token) -> Self

Converts to this type from the input type.

impl From<Token> for Kind

fn from(token: Token) -> Self

Converts to this type from the input type.

impl From<Token> for KindSet

fn from(token: Token) -> Self

Converts to this type from the input type.

impl From<Token> for QuoteStyle

fn from(token: Token) -> Self

Converts to this type from the input type.

impl From<Token> for Whitespace

fn from(token: Token) -> Self

Converts to this type from the input type.

impl Hash for Token

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for Token

fn cmp(&self, other: &Token) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq<AssociatedWhitespaceRules> for Token

fn eq(&self, other: &AssociatedWhitespaceRules) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<CommentStyle> for Token

fn eq(&self, other: &CommentStyle) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<DimensionUnit> for Token

fn eq(&self, other: &DimensionUnit) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Kind> for Token

fn eq(&self, other: &Kind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<KindSet> for Token

fn eq(&self, other: &KindSet) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<PairWise> for Token

fn eq(&self, other: &PairWise) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<QuoteStyle> for Token

fn eq(&self, other: &QuoteStyle) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Token> for Cursor

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<Whitespace> for Token

fn eq(&self, other: &Whitespace) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<char> for Token

fn eq(&self, other: &char) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Token

fn partial_cmp(&self, other: &Token) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Serialize for Token

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for Token

impl Eq for Token

impl StructuralPartialEq for Token