Struct Feature

pub struct Feature { /* private fields */ }

A set of runtime feature flags which can be enabled individually or in combination, which will change the way individual Lexer produces Tokens.

To build multiple features, use the bitwise OR operator.

Example

use css_lexer::*;
let features = Feature::SingleLineComments | Feature::SeparateWhitespace;
let mut lexer = Lexer::new_with_features("// foo", features);

Implementations

impl Feature

pub const SingleLineComments: Feature

With this flag enabled the Lexer will produce Tokens with Kind::Comment when it encounters two consecutative SOLIDUS characters (//), the Token will have a length up to the next newline (\n) character. The contents between the two SOLIDUS (//) characters and the \n will be consumed by this token, so no tokens will be produced for the contents of the comment.

If this flag is not enabled, encountering something that looks like a single line commet will produce two Kind::Delim tokens for the two SOLIDUS (//) characters, and any number of other tokens depending on the contents of the comment, per the CSS specification.

A comment with two leading SOLIDUS characters is not valid CSS syntax, but might be considered valid syntax in other CSS-alike languages for example SASS.

With this feature turned off comments are tokenized per the CSS specification:

<comment>
           ╭──────────────────────────────────────────╮
 │├─ "/*" ─╯-╭─ (anything but "*" followed by "/") ─╮─╰─ "*/" ─┤│
             ╰──────────────────────────────────────╯

With this feature turned on comments are tokenized with the additional grammar:

<comment>
              ╭──────────────────────────────────────────╮
 │├──╮─ "/*" ─╯-╭─ (anything but "*" followed by "/") ─╮─╰─ "*/" ─╭─┤│
     │          ╰──────────────────────────────────────╯          │
     │              ╭───────────────────────────╮                 │
     ├─ "//" ───────╯-╭─ (anything but "\n") ─╮─╰─ "\n" ──────────╯
     ╰─               ╰───────────────────────╯

Example

use css_lexer::*;
let mut lexer = Lexer::new("// foo");
assert_eq!(lexer.advance(), Kind::Delim); // The first `/`
assert_eq!(lexer.advance(), Kind::Delim); // The second `/`
assert_eq!(lexer.advance(), Kind::Whitespace);
assert_eq!(lexer.advance(), Kind::Ident); // The "foo" in the comment

lexer = Lexer::new_with_features("// foo", Feature::SingleLineComments);
let token = lexer.advance();
assert_eq!(token, Kind::Comment); // The whole comment "// foo"
assert_eq!(token, CommentStyle::Single);

pub const SeparateWhitespace: Feature

The CSS Spec mentions that whitespace tokens should be combined into a single Whitespace token. This means a single whitespace token can contain a cominbation of newlines, tabs, and space characters. This is often fine as whitespace is generally ignored during parsing, however for certain IDE features it might be important to tokenize discrete whitespace Tokens, each with their own discrete whitespace. Enabling this flag will enforce that the Lexer outputs these discrete tokens. In other words with this feature enabled, multiple contiguous whitespace tokens may be returned from subsequent calls to Lexer::advance(), but with this feature off this will never be the case (as whitespace is collapsed into a single Token).

With this feature turned off whitespace-tokens are tokenized per the CSS specification:

<newline>
 │├──╮─ "\n" ───╭──┤│
     ├─ "\r\n" ─┤
     ├─ "\r" ───┤
     ╰─ "\f" ───╯

<whitespace>
 │├──╮─ " " ───────╭──┤│
     ├─ "\t" ──────┤
     ╰─ <newline> ─╯

<whitespace-token>
 │├─╭─ <whitespace> ─╮─┤│
    ╰────────────────╯

With this feature turned on whitespace-tokens are tokenized with the additional grammar:

<whitespace-token>
 │├──╮─╭─ " " ───────╮─╭──┤│
     │ ╰─────────────╯ │
     ├─╭─ "\t" ──────╮─┤
     │ ╰─────────────╯ │
     ╰─╭─ <newline> ─╮─╯
       ╰─────────────╯

Example

use css_lexer::*;
let mut lexer = Lexer::new("\n\thello world");
{
	// This token will be collapsed Whitespace.
	let token = lexer.advance();
	assert_eq!(token, Kind::Whitespace);
	// The Whitespace is comprised of many bits:
	assert_eq!(token, Whitespace::Newline | Whitespace::Tab);
}
assert_eq!(lexer.advance(), Kind::Ident);
{
	let token = lexer.advance();
	assert_eq!(token, Kind::Whitespace);
	assert_eq!(token, Whitespace::Space);
}
assert_eq!(lexer.advance(), Kind::Ident);

lexer = Lexer::new_with_features("\n\thello world", Feature::SeparateWhitespace);
{
	// This token will be discrete Whitespace, just the `\n`.
	let token = lexer.advance();
	assert_eq!(token, Kind::Whitespace);
	// The Whitespace is comprised of a single bit:
	assert_eq!(token, Whitespace::Newline);
}
{
	// This token will be discrete Whitespace, just the `\t`.
	let token = lexer.advance();
	assert_eq!(token, Kind::Whitespace);
	// The Whitespace is comprised of a single bit:
	assert_eq!(token, Whitespace::Tab);
}
assert_eq!(lexer.advance(), Kind::Ident);
{
	let token = lexer.advance();
	assert_eq!(token, Kind::Whitespace);
	assert_eq!(token, Whitespace::Space);
}
assert_eq!(lexer.advance(), Kind::Ident);

pub const fn bits(&self) -> u8

Returns the underlying bits of the bitmask.

pub const fn all_bits() -> Self

Returns a bitmask that contains all values.

This will include bits that do not have any flags. Use ::all_flags() if you only want to use flags.

pub const fn all_flags() -> Self

Returns a bitmask that contains all flags.

pub const fn is_all_bits(&self) -> bool

Returns true if the bitmask contains all values.

This will check for bits == !0, use .is_all_flags() if you only want to check for all flags

pub const fn is_all_flags(&self) -> bool

Returns true if the bitmask contains all flags.

This will fail if any unused bit is set, consider using .truncate() first.

pub const fn all() -> Self

👎Deprecated: Please use the ::all_bits() constructor

Returns a bitmask that contains all values.

This will include bits that do not have any flags. Use ::all_flags() if you only want to use flags.

pub const fn is_all(&self) -> bool

👎Deprecated: Please use the .is_all_bits() method

Returns true if the bitmask contains all values.

This will check for bits == !0, use .is_all_flags() if you only want to check for all flags

pub const fn full() -> Self

👎Deprecated: Please use the ::all_flags() constructor

Returns a bitmask that contains all flags.

pub const fn is_full(&self) -> bool

👎Deprecated: Please use the .is_all_flags() method

Returns true if the bitmask contains all flags.

This will fail if any unused bit is set, consider using .truncate() first.

pub const fn none() -> Self

Returns a bitmask that does not contain any values.

pub const fn is_none(&self) -> bool

Returns true if the bitmask does not contain any values.

pub const fn truncate(&self) -> Self

Returns a bitmask that only has bits corresponding to flags

pub const fn intersects(&self, other: Self) -> bool

Returns true if self intersects with any value in other, or if other does not contain any values.

This is equivalent to (self & other) != 0 || other == 0.

pub const fn contains(&self, other: Self) -> bool

Returns true if self contains all values of other.

This is equivalent to (self & other) == other.

pub const fn not(self) -> Self

Returns the bitwise NOT of the bitmask.

pub const fn and(self, other: Self) -> Self

Returns the bitwise AND of the bitmask.

pub const fn or(self, other: Self) -> Self

Returns the bitwise OR of the bitmask.

pub const fn xor(self, other: Self) -> Self

Returns the bitwise XOR of the bitmask.

Trait Implementations

impl Binary for Feature

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl BitAnd for Feature

type Output = Feature

The resulting type after applying the & operator.

fn bitand(self, rhs: Self) -> Self::Output

Performs the & operation.

impl BitAndAssign for Feature

fn bitand_assign(&mut self, rhs: Self)

Performs the &= operation.

impl BitOr for Feature

type Output = Feature

The resulting type after applying the | operator.

fn bitor(self, rhs: Self) -> Self::Output

Performs the | operation.

impl BitOrAssign for Feature

fn bitor_assign(&mut self, rhs: Self)

Performs the |= operation.

impl BitXor for Feature

type Output = Feature

The resulting type after applying the ^ operator.

fn bitxor(self, rhs: Self) -> Self::Output

Performs the ^ operation.

impl BitXorAssign for Feature

fn bitxor_assign(&mut self, rhs: Self)

Performs the ^= operation.

impl Clone for Feature

fn clone(&self) -> Feature

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Feature

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Feature

fn default() -> Feature

Returns the “default value” for a type.

impl From<Feature> for u8

fn from(val: Feature) -> u8

Converts to this type from the input type.

impl From<u8> for Feature

fn from(val: u8) -> Self

Converts to this type from the input type.

impl Hash for Feature

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl LowerHex for Feature

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Not for Feature

type Output = Feature

The resulting type after applying the ! operator.

fn not(self) -> Self::Output

Performs the unary ! operation.

impl Octal for Feature

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Ord for Feature

fn cmp(&self, other: &Feature) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq<u8> for Feature

fn eq(&self, other: &u8) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq for Feature

fn eq(&self, other: &Feature) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Feature

fn partial_cmp(&self, other: &Feature) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl UpperHex for Feature

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Copy for Feature

impl Eq for Feature

impl StructuralPartialEq for Feature