Skip to content

Add Token Filter and Validate Tokenizer #20

Description

@Grant-Nelson

Need to add validation in the inspectors for the tokenizer.

  • Check that all the Tokens in the tokenizer are consumed or used in the grammar, i.e. the token sets match.
  • Add a token filter to let the validators know if a token is going to be filtered between the tokenizer and parser. A filter tool could also be added to callback when a token is being filtered between the to so that it doesn't have to be done like it is in some of the colorizers.
  • Check that all the states reach to a tokenizer.
  • Is is possible to determine the length of a token chain and check it against the token replacement to determine if a replacement is even likely to be tokenized into the original token. Could also be done by running the tokenizer on the replacement text but would have to take into account consumed runes.
  • Add additional checks if they are thought up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions