You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Need to add validation in the inspectors for the tokenizer.
Check that all the Tokens in the tokenizer are consumed or used in the grammar, i.e. the token sets match.
Add a token filter to let the validators know if a token is going to be filtered between the tokenizer and parser. A filter tool could also be added to callback when a token is being filtered between the to so that it doesn't have to be done like it is in some of the colorizers.
Check that all the states reach to a tokenizer.
Is is possible to determine the length of a token chain and check it against the token replacement to determine if a replacement is even likely to be tokenized into the original token. Could also be done by running the tokenizer on the replacement text but would have to take into account consumed runes.
Need to add validation in the inspectors for the tokenizer.