Need a way to indicate a Predef in the tokenizer. For example (Start): \w =>(Word): \w => [Word] where \w indicates to use Predef.Letter. These should try to match regular expressions escapes such as \s indicates to use Predef.WhiteSpace but there will be some which might not.
This should work with grouping, e.g. (Start): \w, '!' => (Wow) and could include inverses, like \W for not word, were the predef would have to be wrapped in a not matcher. You may also add additional predef values as needed, which includes already not-ed predef values instead of wrapping them in a not matcher. This should not work in a range, e.g. (Start): \w..'z' => (Bad).
Note: Not sure on the syntax for this. The \ seem like it would be fine but we would want to make sure it doesn't conflict with Automatic State Generation and Regular Expressions features.
Need a way to indicate a
Predefin the tokenizer. For example(Start): \w =>(Word): \w => [Word]where\windicates to usePredef.Letter. These should try to match regular expressions escapes such as\sindicates to usePredef.WhiteSpacebut there will be some which might not.This should work with grouping, e.g.
(Start): \w, '!' => (Wow)and could include inverses, like\Wfor not word, were thepredefwould have to be wrapped in a not matcher. You may also add additionalpredefvalues as needed, which includes already not-edpredefvalues instead of wrapping them in a not matcher. This should not work in a range, e.g.(Start): \w..'z' => (Bad).Note: Not sure on the syntax for this. The
\seem like it would be fine but we would want to make sure it doesn't conflict with Automatic State Generation and Regular Expressions features.