Some tokenizer states lead to another state with the same transition or leads to the same token. This should be handled as if it was the same as all of those being individual state definitions.
For example (A, B, C): '&' => (D); or (A, B, C) => [Int].
This only works at the start of the state. Something like (A): 'a'..'c' => (A, B, C) would not work since only the first state would be taken and the rest would be unreachable from this transition.
Some tokenizer states lead to another state with the same transition or leads to the same token. This should be handled as if it was the same as all of those being individual state definitions.
For example
(A, B, C): '&' => (D);or(A, B, C) => [Int].This only works at the start of the state. Something like
(A): 'a'..'c' => (A, B, C)would not work since only the first state would be taken and the rest would be unreachable from this transition.