fix(indexing): treat special tokens as text

[Iams4kura]

Description

• Treat tiktoken special-token markers found in indexed documents as ordinary source text when calculating token counts.
• Keep the tokenizer cached while making the intended disallowed_special=() behavior explicit and covered by a regression test.
• Fixes [BUG] - If you want this text to be encoded as a special token, pass it to allowed_special, e.g. allowed_special={'<|endoftext|>', ...}. If you want this text to be encoded as normal text, disable the check for this token by passing disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'}). To disable this check for all special tokens, pass disallowed_special=(). #311.

Type of change

• New features (non-breaking change).
• Bug fix (non-breaking change).
• Breaking change (fix or feature that would cause existing functionality not to work as expected).

Checklist

• I have performed a self-review of my code.
• I have added thorough tests if it is a core feature.
• There is a reference to the original bug report and related work.
• I have commented on my code, particularly in hard-to-understand areas. (The small tokenizer wrapper is self-explanatory; no additional comment is needed.)
• The feature is well documented. (No user-facing documentation changes are needed for this bug fix.)

Regression evidence

• Before the fix, the focused regression test raised ValueError for literal <|endoftext|> input.
• After the fix, the same test passes and matches encoding.encode(text, disallowed_special=()).

Verification

• pytest libs/ktem/ktem_tests/test_file_pipeline.py -q: 1 passed.
• pytest libs/kotaemon/tests/: 116 passed, 20 skipped.
• pre-commit run --all-files: all hooks passed.
• git diff --check: passed.