fix(chunking): measure inter-chunk overlap in tokens under token chunking

[eeshsaxena]

overlap is documented as a size in the chunking window's units, and under max_tokens chunking every other size (max_tokens, new_after_n_tokens, hard_max, soft_max) is measured in tokens via ChunkingOptions.measure. The text splitter follows that: when it splits an oversized element it calls _get_token_overlap_tail(fragment, overlap) and hands the next fragment approximately overlap tokens.

PreChunk.overlap_tail, which produces the overlap between two whole chunks, did not:

overlap = self._opts.inter_chunk_overlap
return self._text[-overlap:].strip() if overlap else ""

That is a character slice regardless of mode, so the same overlap value means two different things depending on which boundary it lands on.

With max_tokens=100, tokenizer="cl100k_base", overlap=50, overlap_all=True over a long paragraph:

split boundary  -> 50 tokens   (as requested)
chunk boundary  -> 50 chars = 13 tokens

So inter-chunk overlap is silently about a quarter of what was asked for, and it varies with the text, since characters-per-token is not constant. Overlap exists to keep context across a boundary for retrieval, so quietly shrinking it degrades results without any error.

The fix pulls the existing binary-search tail lookup out of _TextSplitter into a module-level _token_overlap_tail(text, target_tokens, measure) and calls it from both places. _TextSplitter._get_token_overlap_tail now delegates to it, so the splitter's behaviour is byte-for-byte the same. Character-based chunking still takes the character slice and is untouched.

After the change the chunk boundary carries approximately 50 tokens, matching the split boundary.

Added two cases to DescribePreChunk in test_unstructured/chunking/test_base.py: one asserting the tail measures the requested token count (and is a real suffix of the text), one covering text shorter than the overlap. Verified with pytest test_unstructured/chunking/test_base.py: the token case fails against the current code, and all 226 pass with the fix. The existing character-mode overlap_tail cases pass unchanged either way.

I kept the approximate semantics rather than making the tail exact, since _get_token_overlap_tail already rounds to a word boundary and the docstring says the overlap "will not exceed this number ... but may be less as required to respect splitting-character boundaries".

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured/chunking/base.py">

<violation number="1" location="unstructured/chunking/base.py:740">
P3: With this change, `overlap`/`inter_chunk_overlap` are token counts when `max_tokens` is configured but character counts otherwise. The `overlap`, `inter_chunk_overlap`, and the `ChunkingOptions` kwarg docstrings still describe them as character counts only, so the public API docs are now inconsistent with the token-mode behavior this PR establishes. Consider updating those docstrings to say the value is characters in character-based mode and tokens in token-based mode, so the same option isn't documented two conflicting ways.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: With this change, overlap/inter_chunk_overlap are token counts when max_tokens is configured but character counts otherwise. The overlap, inter_chunk_overlap, and the ChunkingOptions kwarg docstrings still describe them as character counts only, so the public API docs are now inconsistent with the token-mode behavior this PR establishes. Consider updating those docstrings to say the value is characters in character-based mode and tokens in token-based mode, so the same option isn't documented two conflicting ways.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/chunking/base.py, line 740:

<comment>With this change, `overlap`/`inter_chunk_overlap` are token counts when `max_tokens` is configured but character counts otherwise. The `overlap`, `inter_chunk_overlap`, and the `ChunkingOptions` kwarg docstrings still describe them as character counts only, so the public API docs are now inconsistent with the token-mode behavior this PR establishes. Consider updating those docstrings to say the value is characters in character-based mode and tokens in token-based mode, so the same option isn't documented two conflicting ways.</comment>

<file context>
@@ -729,7 +729,16 @@ def overlap_tail(self) -> str:
+        # -- tokens at a split boundary but characters at a chunk boundary, silently
+        # -- shrinking inter-chunk overlap to a fraction of what was requested.
+        if self._opts.use_token_counting:
+            return _token_overlap_tail(self._text, overlap, self._opts.measure).strip()
+        return self._text[-overlap:].strip()
 
</file context>