Expose include_orig_elements param in partition API - #576
Conversation
Add include_orig_elements as a form parameter (default True) so callers can omit orig_elements blobs from chunk metadata and reduce response size for large documents with tables. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Closing — resubmitting from branch on upstream repo. |
Co-authored-by: Cursor <cursoragent@cursor.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Adds include_orig_elements parameter to partition API, defaulting to True, to optionally omit orig_elements blobs from chunk metadata (reducing payload size). Includes version bump and CHANGELOG update. Backward-compatible, low-risk additive change.
Re-trigger cubic
|
@awalker4 what is the process for getting this merged in? |
|
apologies for the delay! |
Assert the new partition kwarg in parallel mode param passing and add Speakeasy-style array form param coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Head branch was pushed to by a user without write access
|
All good. Just updated tests. Didn't see them initially |
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would auto-approve. Adds include_orig_elements boolean parameter to partition API and wires through pipeline, form model, and tests.
Re-trigger cubic
2 participants
Summary
include_orig_elementsas a new form parameter (defaultTrue) to control whether original elements are included in chunk metadataTrue, elements used to form each chunk are attached to that chunk's.metadata.orig_elementsas a gzipped+base64 blobFalse, these blobs are omitted from the response, significantly reducing payload size for large documents (especially those with large tables where the blob is duplicated into every chunk)GeneralFormParams,pipeline_api, and all relevant chunking call sitesChangelog
Added a
0.1.8entry under### FeaturesinCHANGELOG.md.Test plan
include_orig_elements=falseand verify response chunks do not containorig_elementsin metadatainclude_orig_elements=true(or omit the parameter) and confirmorig_elementsis present in chunk metadata as expectedMade with Cursor