Skip to content

[FEATURE] Declare the restart parameters on extract() and document what a restart costs - #672

Open
noel-improv wants to merge 1 commit into
awslabs:mainfrom
noel-improv:docs/restart-api-timings
Open

noel-improv wants to merge 1 commit into
awslabs:mainfrom
noel-improv:docs/restart-api-timings

Conversation

@noel-improv

Copy link
Copy Markdown
Collaborator

Description

extract() accepted run_id and run_plan_store only through **kwargs, so a restartable run was missing from its signature, its docstring and help(). This declares both as named parameters and documents what a restart costs, using the restart benchmark from #590.

Changes

  • LexicalGraphIndex.extract() declares run_id and run_plan_store (both default None), documents them, and passes them to the extraction pipeline only. Before this they also reached BuildPipeline.create, which does not use them.
  • indexing.mdx adds "What a restart costs" to the restart section: per-restart time at 100, 500 and 1,000 documents, the configuration behind each figure, and what the figures leave out.
  • Two unit tests: the parameters reach the extraction pipeline and not the build pipeline, and both are None when not passed.

No feature flag gates restart, so nothing was removed. I searched lexical-graph/src, docs-site, the examples and every env.template and README for one.

Problem

Restart shipped in 3.20, but its parameters were undocumented on the method a caller uses, and there was no published number for what a restart costs.

Related issue (if any): #

Testing

  • Unit tests added/updated
  • Integration tests added (as appropriate)
  • Existing tests pass (pytest)
  • Tested manually (describe below)

tests/unit/test_lexical_graph_index.py and tests/unit/indexing/extract: 409 passed. With lexical_graph_index.py reverted to main, both new tests fail. The docs page was not built locally.

Checklist

  • Code follows existing style and conventions
  • License headers present on new files
  • Documentation updated (if applicable)
  • No breaking changes (or clearly documented)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…publish what a restart costs

extract() took run_id and run_plan_store only through **kwargs, so they
were missing from its signature, its docstring and help(), and they also
reached BuildPipeline.create, which has no use for them. They are now
named parameters, documented, and passed to the extraction pipeline only.

indexing.mdx gains the measured cost of a restart at 100, 500 and 1,000
documents, with the configuration behind each figure and what the
figures leave out.
@noel-improv
noel-improv marked this pull request as ready for review November 5, 2026 22:19
@noel-improv noel-improv self-assigned this Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant