Repository navigation
docs(dataset-release): state that record ids are stable across releases - #498
Merged
Merged
Conversation
- I say in the records.parquet id description and in the release README that a record id is its database primary key, so it stays the same across releases. - I add a unit test that builds twice from one in-memory database, the second time with an added product, and checks every existing record keeps its id. - I add a changelog entry for dataset users. Signed-off-by: Simon van Lierde <90462640+simonvanlierde@users.noreply.github.com>
… in the test - I say a record keeps its id in every release that includes it, since a later release can leave out deleted or excluded records. - I regenerate the docs codebook, which is built from RECORDS_SCHEMA. - The test now checks parent ids too and says which assertion catches which renumbering bug. Signed-off-by: Simon van Lierde <90462640+simonvanlierde@users.noreply.github.com>
simonvanlierde
force-pushed
the
stable-record-ids
branch
from
October 31, 2026 20:02
fec3558 to
8af90da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This states and tests that record ids in the dataset release stay the same across releases. The builder already publishes the database primary key as the id, but nothing stated it and no test pinned it. teardown-classifier splits products by a hash of their id, so a renumbered record would silently move between train and test splits.
records.parquetid description and in the release README that a record id is its database primary key, so it stays the same across releases.[Unreleased]for dataset users.RECORDS_SCHEMA.Tests:
cd backend && uv run --group tests --group dataset-release pytest tests/unit/scripts/test_build_dataset_release.py -q -k stable_idrg -q 'stable' backend/scripts/build_dataset_release.py