Set version on main to `56.0.0-dev`, document release versioning scheme by alamb · Pull Request #26845 · apache/datafusion

alamb · 2026-10-28T11:42:06Z

Which issue does this PR close?

Part of Release DataFusion 56.0.0 (Oct/Nov 2026) #24461

Rationale for this change

As discussed on #24461, it is currently hard to tell what version of DataFusion a build came from: main reports the previous release version (55.1.0) even though it contains unreleased code destined for 56.0.0.

This PR changes the version on main to 56.0.0-dev (the semver pre-release form of the upcoming release). The benefits are:

it is clearer that the version on main is not a release
-dev versions can never be accidentally published to crates.io as if they were a release

What changes are included in this PR?

Details

- Bump the workspace version in `Cargo.toml` / `Cargo.lock` to `56.0.0-dev` - Regenerate `docs/source/user-guide/configs.md` (the default of `datafusion.execution.parquet.created_by` includes the version) - Update `dev/release/README.md` with the new versioning scheme: a new "Version Numbers" section, a new step "1b. Bump Version on main" describing the version bump PR, updated step 2 examples, and step 11 ("Sync Changelog and Documentation to main") no longer syncs the release version back to `main`, only the changelog and the user facing crate examples - Update `docs/source/contributor-guide/release_management.md` to describe the `-dev` convention - Update `dev/update_datafusion_versions.py` so that it: - Sets the release labeler target to the version without the `-dev` suffix (e.g. `56.0.0-dev` -> `v56.0.0`) rather than `major + 1` - Skips updating the user facing `Cargo.toml` examples (e.g. `docs/source/download.md`) for `-dev` versions, since those are never published to crates.io

Notes:

What is the testing strategy for this PR?

Existing CI

Are there any user-facing changes?

Builds from main now report version 56.0.0-dev instead of 55.1.0. The release process documentation is updated.

…the version string The default created_by embeds the DataFusion version, so a longer version string (e.g. 56.0.0-dev) changes the size of every written parquet file, which shifts byte based metrics (scan_efficiency_ratio, file_groups byte ranges, file_size_bytes) in sqllogictest and datafusion-cli expectations. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 4ede8bd)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…dent of the version string (apache#26892) ## Which issue does this PR close? - Part of apache#24461 - Split out from apache#26845 ## Rationale for this change While testing a `56.0.0-dev` version on `main` in apache#26845, CI failed because the default `datafusion.execution.parquet.created_by` value (`datafusion version 55.1.0`) is written into every Parquet file, so changing the version string changes the size of every Parquet file written during tests. Since this happens any time the version string changes, let's just hard code it into the tests that don't matter ## What changes are included in this PR? Hard code a version in tests that don't need it ## What is the testing strategy for this PR? Existing CI. ## Are there any user-facing changes? No, test only. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

alamb · 2026-10-29T17:30:30Z

 08)----------FilterExec: n@0 < 5, metrics=[]
 09)------------WorkTableExec: name=w, metrics=[]
-10)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/hash_join_dynamic_filter.parquet]]}, projection=[k], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ true ], dynamic_rg_pruning=eligible, metrics=[output_rows=4, output_batches=4, files_ranges_pruned_statistics=4 total → 4 matched, row_groups_pruned_statistics=4 total → 4 matched, page_index_pages_pruned=4 total → 4 matched, page_index_rows_pruned=400 total → 400 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=4, files_processed=4, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=396, predicate_cache_inner_records=400, predicate_cache_records=4, scan_efficiency_ratio=157.36% (1.14 K/727)]
+10)--------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/hash_join_dynamic_filter.parquet]]}, projection=[k], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ true ], dynamic_rg_pruning=eligible, metrics=[output_rows=4, output_batches=4, files_ranges_pruned_statistics=4 total → 4 matched, row_groups_pruned_statistics=4 total → 4 matched, page_index_pages_pruned=4 total → 4 matched, page_index_rows_pruned=400 total → 400 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=4, files_processed=4, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=396, predicate_cache_inner_records=400, predicate_cache_records=4, scan_efficiency_ratio=160.67% (1.14 K/712)]


percentage difference is due to updatings to the version

alamb · 2026-11-01T12:49:20Z

+| datafusion.execution.parquet.statistics_enabled                         | page                          | (writing) Sets if statistics are enabled for any column Valid values are: "none", "chunk", and "page" These values are not case sensitive. If NULL, uses default parquet writer setting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| datafusion.execution.parquet.max_row_group_size                         | 1048576                       | (writing) Target maximum number of rows in each row group (defaults to 1M rows). Writing larger row groups requires more memory to write, but can get better compression and be faster to read. When `max_row_group_bytes` is also set, the writer flushes a row group when either limit is reached, whichever comes first.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| datafusion.execution.parquet.max_row_group_bytes                        | NULL                          | (writing) Target maximum size of each row group in bytes. When set, the writer flushes whenever either this limit or `max_row_group_size` is reached, whichever comes first. Useful for bounding writer memory on wide schemas where a row-count limit can map to very different byte sizes. Matches the behavior of `parquet.block.size` in parquet-mr. If `None` (the default), only the row-count limit applies. Currently only honored when `allow_single_file_parallelism` is `false`; by default the parallel file writer ignores this limit.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| datafusion.execution.parquet.created_by                                 | datafusion version 56.0.0-dev | (writing) Sets "created by" property                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |


If you look at this diff with whitespace blind: https://github.com/apache/datafusion/pull/26845/changes?w=1

It only changes the created_by field, everything else is whitespace diffs

alamb · 2026-11-01T12:49:54Z

-### 11. Sync Changelog and Version to main
+### 12. Sync Changelog and Documentation to main
+
+The crate version is **not** synced back to `main`: `main` already carries the


FYI @xudong963 -- here is a proposal for how to do versions after the next release

nuno-faria

Thanks @alamb. I left some minor nits below.

nuno-faria · 2026-11-01T16:46:09Z

+```shell
+python3 dev/update_datafusion_versions.py 57.0.0-dev
+cargo update --workspace       # updates Cargo.lock
+./dev/update_config_docs.sh    # regenerates docs/source/user-guide/configs.md


I guess the update config docs is to update the formatting? I think we could also just run prettier to be faster.

Makes sense -- updated.

nuno-faria · 2026-11-01T16:47:57Z

+  ```shell
+  sed -i '' 's/55\.1\.0/56.0.0/' docs/source/download.md docs/source/user-guide/{crate-configuration,example-usage}.md
+  ```


What if the update versions script had a flag to just update the user facing ones?

like add a --user-facing-docs-only mode

What if the update versions script had a flag to just update the user facing ones?

The sed -i '' form in this step is BSD (macOS) syntax. GNU sed takes the backup suffix only when it's attached to -i, so on Linux the '' becomes the script and s/55\.1\.0/56.0.0/ is read as a file name. A docs-only mode in update_datafusion_versions.py would avoid that as well as the hand-maintained file list.

ok, I had claude whip it up

mbutrovich

Thanks @alamb! Just a minor docs request for anyone trying to use main as a dependency.

mbutrovich · 2026-11-02T19:41:01Z

 rust-version = "1.95.0"
 # Define DataFusion version
-version = "55.1.0"
+version = "56.0.0-dev"


Should we add a sentence to the Using the nightly DataFusion builds section about how to depend on main now that it's a pre-release? A git dependency with no version still resolves. A requirement like version = "56" next to the git source, or a [patch.crates-io] entry used with a "56" requirement, no longer matches, because Cargo only matches a pre-release when the requirement names one:

error: failed to select a version for the requirement `a = "^56"` candidate versions found which didn't match: 56.0.0-dev help: if you are looking for the prerelease package it needs to be specified explicitly a = { version = "56.0.0-dev" }

Downstream projects that test against main will hit this the first time they bump, so one line saying to use version = "57.0.0-dev" (or leave the version off) would save them the search.

# Conflicts: # docs/source/user-guide/configs.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mbutrovich

Thanks @alamb!

alamb · 2026-11-02T21:59:05Z

Thanks. Let's give it a try@

github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Oct 28, 2026

alamb mentioned this pull request Oct 29, 2026

test: pin parquet created_by in tests so expected output is independent of the version string #26892

Merged

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Oct 29, 2026

alamb and others added 3 commits October 29, 2026 15:26

Update limit_pruning.slt byte count for pinned created_by

6405061

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 4ede8bd)

Set version on main to 56.0.0-dev, document release versioning scheme

f98a6e4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

alamb force-pushed the alamb/dev-version branch from 939eead to f98a6e4 Compare October 29, 2026 19:34

nuno-faria self-assigned this Oct 30, 2026

Merge remote-tracking branch 'apache/main' into alamb/dev-version

b1c8264

github-actions Bot removed the sqllogictest SQL Logic Tests (.slt) label Oct 30, 2026

alamb and others added 4 commits November 1, 2026 07:05

Merge remote-tracking branch 'apache/main' into alamb/dev-version

8589c93

Renumber release steps so Bump Version on main is step 2

52c918f

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

trim

e2c919d

clean

d0bbb28

alamb commented Nov 1, 2026

View reviewed changes

alamb marked this pull request as ready for review November 1, 2026 12:50

alamb requested review from mbutrovich and xudong963 November 1, 2026 12:50

nuno-faria approved these changes Nov 1, 2026

View reviewed changes

xudong963 approved these changes Nov 2, 2026

View reviewed changes

mbutrovich requested changes Nov 2, 2026

View reviewed changes

alamb added 3 commits November 2, 2026 16:09

Merge remote-tracking branch 'apache/main' into alamb/dev-version

70f844a

# Conflicts: # docs/source/user-guide/configs.md

review comments

14b458c

Merge remote-tracking branch 'apache/main' into alamb/dev-version

b2a9aba

alamb requested a review from mbutrovich November 2, 2026 21:30

prettier, remove unused link

b536421

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mbutrovich approved these changes Nov 2, 2026

View reviewed changes

alamb added this pull request to the merge queue Nov 2, 2026

Merged via the queue into apache:main with commit d38fde4 Nov 2, 2026
46 checks passed

alamb deleted the alamb/dev-version branch November 2, 2026 22:12

github-actions Bot added the v56.0.0 label Nov 2, 2026

Conversation

alamb commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuno-faria left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb commented Oct 28, 2026 •

edited

Loading