WIP string speed up (hash) by Rich-T-kid · Pull Request #26157 · apache/datafusion

Rich-T-kid · 2026-10-09T13:13:12Z

Specialize the Utf8View/BinaryView hash dispatch:

Inlined-only paths (no external buffers) get an unrolled u128 hot loop for fresh and rehash. Lane widths are narrower than the primitive variants (8-way fresh, 4-way rehash) to avoid spilling u128 lanes.
Nullable inlined-only paths walk the null bitmap in u64 chunks and branchlessly csel-blend mixed chunks, instead of paying per-row is_null.
Buffered paths (external data buffers) keep the per-row loop since buffer-load latency dominates the null check.

Measured on M4 Max, BATCH_SIZE=8192, 3% null density:

case	before	after	Δ
utf8_view: single, no nulls	19.11 µs	17.23 µs	-9.8%
utf8_view: multiple, no nulls (3)	54.13 µs	53.31 µs	-1.5%
utf8_view: single, nulls	19.62 µs	19.54 µs	-0.4%
utf8_view: multiple, nulls (3)	58.86 µs	58.03 µs	-1.4%
utf8_view (small): single, no nulls	2.55 µs	2.25 µs	-11.6%
utf8_view (small): multiple, no	9.77 µs	8.47 µs	-13.3%
utf8_view (small): single, nulls	6.91 µs	3.65 µs	-47.2%
utf8_view (small): multiple, nulls	20.43 µs	10.18 µs	-50.2%

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Specialize the Utf8View/BinaryView hash dispatch: - Inlined-only paths (no external buffers) get an unrolled u128 hot loop for fresh and rehash. Lane widths are narrower than the primitive variants (8-way fresh, 4-way rehash) to avoid spilling u128 lanes. - Nullable inlined-only paths walk the null bitmap in u64 chunks and branchlessly csel-blend mixed chunks, instead of paying per-row `is_null`. - Buffered paths (external data buffers) keep the per-row loop since buffer-load latency dominates the null check. Measured on M4 Max, BATCH_SIZE=8192, 3% null density: | case | before | after | Δ | |-------------------------------------|-----------|-----------|--------| | utf8_view: single, no nulls | 19.11 µs | 17.23 µs | -9.8% | | utf8_view: multiple, no nulls (3) | 54.13 µs | 53.31 µs | -1.5% | | utf8_view: single, nulls | 19.62 µs | 19.54 µs | -0.4% | | utf8_view: multiple, nulls (3) | 58.86 µs | 58.03 µs | -1.4% | | utf8_view (small): single, no nulls | 2.55 µs | 2.25 µs | -11.6% | | utf8_view (small): multiple, no | 9.77 µs | 8.47 µs | -13.3% | | utf8_view (small): single, nulls | 6.91 µs | 3.65 µs | -47.2% | | utf8_view (small): multiple, nulls | 20.43 µs | 10.18 µs | -50.2% |

Rich-T-kid · 2026-10-09T13:13:44Z

run benchmark with_hashes
env:
BENCH_FILTER: view

adriangbot · 2026-10-09T13:16:26Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T13:15:47.997417Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26157, benchmarks=with_hashes
2026-10-09T13:15:48.054074Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T13:15:48.054096Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T13:15:53.066583Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26157/head:rich-T-kid/faster-utf8view-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T13:16:08.070686Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-utf8view-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T13:16:13.072935Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T13:16:18.075246Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T14:12:56Z

run benchmark with_hashes
env:
BENCH_FILTER: utf8

adriangbot · 2026-10-09T14:15:18Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T14:14:40.003392Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26157, benchmarks=with_hashes
2026-10-09T14:14:40.071727Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T14:14:40.071748Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T14:14:45.088061Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26157/head:rich-T-kid/faster-utf8view-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:15:00.092681Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-utf8view-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T14:15:05.093924Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:15:10.096576Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T14:42:25Z

run benchmark with_hashes

adriangbot · 2026-10-09T14:44:44Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T14:44:05.291519Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26157, benchmarks=with_hashes
2026-10-09T14:44:05.351413Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T14:44:05.351435Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T14:44:10.361149Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26157/head:rich-T-kid/faster-utf8view-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:44:30.366757Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-utf8view-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T14:44:35.368870Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:44:40.370181Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

codecov-commenter · 2026-10-09T15:14:54Z

Codecov Report

❌ Patch coverage is 80.55556% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.77%. Comparing base (5134a1a) to head (3c4b8df).
⚠️ Report is 666 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/hash_utils.rs	80.55%	35 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26157      +/-   ##
==========================================
+ Coverage   81.38%   82.77%   +1.38%     
==========================================
  Files        1116     1147      +31     
  Lines      397960   451020   +53060     
  Branches   397960   451020   +53060     
==========================================
+ Hits       323880   373319   +49439     
+ Misses      55120    54975     -145     
- Partials    18960    22726    +3766

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rich-T-kid · 2026-10-09T17:26:10Z

run benchmark with_hashes

adriangbot · 2026-10-09T17:52:17Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T17:51:37.490166Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26157, benchmarks=with_hashes
2026-10-09T17:51:37.542178Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T17:51:37.542199Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T17:51:42.546548Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26157/head:rich-T-kid/faster-utf8view-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T17:52:02.551679Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-utf8view-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T17:52:07.554558Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T17:52:12.556294Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T19:39:33Z

run benchmark with_hashes

adriangbot · 2026-10-09T19:44:09Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T19:43:34.325269Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26157, benchmarks=with_hashes
2026-10-09T19:43:34.387762Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T19:43:34.387784Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T19:43:39.398081Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26157/head:rich-T-kid/faster-utf8view-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T19:43:54.403602Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-utf8view-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T19:43:59.405183Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T19:44:04.407244Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

github-actions Bot added the common Related to common crate label Oct 9, 2026

Rich-T-kid changed the title ~~perf: speed up Utf8View/BinaryView hashing~~ WIP string speed up (hash) Oct 9, 2026

Rich-T-kid closed this Oct 9, 2026

Navigation Menu

Repository navigation

WIP string speed up (hash) - #26157

WIP string speed up (hash)#26157
Rich-T-kid wants to merge 1 commit into
apache:mainfrom
Rich-T-kid:rich-T-kid/faster-utf8view-hash

Rich-T-kid commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

codecov-commenter commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Rich-T-kid commented Oct 9, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

codecov-commenter commented Oct 9, 2026

Codecov Report

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants