Repository navigation
Filter evaluation perf - #2721
Merged
Merged
Conversation
- Parquet answers pushed conjuncts exactly, so Flow no longer re-filters them in PHP - pushed conjuncts run before the ones kept in PHP; offset is pushed into Parquet - float > / >= pushdown keeps NaN rows; partition filters handle files without the partition - native comparison, null checks and arithmetic in flow_php behind new Backend methods - Rust Parquet reads fill batches across row groups and decode scattered selections faster - prose docblocks in IO/Extractor/File replaced by tests that state each rule
…ster arrow-only Parquet - native Backend::sort (stable, PHP order) for memory, external, window and top-n sorts; k-way merge sorts blocks natively - external sort spills in batch-size frames and releases its buffer, so peak PHP memory no longer grows with the data - group by assigns dense group ids (GroupTable, native RustGroupTable) and aggregates each batch per column (GroupedAggregation); sum, avg and count run columnar, every other aggregation through PerGroupAggregation - sum and avg of one column share one running total; avg divides on floats when the result provably equals the decimal one - parallel partial aggregates take the same grouped path as in-process aggregation - a filter evaluates each conjunct of an AND on its own, natively where it can - the optimizer pushes the columns a plan outputs into file sources; Parquet decodes filter-only columns only where statistics leave the filter open, and count() reads no column - arrow-ext skips the row filter on row groups whose statistics prove every row matches - Parquet readers and writers exchange stored values (readStoredColumns/writeStoredColumns): dates, timestamps, times and uuids as their physicals; the Parquet value converters and ParquetEncoder are removed - arrow-ext builds datetimes without a string parse on verified PHP layouts; arrow-only reads fill batches across row groups - Floe files are read in exact windows, without buffer concatenation; Floe frames are validated once - Aggregators::aggregate() and merge() are removed - partitioned reads with metadata columns no longer declare _input_file_uri twice - Sort values moved to Flow\ETL\Values\Sort and KeyEncoder to Flow\ETL\Values\Grouping
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2721 +/- ##
============================================
+ Coverage 87.00% 90.56% +3.55%
- Complexity 2232 31336 +29104
============================================
Files 2197 2185 -12
Lines 93901 90474 -3427
============================================
+ Hits 81703 81941 +238
+ Misses 12198 8533 -3665
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #xxx
Change Log
Added
flow-php/etl- Native sort, group ids and grouped sum/avg/count kernels behind Backendflow-php/etl- Projection pushdown of the columns a plan outputs into file sourcesflow-php/parquet- readStoredColumns() and writeStoredColumns() exchanging dates, timestamps, times and uuids as stored valuesflow-php/flow-php-ext- RustGroupTable, native sort and grouped totals/countsflow-php/arrow-ext- Stored column reads and writesFixed
flow-php/etl-adapter-parquet- Partitioned reads with metadata columns declared _input_file_uri twiceChanged
flow-php/etl- Filters evaluate each AND conjunct on its ownflow-php/etl- count() reads no columnsflow-php/etl- Sum and avg of one column share one running totalflow-php/etl- External sort spills in batch-size frames and releases its bufferflow-php/etl- Parallel partial aggregates take the grouped pathflow-php/etl- Floe files are read in exact windowsflow-php/etl- Sort values moved to Values\Sort and KeyEncoder to Values\Groupingflow-php/etl-adapter-parquet- Filter-only columns are decoded only where statistics leave the filter openflow-php/etl-adapter-parquet- Reads and writes stored values instead of PHP objectsflow-php/arrow-ext- Skips the row filter on row groups whose statistics prove every row matchesflow-php/arrow-ext- Builds datetimes without a string parse on verified PHP layoutsflow-php/arrow-ext- Out-of-range values are refused with InvalidArgumentExceptionflow-php/flow-php-ext- Floe frames are validated onceRemoved
flow-php/etl- Aggregators::aggregate() and Aggregators::merge()flow-php/etl- Group and GroupKeyflow-php/etl-adapter-parquet- ParquetEncoder and the value convertersDeprecated
Security