Skip to content

GH-430: Add VECTOR logical type — LIST storage (Option C1) - #3

Open
rok wants to merge 19 commits into
masterfrom
vector-option-c1-list
Open

rok wants to merge 19 commits into
masterfrom
vector-option-c1-list

Conversation

@rok

@rok rok commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

Option C1

Adds a VECTOR logical type for fixed-length vectors (embeddings, Arrow FixedSizeList), stored as the standard 3-level LIST structure.

optional group embedding (VECTOR(768)) {
  repeated group list {
    required float element;
  }
}
  • VectorType carries num_elements (> 0). Every non-null vector has exactly that many elements.
  • Elements are required and finite (no NaN or ±Inf). The vector itself may be nullable.
  • Element types: BOOLEAN, INT32/INT64 (plain, INT or DECIMAL), FLOAT, DOUBLE, FLOAT16 and FLBA DECIMAL. No nested elements.
  • Writers must enforce these constraints. Readers may rely on them, and must treat data that violates them as invalid if they detect it.
  • No new encoding or physical layout. Levels, pages, encodings, statistics and Bloom filters are those of the element column. nan_count / nan_counts must be zero when written.
  • Writers also set ConvertedType.LIST, so readers that don't know VECTOR read an ordinary LIST.

Trade-off: existing readers and element-level encodings keep working, but repetition/definition levels are still written and the fixed length is a contract rather than structural.

This branch is independent of the packed-FLBA Option A proposal.

@rok
rok force-pushed the vector-option-c1-list branch 3 times, most recently from 4b40d63 to 813d41d Compare September 21, 2026 22:48
@rok rok changed the title GH-430: VECTOR Option C1 — LIST storage GH-430: Add VECTOR logical type — LIST storage (Option C1) Sep 21, 2026
@rok
rok force-pushed the vector-option-c1-list branch from 813d41d to 36ce552 Compare September 22, 2026 12:30
dependabot Bot and others added 9 commits October 11, 2026 23:44
* Bump org.apache.thrift:libthrift from 0.24.0 to 0.25.0

Bumps [org.apache.thrift:libthrift](https://github.com/apache/thrift) from 0.24.0 to 0.25.0.
- [Release notes](https://github.com/apache/thrift/releases)
- [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)
- [Commits](apache/thrift@v0.24.0...v0.25.0)

---
updated-dependencies:
- dependency-name: org.apache.thrift:libthrift
  dependency-version: 0.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump Thrift compiler to 0.25.0 in CI install script

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
)

Bumps [org.codehaus.mojo:exec-maven-plugin](https://github.com/mojohaus/exec-maven-plugin) from 3.5.1 to 3.6.4.
- [Release notes](https://github.com/mojohaus/exec-maven-plugin/releases)
- [Commits](mojohaus/exec-maven-plugin@3.5.1...3.6.4)

---
updated-dependencies:
- dependency-name: org.codehaus.mojo:exec-maven-plugin
  dependency-version: 3.6.4
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 6.0.0 to 6.0.1.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](actions/setup-java@dd06d9c...de7274f)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
@rok
rok force-pushed the vector-option-c1-list branch from ac3438d to 2bdc543 Compare October 30, 2026 11:37
@rok
rok force-pushed the vector-option-c1-list branch from 268ab40 to daac474 Compare October 30, 2026 11:52
@rok
rok force-pushed the vector-option-c1-list branch from 97fb091 to 6c9f73e Compare October 30, 2026 11:56
@rok
rok force-pushed the vector-option-c1-list branch 2 times, most recently from 7c67461 to 81bc323 Compare November 4, 2026 18:38
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
@rok
rok force-pushed the vector-option-c1-list branch from 81bc323 to 70b3023 Compare November 4, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant