Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish_snapshot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
with:
persist-credentials: false

- uses: actions/setup-java@dd06d9cba3e5552c54d9f8ea23572deb30010f7c # v6.0.0
- uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1
with:
distribution: temurin
java-version: 17
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
with:
persist-credentials: false
- name: Set up JDK ${{ matrix.java }}
uses: actions/setup-java@dd06d9cba3e5552c54d9f8ea23572deb30010f7c # v6.0.0
uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1
with:
java-version: ${{ matrix.java }}
distribution: temurin
Expand Down
42 changes: 42 additions & 0 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -1097,6 +1097,48 @@ optional group my_map (MAP_KEY_VALUE) {
}
```

### Finite Vectors

`FINITE_VECTOR` is used to annotate fixed-length ordered sequences of finite and
non-null elements.

`FiniteVectorType` annotation has one required parameter, `num_elements`,
which is the number of elements in each non-null vector and MUST be greater
than zero.

`FINITE_VECTOR` MUST always annotate the canonical 3-level structure:

```
<vector-repetition> group <name> (FINITE_VECTOR(<num_elements>)) {
repeated group list {
required <element-type> element;
}
}
```

The element types supported are:

* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
`DECIMAL`
* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.

Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`,
this excludes NaN, positive infinity, and negative infinity.

For example, a nullable vector containing 768 required `FLOAT` elements is:

```
optional group embedding (FINITE_VECTOR(768)) {
repeated group list {
required float element;
}
}
```

Every non-null vector must contain exactly `num_elements` elements.
Because `num_elements` is greater than zero, a non-null vector cannot be empty.

## UNKNOWN (always null)

Sometimes, when discovering the schema of existing data, values are always null
Expand Down
2 changes: 1 addition & 1 deletion dev/install-thrift.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

set -euo pipefail

THRIFT_VERSION="${THRIFT_VERSION:-0.24.0}"
THRIFT_VERSION="${THRIFT_VERSION:-0.25.0}"

sudo apt-get update -qq
sudo apt-get install -qq protobuf-compiler
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
<maven.compiler.target>1.8</maven.compiler.target>
<shade.prefix>shaded.parquet</shade.prefix>
<thrift.executable>thrift</thrift.executable>
<thrift.version>0.24.0</thrift.version>
<thrift.version>0.25.0</thrift.version>
<thrift-maven-plugin.version>0.10.0</thrift-maven-plugin.version>
</properties>

Expand Down Expand Up @@ -243,7 +243,7 @@
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.5.1</version>
<version>3.6.4</version>
<executions>
<execution>
<id>check-thrift-version</id>
Expand Down
15 changes: 15 additions & 0 deletions src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,19 @@ struct GeographyType {
struct FileType {
}

/**
* Finite fixed-length vector logical type annotation
*
* Annotates the outer group of a three-level structure that is the same as
* the one used for LIST-annotated groups. Each non-null vector MUST contain
* exactly num_elements elements.
*
* See LogicalTypes.md for details.
*/
struct FiniteVectorType {
1: required i32 num_elements
}

/**
* LogicalType annotations to replace ConvertedType.
*
Expand Down Expand Up @@ -513,6 +526,7 @@ union LogicalType {
17: GeometryType GEOMETRY // no compatible ConvertedType
18: GeographyType GEOGRAPHY // no compatible ConvertedType
19: FileType FILE // no compatible ConvertedType
20: FiniteVectorType FINITE_VECTOR // use ConvertedType LIST
}

/**
Expand Down Expand Up @@ -1129,6 +1143,7 @@ union ColumnOrder {
* GEOMETRY - undefined
* GEOGRAPHY - undefined
* FILE - undefined
* FINITE_VECTOR - undefined
*
* In the absence of logical types, the sort order is determined by the physical type:
* BOOLEAN - false, true
Expand Down
Loading