Skip to content

Added support of ApacheArrow batches for BulkUpsert #593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alex268 merged 9 commits into from
Feb 20, 2026
Merged

Added support of ApacheArrow batches for BulkUpsert #593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2c2278c
Switch BulkUpsertDate from class to interface
alex268 Feb 18, 2026
b53e483
Added impelementations for CSV and ApacheArrow formats
alex268 Feb 18, 2026
fb98b87
Added ApacheArrowWriter helper to build arrow batches
alex268 Feb 18, 2026
d9e31c7
Added tests for BulkUpsert with validation
alex268 Feb 18, 2026
96c01e8
Fixed JDK8 compability
alex268 Feb 18, 2026
81ccc5f
Fixes by Copilot
alex268 Feb 18, 2026
7c18140
Added small test for data validation
alex268 Feb 18, 2026
8624d3e
Removed BulkUpsertCsvData
alex268 Feb 19, 2026
ca0d8cb
Merge remote-tracking branch 'origin/release_v2.4.0' into release_v2.4.0
alex268 Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,9 @@
<log4j2.version>2.25.3</log4j2.version>
<gson.version>2.13.2</gson.version>
<mockito.version>5.21.0</mockito.version>

<testcontainers.version>2.0.3</testcontainers.version>
<!-- last version with JDK8 compability -->
<apache.arrow.version>17.0.0</apache.arrow.version>
</properties>

<licenses>
Expand Down Expand Up @@ -88,6 +89,17 @@
<version>${gson.version}</version>
</dependency>

<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
<version>${apache.arrow.version}</version>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-netty</artifactId>
<version>${apache.arrow.version}</version>
</dependency>

<!-- Test dependencies -->
<dependency>
<groupId>junit</groupId>
Expand Down Expand Up @@ -238,11 +250,14 @@
<profile>
<id>jdk8-bootstrap</id>
<activation>
<jdk>[9</jdk>
<jdk>[9,)</jdk>
<property>
<name>argLine</name>
</property>
</activation>
<properties>
<argLine></argLine>
</properties>
<build>
<plugins>
<plugin>
Expand Down
27 changes: 25 additions & 2 deletions table/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,26 @@
</dependency>

<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-netty</artifactId>
<optional>true</optional>
</dependency>

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>tech.ydb.test</groupId>
<artifactId>ydb-junit4-support</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
Expand All @@ -57,9 +66,23 @@
<environmentVariables>
<TESTCONTAINERS_REUSE_ENABLE>true</TESTCONTAINERS_REUSE_ENABLE>
<YDB_DOCKER_IMAGE>ydbplatform/local-ydb:trunk</YDB_DOCKER_IMAGE>
<YDB_DOCKER_FEATURE_FLAGS>enable_columnshard_bool</YDB_DOCKER_FEATURE_FLAGS>
</environmentVariables>
</configuration>
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>jdk8-bootstrap</id>
<activation>
<jdk>[9,)</jdk>
</activation>
<properties>
<!-- Hide warnings on JDK later 8 -->
<argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine>
</properties>
</profile>
</profiles>
</project>
2 changes: 1 addition & 1 deletion table/src/main/java/tech/ydb/table/Session.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ default CompletableFuture<Status> executeScanQuery(String query, Params params,
CompletableFuture<Result<State>> keepAlive(KeepAliveSessionSettings settings);

default CompletableFuture<Status> executeBulkUpsert(String tablePath, ListValue rows, BulkUpsertSettings settings) {
return executeBulkUpsert(tablePath, new BulkUpsertData(rows), settings);
return executeBulkUpsert(tablePath, BulkUpsertData.fromRows(rows), settings);
}

CompletableFuture<Status> executeBulkUpsert(String tablePath, BulkUpsertData data, BulkUpsertSettings settings);
Expand Down
Loading
Loading