Skip to content

Upgrade lancedb and merge insert predicates - #64

Merged
yuvalif merged 2 commits into
lancedb:mainfrom
yuvalif:upgrade-lancedb-and-merge-insert-predicates
Aug 11, 2026
Merged

yuvalif merged 2 commits into
lancedb:mainfrom
yuvalif:upgrade-lancedb-and-merge-insert-predicates

Conversation

@yuvalif

@yuvalif yuvalif commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

* lance 9.0.0
* datafusion 54
* is needed for the datafusion expression support in the merge insert
datafusion predicates

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR upgrades the Rust LanceDB/Lance/DataFusion dependency stack and extends the C merge-insert API to support optional “when matched update-all” predicates (either SQL or a DataFusion expression), with accompanying test and example updates.

Changes:

  • Bump lancedb/lance*/datafusion* dependencies (and lockfiles) to newer major/minor versions.
  • Extend LanceDBMergeInsertConfig with optional SQL/expression conditions and implement predicate handling in lancedb_table_merge_insert.
  • Add/adjust tests, examples, and valgrind suppressions to validate and accommodate the new behavior.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
valgrind.supp Broadens/extends suppressions for updated dependency behavior and Rust runtime allocations.
tests/test_table.cpp Adds coverage for merge-insert predicates (SQL + expression) and related edge cases.
tests/test_df_query.cpp Updates expectations for expression type-mismatch execution behavior.
src/types.rs Extends the C-ABI merge-insert config struct with new condition fields.
src/table.rs Implements merge-insert predicate extraction/validation and wiring into merge builder.
include/lancedb.h Documents the new config fields and expression ownership expectations.
examples/full.cpp Updates merge-insert config initialization for the new fields.
examples/asio_coroutine.cpp Updates merge-insert config initialization for the new fields.
examples/external_provider/Cargo.toml Updates example provider dependency version pin.
examples/external_provider/Cargo.lock Regenerates lockfile for the example provider after dependency bumps.
Cargo.toml Updates crate version and dependency versions/pins.
Cargo.lock Regenerates lockfile after dependency upgrades.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/table.rs Outdated
Comment thread include/lancedb.h
Comment thread include/lancedb.h
the condition limiting which matched records are updated can be given
either as an SQL string or as a datafusion expression, using the
"target." and "source." prefixes to refer to the table and to the
merged data.

the two flavors are mutually exclusive, and setting both of them fails
with LANCEDB_INVALID_ARGUMENT. the expression is not consumed, since it
is held by a config structure that the caller may reuse, and must be
freed with lancedb_expr_free().

Fixes: lancedb#49

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@yuvalif
yuvalif force-pushed the upgrade-lancedb-and-merge-insert-predicates branch from 3a27bf8 to 01f5a95 Compare August 11, 2026 06:24
@yuvalif
yuvalif requested a lite review from Copilot August 11, 2026 06:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tests/test_table.cpp:920

  • This helper frees the exported Arrow arrays/schema but never frees the underlying LanceDBQuery / LanceDBQueryResult it created. This leaks memory and can cause valgrind failures; please call lancedb_query_result_free(query_result) and lancedb_query_free(query) on the success path.
  lancedb_free_arrow_arrays(result_arrays, count);
  lancedb_free_arrow_schema(result_schema);

  return value;

tests/test_table.cpp:89

  • The Arrow arrays/schema are freed, but the query and query result objects allocated earlier in this helper are never released. Per the public C API, callers must free LanceDBQuery and LanceDBQueryResult with lancedb_query_free / lancedb_query_result_free; otherwise this helper leaks memory (and can show up in valgrind runs).

This issue also appears on line 917 of the same file.

  lancedb_free_arrow_arrays(result_arrays, count);
  lancedb_free_arrow_schema(result_schema);

  return value;
}

tests/test_df_query.cpp:683

  • This section allocates a LanceDBQuery via lancedb_query_new but never frees it. Even though execution fails (query_result == nullptr), the query object still needs to be released with lancedb_query_free(query) to avoid leaking memory in the test suite.
    // Should fail at execution due to the cast error
    LanceDBQueryResult* query_result = lancedb_query_execute(query);
    REQUIRE(query_result == nullptr);
  }

@yuvalif
yuvalif merged commit aa6f0a0 into lancedb:main Aug 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants