Repository navigation
Upgrade lancedb and merge insert predicates - #64
Conversation
* lance 9.0.0 * datafusion 54 * is needed for the datafusion expression support in the merge insert datafusion predicates Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR upgrades the Rust LanceDB/Lance/DataFusion dependency stack and extends the C merge-insert API to support optional “when matched update-all” predicates (either SQL or a DataFusion expression), with accompanying test and example updates.
Changes:
- Bump
lancedb/lance*/datafusion*dependencies (and lockfiles) to newer major/minor versions. - Extend
LanceDBMergeInsertConfigwith optional SQL/expression conditions and implement predicate handling inlancedb_table_merge_insert. - Add/adjust tests, examples, and valgrind suppressions to validate and accommodate the new behavior.
Reviewed changes
Copilot reviewed 10 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
valgrind.supp |
Broadens/extends suppressions for updated dependency behavior and Rust runtime allocations. |
tests/test_table.cpp |
Adds coverage for merge-insert predicates (SQL + expression) and related edge cases. |
tests/test_df_query.cpp |
Updates expectations for expression type-mismatch execution behavior. |
src/types.rs |
Extends the C-ABI merge-insert config struct with new condition fields. |
src/table.rs |
Implements merge-insert predicate extraction/validation and wiring into merge builder. |
include/lancedb.h |
Documents the new config fields and expression ownership expectations. |
examples/full.cpp |
Updates merge-insert config initialization for the new fields. |
examples/asio_coroutine.cpp |
Updates merge-insert config initialization for the new fields. |
examples/external_provider/Cargo.toml |
Updates example provider dependency version pin. |
examples/external_provider/Cargo.lock |
Regenerates lockfile for the example provider after dependency bumps. |
Cargo.toml |
Updates crate version and dependency versions/pins. |
Cargo.lock |
Regenerates lockfile after dependency upgrades. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
the condition limiting which matched records are updated can be given either as an SQL string or as a datafusion expression, using the "target." and "source." prefixes to refer to the table and to the merged data. the two flavors are mutually exclusive, and setting both of them fails with LANCEDB_INVALID_ARGUMENT. the expression is not consumed, since it is held by a config structure that the caller may reuse, and must be freed with lancedb_expr_free(). Fixes: lancedb#49 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3a27bf8 to
01f5a95
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (3)
tests/test_table.cpp:920
- This helper frees the exported Arrow arrays/schema but never frees the underlying LanceDBQuery / LanceDBQueryResult it created. This leaks memory and can cause valgrind failures; please call lancedb_query_result_free(query_result) and lancedb_query_free(query) on the success path.
lancedb_free_arrow_arrays(result_arrays, count);
lancedb_free_arrow_schema(result_schema);
return value;
tests/test_table.cpp:89
- The Arrow arrays/schema are freed, but the query and query result objects allocated earlier in this helper are never released. Per the public C API, callers must free LanceDBQuery and LanceDBQueryResult with lancedb_query_free / lancedb_query_result_free; otherwise this helper leaks memory (and can show up in valgrind runs).
This issue also appears on line 917 of the same file.
lancedb_free_arrow_arrays(result_arrays, count);
lancedb_free_arrow_schema(result_schema);
return value;
}
tests/test_df_query.cpp:683
- This section allocates a LanceDBQuery via lancedb_query_new but never frees it. Even though execution fails (query_result == nullptr), the query object still needs to be released with lancedb_query_free(query) to avoid leaking memory in the test suite.
// Should fail at execution due to the cast error
LanceDBQueryResult* query_result = lancedb_query_execute(query);
REQUIRE(query_result == nullptr);
}
No description provided.