test: add it for produce & scan log for all supported datatypes #205
Conversation
|
@luoyuxia Hi, can you take a look? |
|
@fresh-borzoni Could you please also help review this? |
There was a problem hiding this comment.
Pull request overview
This PR adds an end-to-end integration test for producing and scanning log tables across all supported data types and fixes how Arrow temporal types and CHAR are read during log table deserialization.
Changes:
- Add
all_supported_datatypesintegration test to validate append/scan behavior across integers, floats, booleans, strings, decimals, dates, times (various precisions), timestamps (NTZ/LTZ with multiple precisions), bytes, and nulls. - Update
ColumnarRowto read ArrowTimestamp,Date, andTimecolumns via their concrete Arrow array types, correctly converting to Fluss temporal representations. - Enhance
get_charto handle bothFixedSizeBinary(KV table) andUtf8(log table) storage formats.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| crates/fluss/tests/integration/table.rs | Adds all_supported_datatypes IT that creates a log table with all supported data types, appends one fully-populated row plus a fully-null row, scans via the log scanner, and asserts correct round-trip values for each column and null handling. |
| crates/fluss/src/row/column.rs | Fixes deserialization of Arrow Timestamp, Date32, and Time32/Time64 into Fluss TimestampNtz, TimestampLtz, Date, and Time, and extends get_char to support both FixedSizeBinary and Utf8 representations, improving compatibility with log tables. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@linguoxuan Thank you for the PR.
LGTM
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@linguoxuan Thanks for the pr. LGTM. I only have one question. PTAL
| std::str::from_utf8(bytes).expect("Invalid UTF-8 in char field") | ||
| match arrow_field.data_type() { | ||
| ArrowDataType::FixedSizeBinary(_) => { | ||
| // KV table format: char stored as FixedSizeBinary |
There was a problem hiding this comment.
hi, what does it mean? Why said KV table format stored as FixedSizeBinary. Do you mean change log for kv table format?
Shouldn't KV table format and Log table format use same tye for arrow change?
3 participants
Purpose
Linked issue: close #199
Brief change log
Tests
API and Format
Documentation