Describe the bug
unshred_variant still panics when a present shredded object field has no entry in the row's metadata dictionary. VariantArray::try_new accepts the array, but the read-only metadata builder later returns an error that ObjectBuilder::insert unwraps.
Reproduced on arrow-rs 4cd8be954f6bc6b6dd265140207365b59a9900ec, after #9741. This is malformed input under the current Variant metadata requirement, which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data.
To Reproduce
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray, Int32Array, StructArray};
use arrow::datatypes::Field;
use parquet_variant_compute::{VariantArray, unshred_variant};
fn structure(fields: Vec<(&str, ArrayRef, bool)>) -> StructArray {
let schema = fields.iter().map(|(name, array, nullable)|
Arc::new(Field::new(*name, array.data_type().clone(), *nullable)))
.collect::<Vec<_>>();
StructArray::new(schema.into(), fields.into_iter().map(|(_, a, _)| a).collect(), None)
}
fn main() {
let missing: ArrayRef = Arc::new(BinaryArray::from(vec![None::<&[u8]>]));
let a: ArrayRef = Arc::new(structure(vec![
("value", missing.clone(), true),
("typed_value", Arc::new(Int32Array::from(vec![1])), true),
]));
let typed: ArrayRef = Arc::new(structure(vec![("a", a, false)]));
let input = structure(vec![
("metadata", Arc::new(BinaryArray::from_vec(vec![&[1, 0, 0]])), false),
("value", missing, true),
("typed_value", typed, true),
]);
let input = VariantArray::try_new(&input).unwrap();
let _ = unshred_variant(&input); // panics instead of returning Err
}
Panic: called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary") at parquet-variant/src/builder/object.rs:104.
Expected behavior
Return ArrowError for a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix.
Additional context
Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently extends metadata and remaps residual field IDs before unshredding in apache/datafusion-comet#5868.
Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior.
Describe the bug
unshred_variantstill panics when a present shredded object field has no entry in the row's metadata dictionary.VariantArray::try_newaccepts the array, but the read-only metadata builder later returns an error thatObjectBuilder::insertunwraps.Reproduced on arrow-rs
4cd8be954f6bc6b6dd265140207365b59a9900ec, after #9741. This is malformed input under the current Variant metadata requirement, which requires shredded field names in metadata. The bug is the panic from a fallible kernel consuming file data.To Reproduce
Panic:
called Result::unwrap() on an Err value: InvalidArgumentError("Field name 'a' not found in metadata dictionary")atparquet-variant/src/builder/object.rs:104.Expected behavior
Return
ArrowErrorfor a present field whose name is absent from metadata, including nested objects and list elements. Absent object fields and rows masked by parent nulls should not trigger this error. Supporting permissive metadata repair would be a separate compatibility choice, not a requirement of this bug fix.Additional context
Related prior panic report: #9740 / #9741. Comet encounters this through Spark-compatible input handling and currently extends metadata and remaps residual field IDs before unshredding in apache/datafusion-comet#5868.
Downstream tracking: apache/datafusion-comet#5477. Returning an error fixes the Arrow panic but does not by itself replace Comet's permissive Spark compatibility behavior.