Describe the bug
remap_physical_schema in native/core/src/parquet/schema_adapter.rs gates the ParquetMissingFieldIds rejection on schema_has_field_ids, which looks at root fields only and runs only when spark.sql.parquet.fieldId.read.enabled is set. Spark's ParquetReadSupport.getRequestedSchema uses containsFieldIds on the file schema and ParquetUtils.hasFieldIds on the requested schema, both recursive, and consults neither flag.
Two observable differences follow, both reproducible on main:
- With the default
fieldId.read.enabled=false, a requested schema carrying parquet.field.id metadata read against a file with no ids returns rows in Comet where Spark raises.
- A file whose ids sit only on nested fields is rejected by Comet where Spark null-fills the missing nested ids through
matchIdField.
Steps to reproduce
- Write a Parquet file with no field ids and read it with a requested schema whose fields carry
parquet.field.id, native scan on, fieldId.read.enabled at its default. Comet returns rows; Spark raises.
- Write a Parquet file whose ids are only on struct children and read it with field id matching enabled and a requested schema that carries ids at the root. Comet rejects the file; Spark reads it and null-fills the unmatched nested children.
Expected behavior
The gate matches Spark: recursive on both schemas and independent of fieldId.read.enabled.
Additional context
#5654 adds the recursive predicate any_nested_field_has_id one module away from the root-only schema_has_field_ids that makes this decision, and documents the root-only one as deliberate for now. #5786 adds a footer-time validator on the same call site. Neither changes the gate; this issue tracks the gate itself, as asked for in the #5654 review. Nested duplicate resolution is tracked separately in #5884.
Describe the bug
remap_physical_schemainnative/core/src/parquet/schema_adapter.rsgates theParquetMissingFieldIdsrejection onschema_has_field_ids, which looks at root fields only and runs only whenspark.sql.parquet.fieldId.read.enabledis set. Spark'sParquetReadSupport.getRequestedSchemausescontainsFieldIdson the file schema andParquetUtils.hasFieldIdson the requested schema, both recursive, and consults neither flag.Two observable differences follow, both reproducible on
main:fieldId.read.enabled=false, a requested schema carryingparquet.field.idmetadata read against a file with no ids returns rows in Comet where Spark raises.matchIdField.Steps to reproduce
parquet.field.id, native scan on,fieldId.read.enabledat its default. Comet returns rows; Spark raises.Expected behavior
The gate matches Spark: recursive on both schemas and independent of
fieldId.read.enabled.Additional context
#5654 adds the recursive predicate
any_nested_field_has_idone module away from the root-onlyschema_has_field_idsthat makes this decision, and documents the root-only one as deliberate for now. #5786 adds a footer-time validator on the same call site. Neither changes the gate; this issue tracks the gate itself, as asked for in the #5654 review. Nested duplicate resolution is tracked separately in #5884.