Skip to content

Field id gating differs from Spark: root-only check and no dependence on fieldId.read.enabled #5936

Description

@dwsmith1983

Describe the bug

remap_physical_schema in native/core/src/parquet/schema_adapter.rs gates the ParquetMissingFieldIds rejection on schema_has_field_ids, which looks at root fields only and runs only when spark.sql.parquet.fieldId.read.enabled is set. Spark's ParquetReadSupport.getRequestedSchema uses containsFieldIds on the file schema and ParquetUtils.hasFieldIds on the requested schema, both recursive, and consults neither flag.

Two observable differences follow, both reproducible on main:

  1. With the default fieldId.read.enabled=false, a requested schema carrying parquet.field.id metadata read against a file with no ids returns rows in Comet where Spark raises.
  2. A file whose ids sit only on nested fields is rejected by Comet where Spark null-fills the missing nested ids through matchIdField.

Steps to reproduce

  1. Write a Parquet file with no field ids and read it with a requested schema whose fields carry parquet.field.id, native scan on, fieldId.read.enabled at its default. Comet returns rows; Spark raises.
  2. Write a Parquet file whose ids are only on struct children and read it with field id matching enabled and a requested schema that carries ids at the root. Comet rejects the file; Spark reads it and null-fills the unmatched nested children.

Expected behavior

The gate matches Spark: recursive on both schemas and independent of fieldId.read.enabled.

Additional context

#5654 adds the recursive predicate any_nested_field_has_id one module away from the root-only schema_has_field_ids that makes this decision, and documents the root-only one as deliberate for now. #5786 adds a footer-time validator on the same call site. Neither changes the gate; this issue tracks the gate itself, as asked for in the #5654 review. Nested duplicate resolution is tracked separately in #5884.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions