Skip to content

core: proto import types an outer FieldReference as the whole enclosing row instead of the referenced field #1372

Description

@nielspardon

On proto → POJO import, an outer FieldReference is typed as the whole enclosing row, not as the field it selects. ProtoExpressionConverter (core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java:101 and :106) passes the scope from outerScopeForStepsOut / outerScopeForRelReference straight in as the knownType of newRootStructOuterReference / newRootStructOuterReferenceByRelReference. Root references don't have this problem: they go through StructField.constructOnRoot, which types the reference as struct.fields().get(offset). FieldReference.type() is documented as "the type of the referenced field".

With an enclosing row (name VARCHAR, s ROW(v INT)) and a reference to s:

FieldReference ref = FieldReference.newRootStructOuterReference(1, R.struct(R.I32), 1);
FieldReference back = (FieldReference) converterWithRoot.from(expressionProtoConverter.toProto(ref));
back.getType();                // Struct{Str, Struct{I32}}, the whole row, not Struct{I32}
ref.equals(back);              // false
back.dereferenceStruct(0);     // segments [SF(0), SF(1)] (s.v, correct), type Str (name's type)

Consequences:

  • A POJO built from SQL (typed with the field type) never equals its proto round trip, so assertProtoPlanRoundrip / assertFullRoundTrip cannot be used on any correlated SQL, and the existing outer-reference round-trip tests declare the whole row as the reference type to match the import.
  • getType() on an imported outer reference is wrong for any consumer that reads it, and since fix(core): retain reference scope during dereference #1268 made dereference* keep the outer scope, dereferencing one yields a correctly scoped reference with the wrong type, or throws for list/map dereferences. No in-repo path dereferences an imported outer reference today, but anyone adding nested outer-path support to the reader (rejected since fix(core): retain reference scope during dereference #1268) will hit this first.

The current behavior predates #1034, which corrected which scope is used but kept typing the reference as that scope. Was the whole-row type intended? If not, the fix is to type the reference as scope.fields().get(field) with the same bounds check as constructOnRoot, and to update the round-trip tests that encode the whole-row type. That changes what getType() returns on import, so it likely wants a fix(core)!. Related: #1095 (unvalidated knownType on the root-reference factory).

Measured on main at ca8315b.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions