You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
core: proto import types an outer FieldReference as the whole enclosing row instead of the referenced field #1372
On proto → POJO import, an outer FieldReference is typed as the whole enclosing row, not as the field it selects. ProtoExpressionConverter (core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java:101 and :106) passes the scope from outerScopeForStepsOut / outerScopeForRelReference straight in as the knownType of newRootStructOuterReference / newRootStructOuterReferenceByRelReference. Root references don't have this problem: they go through StructField.constructOnRoot, which types the reference as struct.fields().get(offset). FieldReference.type() is documented as "the type of the referenced field".
With an enclosing row (name VARCHAR, s ROW(v INT)) and a reference to s:
FieldReferenceref = FieldReference.newRootStructOuterReference(1, R.struct(R.I32), 1);
FieldReferenceback = (FieldReference) converterWithRoot.from(expressionProtoConverter.toProto(ref));
back.getType(); // Struct{Str, Struct{I32}}, the whole row, not Struct{I32}ref.equals(back); // falseback.dereferenceStruct(0); // segments [SF(0), SF(1)] (s.v, correct), type Str (name's type)
Consequences:
A POJO built from SQL (typed with the field type) never equals its proto round trip, so assertProtoPlanRoundrip / assertFullRoundTrip cannot be used on any correlated SQL, and the existing outer-reference round-trip tests declare the whole row as the reference type to match the import.
getType() on an imported outer reference is wrong for any consumer that reads it, and since fix(core): retain reference scope during dereference #1268 made dereference* keep the outer scope, dereferencing one yields a correctly scoped reference with the wrong type, or throws for list/map dereferences. No in-repo path dereferences an imported outer reference today, but anyone adding nested outer-path support to the reader (rejected since fix(core): retain reference scope during dereference #1268) will hit this first.
The current behavior predates #1034, which corrected which scope is used but kept typing the reference as that scope. Was the whole-row type intended? If not, the fix is to type the reference as scope.fields().get(field) with the same bounds check as constructOnRoot, and to update the round-trip tests that encode the whole-row type. That changes what getType() returns on import, so it likely wants a fix(core)!. Related: #1095 (unvalidated knownType on the root-reference factory).
On proto → POJO import, an outer
FieldReferenceis typed as the whole enclosing row, not as the field it selects.ProtoExpressionConverter(core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java:101and:106) passes the scope fromouterScopeForStepsOut/outerScopeForRelReferencestraight in as theknownTypeofnewRootStructOuterReference/newRootStructOuterReferenceByRelReference. Root references don't have this problem: they go throughStructField.constructOnRoot, which types the reference asstruct.fields().get(offset).FieldReference.type()is documented as "the type of the referenced field".With an enclosing row
(name VARCHAR, s ROW(v INT))and a reference tos:Consequences:
assertProtoPlanRoundrip/assertFullRoundTripcannot be used on any correlated SQL, and the existing outer-reference round-trip tests declare the whole row as the reference type to match the import.getType()on an imported outer reference is wrong for any consumer that reads it, and since fix(core): retain reference scope during dereference #1268 madedereference*keep the outer scope, dereferencing one yields a correctly scoped reference with the wrong type, or throws for list/map dereferences. No in-repo path dereferences an imported outer reference today, but anyone adding nested outer-path support to the reader (rejected since fix(core): retain reference scope during dereference #1268) will hit this first.The current behavior predates #1034, which corrected which scope is used but kept typing the reference as that scope. Was the whole-row type intended? If not, the fix is to type the reference as
scope.fields().get(field)with the same bounds check asconstructOnRoot, and to update the round-trip tests that encode the whole-row type. That changes whatgetType()returns on import, so it likely wants afix(core)!. Related: #1095 (unvalidatedknownTypeon the root-reference factory).Measured on
mainat ca8315b.