What features do we need to reach 1.0

[westonpace]

Substrait has been in use in a number of situations for some time now. It is quite stable. We already go somewhat out of our way to avoid breaking changes. Substrait has good set of the most common logical relations (e.g. enough to express TPC-H and maybe TPC-DS?). I think we are near what we would want for a 1.0 release, and making one would help communicate these facts.

I will start an email with questions like "do we want a 1.0 release now?" and "how should we tactically execute such a release?". Those questions are not the purpose of this discussion.

The purpose of this discussion is to identify the features the we think are missing which would block a 1.0 release. Ideally, we can structure this in a way that each new thread is a separate feature that we think should be added and then that thread's discussion can argue the merits and details. Eventually we will want to boil this into an issue / checklist of some kind.

[westonpace]

Collation & Precise definitions for ordering

[westonpace]

More precise definitions of casting

[EpsilonPrime]

Desirable:

• Standardized complex data type functions (for map, struct, list)
• Normalized protobuf usage (join types, field numbers, enum vs bool)
  • Organize types into basic, compound, and parameterized sections.
  • Move the oneof (functionality switch) to the top of the each relation/expression for consistency.
• Standardized documentation style in protobuf file.
• Remove all deprecated features.
• Consider moving enums (especially common ones) outside of message definitions.
• Text format updated and available for use.
• Validator updated.
• PrecisionTimestampTz alternative (with timezone), fix names
• Definition of query endpoint
• Engine compatibility definition (grouped YAMLs) -- also applicable to other categories

Not needed:

• Runtime constraints
• Unfork Gluten
• Streaming features?
• Iceberg writes
• Clean documentation (this is probably better done as a post-1.0 task)
• Move Go code into a subdirectory to clean up the top level of the core repository

[vbarua]

Per our discussion at the last sync, I've created a 1.0 issue label to make it easier to track and discuss items we want for our 1.0 release.

[tokoko]

I think we should add defining the role and usage of ExtendedExpression messages (as discussed in #794) to 1.0 milestone as well. I'll create a separate issue for it if you agree.