Why not use OpenAPI and CRDs as opposed to a new IDL?

[jlewi]

It looks like you are defining a new way to describe schemas using YAML; i.e. creating a new interface descriptor language (IDL).

e.g. job.yaml is actually describing a job resource.

Why not use OpenAPI to describe resources? Additionally you could wrap it in Kubernetes CRD.

Even if you don't use Kubernetes you can still use the Kubernetes resource model and use CRD's as a familiar way to describe resources.

[aronchick]

No objection! However, I had a goal of letting end users be able to define APIs incredibly simply - so, an IT department can define a schema with the following:

• name
• type
• required
• empty

And with that I can give an enormous amount of tooling - validation, null checking, simple lambdas, regex matching, lists, etc.

I don't mind using literally any other format - and it should be pretty straightforward to do so. But OpenAPI (for example) just felt quite large and mostly targeted at live endpoints.

Let's get to brass tacks though :) Here's the default schema i used to build the library (https://github.com/mlspec/mlspec-lib/blob/master/mlspeclib/schemas/0/0/1/datapath.yaml) - if we were to do this in OpenAPI, what would it look like?

mlspec_base_type:
  # Base schema type that this extends
  meta: base

mlspec_schema_version:
  # Identifies the version of this schema
  meta: 0.0.1

mlspec_schema_type:
  # Identifies the type of this schema
  meta: datapath

schema_version:
  # Identifies version of MLSpec to use to instantiate
  type: semver
  required: True

schema_type:
  # Identifies type of MLSpec to use to instantiate
  type: allowed_schema_types
  required: True

data_store:
  # Identifies name of datastore
  type: string
  required: True
  empty: False

storage_connection_type:
  # Type of storage for the datastore, or CUSTOM for not present
  type: string
  required: True
  allowed:
    - "CUSTOM" # Custom connection
    - "AWS_BLOB" # AWS Blob
    - "GCP_BLOB" # Google Cloud Blob
    - "AZURE_BLOB" # Azure Blob Storage
    - "NFS_BLOB" # NFS Blob Storage
    - "SMB_BLOB" # Samba Blob Storage

# Connection to datapath
connection:
  type: nested
  schema:
    endpoint:
      # URI for the location of the data store
      type: URI
      required: True

    access_key_id:
      # AWS access key (NOT RECOMMENDED - Use secret storage to provide connection)
      type: string
      regex: (?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9])

    secret_access_key:
      # AWS access key (NOT RECOMMENDED - Use secret storage to provide connection)
      type: string
      regex: (?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])