Skip to content

[FEATURE] Add Apache Druid provider (Druid SQL over HTTP, no native dependency) #265

Description

@cevheri

Summary

Add Apache Druid as a database provider, over its SQL HTTP API, with no runtime dependency
the same shape as the Couchbase provider shipped in #262 (PR #263).

Druid is a real-time analytics database. It fits the driver-free pattern well, and unlike most
candidates its SQL surface, its catalog and its monitoring data all arrive through the same
endpoint.

Why Druid fits

Measured against the seven questions that decided #262:

Druid
HTTP a first-class interface Yes — POST /druid/v2/sql on the Router (8888) or Broker (8082). There is no binary client protocol to miss out on
SQL-shaped query language Yes, Druid SQL (Calcite-based)
Catalog introspection INFORMATION_SCHEMA.SCHEMATA / TABLES / COLUMNS — standard, declared, no inference needed
Monitoring The sys schema: sys.segments, sys.servers, sys.tasks, sys.supervisors
EXPLAIN EXPLAIN PLAN FOR <query> — but see the caveat below, this one does not fit our existing render model
Auth complexity None by default; optional basic-auth extension
Data model flattens into TableSchema Yes — a datasource is a table, and INFORMATION_SCHEMA.COLUMNS gives real column types

One genuinely nice property: Druid returns errors with real HTTP status codes (400, 500) and a
JSON body carrying error, errorMessage, errorClass and host. Couchbase and Trino both bury
failures inside a 200 response, which was the single most dangerous behaviour in #262. Druid does
not have that trap.

The EXPLAIN caveat, stated up front

EXPLAIN PLAN FOR does not return an operator tree. It returns a PLAN column containing a
JSON array of the native Druid queries the SQL was translated into. That is a translation view,
not an execution plan, and it will not map onto the existing { kind: "tree" } render model that
Couchbase and SQLite share.

Three honest options, to be decided during implementation:

  1. Render the native-query translation as a read-only JSON view and set supportsExplain: true with
    a render model that says what it actually is.
  2. Set supportsExplain: false for the first PR, so the button and tab stay hidden, and add explain
    support in a follow-up with its own render model.
  3. Extend ExplainPlanInput with a third variant for translation-style plans, which would also serve
    Elasticsearch's _sql/translate later.

Option 2 is the smallest honest first step; option 3 is the most valuable if a second
translation-style engine is ever added. Do not force the translation into the tree model — that
is exactly the kind of dishonest capability #194 and #201 were about.

Note also the interaction with use-query-execution.ts:165, which builds the direct Explain action
with mode analyze: whatever is chosen, buildSql must not return null for analyze while
supportsExplain is true, or the button is dead. That defect was found in review on #263.

Design shape

Follows #262 unless a live check says otherwise: HTTP transport only with a seam and an envelope
guard test, a neutral result type, queryLanguage: "sql", monitoring that degrades to empty, and
tri-sync across code, docs/providers/druid.md and
tests/integration/db/druid-provider.test.ts.

Request body fields worth knowing: query, resultFormat (object, array, objectLines,
arrayLines, csv), header, typesHeader, sqlTypesHeader, context, and parameters.
Setting header: true with sqlTypesHeader: true returns column names and SQL types in the
response itself, so the field list does not have to be derived from the rows the way Couchbase's
wildcard signature forced.

Adding a new Provider

Verify against a live instance before writing code

Stand up the Druid quickstart and confirm each of these, because each changes the implementation:

  • Which port the provider should target by default: the Router (8888) fronts everything, the
    Broker (8082) serves queries directly. Router is the friendlier default; confirm both work.
  • Whether resultFormat: "object" plus sqlTypesHeader gives everything needed for
    fields and ColumnSchema.type, or whether INFORMATION_SCHEMA.COLUMNS is still required.
  • What EXPLAIN PLAN FOR actually returns on the installed version, to decide between the three
    options above with real output in hand rather than from documentation.
  • Whether sys.* tables require any permission that a default install withholds, and what the
    degradation path looks like.
  • How a datasource with no segments, or an in-flight ingestion task, appears in
    INFORMATION_SCHEMA.TABLES — the equivalent of Couchbase's empty-collection case.
  • Druid's type system (LONG, DOUBLE, FLOAT, STRING, COMPLEX<...>) and how those should
    map to ColumnSchema.type.
  • Whether /druid/v2/sql/statements (the async, deep-storage endpoint) is worth supporting for
    long queries, or whether the synchronous endpoint is enough for an editor.

Capabilities (draft, confirm during implementation)

{
  queryLanguage: "sql",
  supportsExplain: false,       // see the EXPLAIN caveat; revisit once the real output is inspected
  supportsExternalQueryLimiting: true,
  supportsCreateTable: false,   // datasources are created by ingestion, not DDL
  supportsMaintenance: false,   // no VACUUM/ANALYZE equivalent; compaction is an ingestion concern
  supportsConnectionString: false,
  defaultPort: 8888,
}

supportsCreateTable is false for a real reason: Druid datasources come from ingestion specs, not
CREATE TABLE. Maintenance is likewise false — compaction and retention are managed through the
Coordinator and ingestion supervisors, not through anything the maintenance panel models.

Out of scope for the first PR

Ingestion (native specs, INSERT/REPLACE via MSQ), supervisor and task management, Coordinator
and Overlord administration APIs, retention rules, and compaction configuration.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions