diff --git a/README.md b/README.md index 0f4c87d2..3eb03cd8 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn

Multi-Database Connection Manager -
Connect to PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Redis, or SQLite with SSL/TLS and SSH Tunnel support. +
Connect to PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis, or SQLite with SSL/TLS and SSH Tunnel support.

--- @@ -185,9 +185,10 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn | **MongoDB** | `mongodb` | JSON query editor, collection operations (find, aggregate, insert, update, delete) | | **Couchbase** | none — HTTP (Query + management REST) | Full SQL++ IDE, EXPLAIN plans, bucket/scope/collection explorer, `INFER` column inference, read-your-writes consistency, `UPDATE STATISTICS` / `BUILD INDEX` / request kill | | **ClickHouse** | none — HTTP (SQL interface, port 8123) | Full SQL IDE, JSON EXPLAIN plan trees, system-table schema introspection, `OPTIMIZE TABLE` / table statistics / query kill maintenance | +| **Apache Druid** | none — HTTP (`POST /druid/v2/sql`, Router port 8888 or Broker 8082) | Read-only SQL IDE, native-query EXPLAIN plan trees, `INFORMATION_SCHEMA` datasource introspection, `sys.*` monitoring (segments, servers, ingestion tasks). Druid SQL has no `UPDATE`, no `DELETE` and no `CREATE TABLE`, and nothing it can do counts as a maintenance operation — a datasource changes through ingestion, not from the editor | | **Redis** | `ioredis` | Command editor, key browser, INFO-based monitoring | -> All SQL databases share: schema explorer, ER diagrams, schema diff & migration, display masking (preview), monitoring dashboard, and connection string import. +> All SQL databases share: schema explorer, ER diagrams, schema diff & migration, display masking (preview), monitoring dashboard, and connection string import. Druid is the exception twice over: its HTTP SQL API has no URI convention to paste, so it is configured by host and port only, and generated migration SQL has nothing to apply against an engine whose SQL contains no DDL. > **Provider reference docs:** each database has an in-depth reference (design, connection, query format, monitoring, limitations) under [`docs/providers/`](docs/providers/README.md). For the provider architecture see [`docs/DATABASE_PROVIDERS.md`](docs/DATABASE_PROVIDERS.md), and to add a new database see [`docs/ADDING_A_PROVIDER.md`](docs/ADDING_A_PROVIDER.md). @@ -203,7 +204,7 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn | **Editor** | Monaco Editor (VS Code Engine) | Web | | **AI** | Multi-Model (Gemini, OpenAI, Ollama, Custom) | Web, Mobile | | **Auth** | JWT (`jose`) + OIDC (`openid-client`), PKCE, Role Mapping | Web, Mobile | -| **Database** | PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Redis | Web, Mobile | +| **Database** | PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis | Web, Mobile | | **Charts** | Recharts (Bar, Line, Pie, Area, Scatter, Histogram, Stacked) | Web, Mobile | | **ERD** | React Flow, ELK.js (auto-layout) | Web | | **State/Grid** | TanStack Table & Virtual | Web, Mobile | @@ -294,7 +295,7 @@ journalctl -u libredb-studio ### Prerequisites - [Bun](https://bun.sh/) (Recommended) or Node.js 24+ - - A target database to query (PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, or Redis) + - A target database to query (PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Apache Druid, or Redis) ### Quick Start (Local) 1. **Clone & Install** @@ -340,7 +341,7 @@ journalctl -u libredb-studio Need databases to test with? We provide ready-to-use containers for all supported engines: ```bash -# Start all development databases (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle) +# Start every default-profile database (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, ...) docker compose -f database-compose.yml up -d # Or start a specific database @@ -348,6 +349,13 @@ docker compose -f database-compose.yml up -d postgres docker compose -f database-compose.yml up -d mssql docker compose -f database-compose.yml up -d oracle +# Apache Druid: profile-gated, so a bare `up -d` does NOT start it. Druid is a distributed +# system with no single-container mode - five Druid processes plus ZooKeeper plus its own +# metadata database is the minimum that can answer a SQL query, so all seven services carry +# `profiles: [druid]` rather than doubling the default stack. Connect to the Router on 8888 +# (or the Broker on 8082 - the same endpoint, no different configuration). +docker compose -f database-compose.yml --profile druid up -d + # Start PostgreSQL with sample e-commerce data docker compose -f docker/postgres.yml up -d @@ -356,6 +364,9 @@ docker compose -f database-compose.yml down # Stop and remove all data docker compose -f database-compose.yml down -v + +# The Druid containers need the profile flag here too - without it `down` leaves them running +docker compose -f database-compose.yml --profile druid down -v ``` ### Connection Details @@ -367,6 +378,7 @@ docker compose -f database-compose.yml down -v | **SQL Server** | localhost | 1433 | sa | Password123! | master | | **Oracle** | localhost | 1521 | system | Password123! | freepdb1 | | **MongoDB** | localhost | 27017 | admin | admin | — | +| **Apache Druid** | localhost | 8888 (Router) or 8082 (Broker) | — | — | — (one catalog, always `druid`) | ### PostgreSQL Sample Data @@ -416,7 +428,7 @@ bun run test:coverage |-------|-----------|--------|-------|----------------| | **Unit** | `tests/unit/` | `bun:test` | ~1,609 | Pure functions: SQL parser, connection strings, data masking, query limiter, schema diff, error classes, DB icons, showcase queries | | **API** | `tests/api/` | `bun:test` | ~279 | Route handlers: auth, query, transaction, maintenance, AI endpoints, middleware | -| **Integration** | `tests/integration/` | `bun:test` | ~346 | Database providers: PG, MySQL, SQLite, MongoDB, Couchbase, Redis, Oracle, MSSQL, ClickHouse| +| **Integration** | `tests/integration/` | `bun:test` | ~346 | Database providers: PG, MySQL, SQLite, MongoDB, Couchbase, Redis, Oracle, MSSQL, ClickHouse, Druid| | **Hooks** | `tests/hooks/` | `bun:test` | ~251 | React hooks: auth, connections, tabs, query execution, transactions, inline editing, AI chat, monitoring | | **Components** | `tests/components/` | `bun:test` + happy-dom | ~570 | UI components: Studio, Sidebar, QueryEditor, ResultsGrid, Admin Dashboard, Charts, ERD | | **E2E** | `e2e/` | Playwright | ~32 | Full browser flows: login, connections, query execution, tabs, export, admin | @@ -707,7 +719,7 @@ extraEnvFrom: | `defaults` | No | Default values merged into all connections | | `connections[].id` | Yes | Unique slug (`[a-z0-9-]+`, max 64 chars) | | `connections[].name` | Yes | Display name in UI | -| `connections[].type` | Yes | `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse` | +| `connections[].type` | Yes | `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse`, `druid` | | `connections[].roles` | Yes | `["*"]` (everyone), `["admin"]`, `["user"]`, or `["admin", "user"]` | | `connections[].managed` | No | `true` = read-only (default), `false` = editable copy for user | | `connections[].password` | No | Use `${ENV_VAR}` syntax for secrets | @@ -745,7 +757,7 @@ extraEnvFrom: - [ ] **Phase 17**: Enterprise Collaboration (User Identity, Shared Workspaces, SAML 2.0). - [ ] **Phase 18**: Server-Enforced Data Masking (SQL output-lineage, deployment-global policy, fail-closed API masking, alias/aggregate coverage). - [x] **Phase 19**: Driver-Free Providers — Couchbase (SQL++ over the Query REST API), the first provider that adds no runtime dependency. Pattern documented in [Adding a Provider](docs/ADDING_A_PROVIDER.md). -- [ ] **Phase 20**: Analytics Databases — ClickHouse ([#264](https://github.com/libredb/libredb-studio/issues/264)) and Apache Druid ([#265](https://github.com/libredb/libredb-studio/issues/265)), both driver-free over HTTP. +- [x] **Phase 20**: Analytics Databases — ClickHouse ([#264](https://github.com/libredb/libredb-studio/issues/264)) and Apache Druid ([#265](https://github.com/libredb/libredb-studio/issues/265)), both driver-free over HTTP. Druid is read-only by nature — no `UPDATE`, no `DELETE`, no `CREATE TABLE` — so it also demonstrates a provider that reports absent capabilities honestly instead of offering controls that can only fail. - [ ] **Phase 21**: Federated Query — Trino/Starburst. Deliberately unscheduled: a Trino catalog is another *system*, so what a connection pins is a product question that has to be answered before the work can be specified. --- diff --git a/database-compose.yml b/database-compose.yml index d351b01d..53b7ac85 100644 --- a/database-compose.yml +++ b/database-compose.yml @@ -1,3 +1,39 @@ +# Shared configuration for every Druid process. Druid is configured entirely through +# `druid_*` environment variables (the image's entrypoint translates them into +# runtime.properties), so one anchor keeps the five processes provably identical - a cluster +# whose processes disagree about ZooKeeper or the metadata store fails in confusing ways. +x-druid-env: &druid-env + # Sizes every process for a laptop. Without it each one claims cluster-scale heap and + # direct memory, and the five together will not fit. + DRUID_SINGLE_NODE_CONF: micro-quickstart + # druid-multi-stage-query is included on purpose: it is what makes `INSERT INTO ... SELECT` + # (MSQ) available on /druid/v2/sql/task. The provider does not use that endpoint - the point + # is that the live pass can prove the *synchronous* endpoint rejects INSERT even when the + # engine is fully capable of it, which is the honest reason supportsCreateTable is false. + druid_extensions_loadList: '["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]' + druid_zk_service_host: druid-zookeeper + druid_metadata_storage_host: "" + druid_metadata_storage_type: postgresql + druid_metadata_storage_connector_connectURI: jdbc:postgresql://druid-metadata:5432/druid + druid_metadata_storage_connector_user: druid + druid_metadata_storage_connector_password: druid + # Deep storage on a shared local volume. Real deployments use S3/HDFS; local is what makes + # this a single-machine fixture, and the historical and middlemanager must see the same files. + druid_storage_type: local + druid_storage_storageDirectory: /opt/shared/segments + druid_indexer_logs_type: file + druid_indexer_logs_directory: /opt/shared/indexing-logs + druid_indexer_runner_javaOptsArray: '["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=1g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]' + druid_indexer_fork_property_druid_processing_buffer_sizeBytes: 100MiB + druid_processing_numThreads: "2" + druid_processing_numMergeBuffers: "2" + # Lets the Router at 8888 proxy Coordinator and Overlord APIs, so the whole cluster is + # reachable through the one port the provider targets. + druid_router_managementProxy_enabled: "true" + # info, not the upstream compose's debug: the debug logger emits one line per internal + # request and buries the errors a live pass is looking for. + druid_emitter_logging_logLevel: info + services: postgres: image: postgres:18 @@ -163,3 +199,172 @@ services: # No seed sidecar: CLICKHOUSE_DB above creates `demo` on first boot, and unlike Couchbase # there is no index prerequisite - a SELECT against a freshly created table works # immediately. Do not add one. + + # --------------------------------------------------------------------------------------- + # Apache Druid. Unlike every other service here, Druid is a distributed system and has no + # single-container mode: five Druid processes plus ZooKeeper plus a metadata database is + # the minimum that can answer a SQL query. That is why all seven carry `profiles: [druid]` + # - a bare `docker compose -f database-compose.yml up -d` would otherwise grow from 7 + # containers to 14 and from roughly 6 GB of RAM to 12. Start it explicitly: + # + # docker compose -f database-compose.yml --profile druid up -d + # + # The provider only ever talks to the Router (8888) or the Broker (8082); the other five + # processes are cluster internals and deliberately publish no host port. 8091, the + # MiddleManager's usual port, is already taken by couchbase above - another reason not to. + # --------------------------------------------------------------------------------------- + druid-metadata: + # Druid's metadata store (segment table, task history, supervisors, config). Distinct + # from the `postgres` service above on purpose: sharing it would put Druid's ~20 internal + # tables into the demo database the studio's own postgres connection browses. + image: postgres:17.6 + container_name: libredb-druid-metadata + profiles: ["druid"] + restart: unless-stopped + environment: + POSTGRES_USER: druid + POSTGRES_PASSWORD: druid + POSTGRES_DB: druid + # No published port: only the Druid Coordinator/Overlord reads it, over the compose + # network. Publishing it would collide with the `postgres` service on 5432. + healthcheck: + test: ["CMD-SHELL", "pg_isready -U druid -d druid"] + interval: 5s + timeout: 5s + retries: 30 + druid-zookeeper: + # Druid's cluster coordination: process discovery, segment load queues, task assignment. + image: zookeeper:3.9.3 + container_name: libredb-druid-zookeeper + profiles: ["druid"] + restart: unless-stopped + environment: + ZOO_MY_ID: "1" + # zkServer.sh's own status check uses the `srvr` four-letter word, which 3.9 refuses + # unless it is whitelisted. + ZOO_4LW_COMMANDS_WHITELIST: "srvr,ruok,conf" + healthcheck: + test: ["CMD-SHELL", "echo ruok | nc -w 2 localhost 2181 | grep -q imok"] + interval: 5s + timeout: 5s + retries: 30 + druid-coordinator: + # Coordinator + Overlord in one process (the image's coordinator config sets + # druid.coordinator.asOverlord.enabled), so segment balancing and task management both + # live here. This is what accepts an ingestion task, which is how a datasource is + # created at all - Druid has no CREATE TABLE. + image: apache/druid:37.0.0 + container_name: libredb-druid-coordinator + profiles: ["druid"] + restart: unless-stopped + command: ["coordinator"] + environment: *druid-env + volumes: + - druid_shared:/opt/shared + - druid_coordinator_var:/opt/druid/var + depends_on: + druid-metadata: + condition: service_healthy + druid-zookeeper: + condition: service_healthy + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:8081/status/health"] + interval: 10s + timeout: 5s + retries: 40 + druid-historical: + # Serves published segments. Without it a datasource exists in the metadata store but + # returns no rows, which is the "datasource with no segments" case the provider has to + # render honestly. + image: apache/druid:37.0.0 + container_name: libredb-druid-historical + profiles: ["druid"] + restart: unless-stopped + command: ["historical"] + environment: *druid-env + volumes: + - druid_shared:/opt/shared + - druid_historical_var:/opt/druid/var + depends_on: + druid-coordinator: + condition: service_started + druid-zookeeper: + condition: service_healthy + druid-middlemanager: + # Runs ingestion tasks as forked peon processes. Needed for the live pass to load data. + image: apache/druid:37.0.0 + container_name: libredb-druid-middlemanager + profiles: ["druid"] + restart: unless-stopped + command: ["middleManager"] + environment: *druid-env + volumes: + - druid_shared:/opt/shared + - druid_middlemanager_var:/opt/druid/var + depends_on: + druid-coordinator: + condition: service_started + druid-zookeeper: + condition: service_healthy + druid-broker: + # Plans and merges queries. POST /druid/v2/sql here is the same API the Router proxies, + # so the provider works against 8082 directly too - published so that can be proven + # rather than assumed. + image: apache/druid:37.0.0 + container_name: libredb-druid-broker + profiles: ["druid"] + restart: unless-stopped + command: ["broker"] + environment: *druid-env + volumes: + - druid_broker_var:/opt/druid/var + ports: + - "8082:8082" + depends_on: + druid-coordinator: + condition: service_started + druid-zookeeper: + condition: service_healthy + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:8082/status/health"] + interval: 10s + timeout: 5s + retries: 40 + druid-router: + # The provider's default target. It fronts the SQL API, the web console and - because + # druid_router_managementProxy_enabled is set - the Coordinator and Overlord APIs, so a + # single port is enough for both querying and loading data. + image: apache/druid:37.0.0 + container_name: libredb-druid-router + profiles: ["druid"] + restart: unless-stopped + command: ["router"] + environment: *druid-env + volumes: + - druid_router_var:/opt/druid/var + ports: + - "8888:8888" + depends_on: + druid-broker: + condition: service_started + druid-coordinator: + condition: service_started + healthcheck: + test: ["CMD", "wget", "--spider", "-q", "http://localhost:8888/status/health"] + interval: 10s + timeout: 5s + retries: 40 + +volumes: + # Named volumes only for Druid: it is the one service here whose processes must share + # state (deep storage) and whose task history is worth surviving a restart. Everything + # above uses the image's own anonymous volumes. + # + # druid_shared is the deep-storage + indexing-log volume, mounted by the three processes + # that write or read segments. A `down -v` resets the cluster to empty. + druid_shared: {} + druid_coordinator_var: {} + druid_historical_var: {} + druid_middlemanager_var: {} + druid_broker_var: {} + druid_router_var: {} diff --git a/docs/ADDING_A_PROVIDER.md b/docs/ADDING_A_PROVIDER.md index 3ae132e3..196e5fe4 100644 --- a/docs/ADDING_A_PROVIDER.md +++ b/docs/ADDING_A_PROVIDER.md @@ -19,11 +19,12 @@ Three decisions. The first is the consequential one, which is why it is first. 1. **Does it need a driver at all?** Score the engine against the rubric below. A database with a first-class HTTP API can be supported with no dependency at all, and that is worth real effort to - establish before you start. Three shipped providers need no driver: SQLite uses the built-in - `bun:sqlite`/`node:sqlite` via `sqlite-driver.ts`, Couchbase talks to the cluster over - documented REST endpoints with `fetch`/`node:https` ([couchbase.md](./providers/couchbase.md)), - and ClickHouse speaks SQL over its HTTP interface the same way - ([clickhouse.md](./providers/clickhouse.md)). If it does need one, it will be something like `pg`, + establish before you start. Four shipped providers need no driver: SQLite uses the built-in + `bun:sqlite`/`node:sqlite` via `sqlite-driver.ts`, and three reach the engine over HTTP with + nothing but `fetch`/`node:https` — Couchbase over the documented REST endpoints + ([couchbase.md](./providers/couchbase.md)), ClickHouse over its HTTP interface + ([clickhouse.md](./providers/clickhouse.md)), and Apache Druid over `POST /druid/v2/sql` + ([druid.md](./providers/druid.md)). If it does need one, it will be something like `pg`, `mysql2`, `mongodb`, `ioredis`, `oracledb` or `mssql`. 2. **Which base class?** @@ -32,8 +33,11 @@ Three decisions. The first is the consequential one, which is why it is first. `this.type` — identifier and string escaping, `LIMIT` clause building, placeholder style, read-only and DDL detection — plus a `prepareQuery()` that applies the shared query limiter. None of it touches a pool, a driver or a connection, so **an HTTP transport is no reason to - avoid it.** A standard-SQL engine reached over HTTP, such as ClickHouse, should extend it and - get all of that for free. + avoid it.** A standard-SQL engine reached over HTTP, such as ClickHouse or Apache Druid, should + extend it and get all of that for free. Druid is the clearest case of how little is left over: + double-quoted identifiers and `LIMIT n OFFSET m` are both correct Druid SQL, so + `escapeIdentifier()`, `buildLimitClause()` and `getPlaceholder()` are inherited unchanged and + `prepareQuery()` is the only override — for a single dialect trap, not for the transport. - **Non-SQL databases → extend `BaseDatabaseProvider`** directly, like MongoDB and Redis. - The one reason a SQL-speaking provider extends `BaseDatabaseProvider` anyway is a dialect the shared helpers cannot express. Couchbase is that case: SQL++ quotes identifiers with doubled @@ -151,10 +155,10 @@ directly; reshaping rows inside the transport would have broken schema loading. ```typescript // Before: -export type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase'; +export type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase' | 'clickhouse' | 'druid'; // After (example: adding CockroachDB): -export type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase' | 'cockroachdb'; +export type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase' | 'clickhouse' | 'druid' | 'cockroachdb'; ``` ### 1.2 — Add to `QueryTab.type` if needed @@ -185,7 +189,7 @@ is kept in sync with its per-provider doc). Don't copy a skeleton from this guid |-------------------|--------|------------------|-----------| | Pooled SQL (wire-protocol DB) | `SQLBaseProvider` | `postgres.ts` / `mysql.ts` | [postgres.md](./providers/postgres.md) · [mysql.md](./providers/mysql.md) | | Embedded / file SQL | `SQLBaseProvider` | `sqlite.ts` | [sqlite.md](./providers/sqlite.md) | -| SQL database reached over HTTP (no driver) | `SQLBaseProvider` | `sql/clickhouse/` | [clickhouse.md](./providers/clickhouse.md) | +| SQL database reached over HTTP (no driver) | `SQLBaseProvider` | `sql/clickhouse/` or `sql/druid/` | [clickhouse.md](./providers/clickhouse.md) · [druid.md](./providers/druid.md) | | Document store | `BaseDatabaseProvider` | `mongodb.ts` | [mongodb.md](./providers/mongodb.md) | | Document store reached over HTTP/REST (no driver) | `BaseDatabaseProvider` | `document/couchbase/` | [couchbase.md](./providers/couchbase.md) | | Key-value store | `BaseDatabaseProvider` | `redis.ts` | [redis.md](./providers/redis.md) | @@ -294,6 +298,7 @@ Then add the type to the selectable list that drives the ConnectionModal picker: // Append to the existing list - do not retype it, or you will drop a provider from the picker. const selectableTypes: DatabaseType[] = [ 'postgres', 'mysql', 'sqlite', 'oracle', 'mssql', 'mongodb', 'couchbase', 'redis', 'libredb', + 'clickhouse', 'druid', 'cockroachdb', ]; ``` @@ -312,6 +317,8 @@ bun add # bun add ioredis (Redis) # SQLite needs no driver — bun:sqlite / node:sqlite are runtime built-ins (see sqlite-driver.ts) # Couchbase needs no driver — it speaks the Query and management REST APIs over fetch/node:https +# ClickHouse needs no driver — plain SQL over its HTTP interface (port 8123) +# Apache Druid needs no driver — plain SQL over POST /druid/v2/sql (Router 8888 or Broker 8082) ``` If your engine exposes a documented HTTP API, weigh it against the native driver before adding a @@ -326,8 +333,32 @@ rather than a mock. **HTTP 200 does not mean success.** Couchbase returns syntax and semantic errors inside a 200 response with `status: "errors"`, and Trino does the same with a `QueryError` field. Check the payload before the HTTP status, or a failed statement reads as "0 rows". This is not universal — -Apache Druid uses real 400/500 codes — so establish which behaviour applies before writing the error -path. +Apache Druid does use real 400 / 500 / 504 codes — so establish which behaviour applies before +writing the error path. + +**Real status codes still misclassify.** Druid answers `SELECT 1/0` with **HTTP 500**, +`persona: "ADMIN"` and `category: "UNCATEGORIZED"`, message "/ by zero" — an ordinary user mistake +reported as an admin-facing server failure, so reading 5xx as "the cluster is broken" would tell the +user something false. ClickHouse has the same hazard: a denied grant is a 500 rather than a 403, and +its message says "Not enough privileges" while containing neither "access denied" nor "permission +denied". **Classify on the engine's own error category or code, never on the status and never by +sniffing message text.** Druid's `category` is present in both of the envelopes it uses (the +structured `druidException` and the legacy wrapper) and is a closed enum; ClickHouse's numeric +exception code is in its plain-text error body. Each provider branches on that one field and on +nothing else. + +**64-bit integers can arrive as unquoted JSON numbers, and `JSON.parse` rounds them silently.** +ClickHouse turns `18446744073709551615` into `...552000`; Druid turns `9007199254740993` into +`9007199254740992`. No error is raised in either case, so the wrong number reaches the grid looking +exactly like the right one. Ask the server to quote them if it can — ClickHouse takes +`output_format_json_quote_64bit_integers=1` — and if it cannot, own the fix: Druid has no such +setting, so its transport runs a string-aware pass over the **raw body** before parsing and quotes +every integer literal outside `Number.MIN_SAFE_INTEGER … Number.MAX_SAFE_INTEGER`. String-aware is +the load-bearing part; a naive digit-run rewrite corrupts `"id: 9007199254740993"` inside a value. +Either way the number reaches the UI as an exact string, which is what the `pg` driver already does +for `int8`. The generalisable lesson: check the widest integer type your engine supports against +`Number.MAX_SAFE_INTEGER` before trusting `JSON.parse`, and expect to write the fix yourself when the +server offers no switch. **The response envelope does not always describe the rows.** Couchbase's `signature` is `"*"` for `SELECT *`, and `{ id, "*" }` for a wildcard mixed with named projections. Taking those keys @@ -373,6 +404,14 @@ control that only emits invalid input. That is the defect class strategy declines, so the button is dead while only the background pre-warm works. When the engine has no analyze equivalent, return the estimate for both modes — `sqlite-queryplan.ts` and `couchbase-json.ts` both do exactly that. +- A capability can be absent because the **grammar** lacks it rather than because nobody implemented + it, and the flag reads the same either way — so check, and then say so. Druid answers + `CREATE TABLE t (id BIGINT)` with a syntax error, because `CREATE` is not one of its statements at + all (a datasource comes into existence by being ingested into), so `supportsCreateTable` is + `false`. Nothing in `MaintenanceType` has a SQL-reachable Druid analogue either — compaction and + retention are Coordinator and task concerns, and `kill` has nowhere to get a query id from because + Druid publishes no catalog of running queries — so `supportsMaintenance` is `false` with an empty + operation list, rather than true with nothing behind it. The same honesty rule governs monitoring: **a source the connected user cannot read returns empty, it never throws.** Monitoring catalogs are frequently permission-gated, so a denial is the normal @@ -385,12 +424,16 @@ case for a restricted user and must not break an otherwise working connection. Mock-based tests are the repo standard and they are **not sufficient on their own**. On the Couchbase provider a live pass against a real cluster disproved a design decision — un-indexed collections turned out to be queryable on Server 7.6+ through a sequential scan — and found three -defects the mocks had accepted without complaint. +defects the mocks had accepted without complaint. On Druid it overturned a verdict recorded in **this +guide** (see [Driver-free candidates](#driver-free-candidates)): the EXPLAIN output was predicted not +to fit the tree render model, and the real plan turned out to be a genuine nested tree. Before opening the PR, drive the provider through the running application against a real server: - full `INSERT` / `UPDATE` / `SELECT` / `DELETE`, including a `SELECT` immediately after a write, to - catch read-your-writes problems + catch read-your-writes problems — or establish that the engine has no write statement to test. + Druid SQL has neither `UPDATE` nor `DELETE` in its grammar and rejects `INSERT`/`REPLACE` on the + native engine, and each of those is a claim only an actual attempt can settle - both error paths — a syntax error and a missing object — confirming each surfaces as an error rather than as zero rows - schema introspection, checking column types and the object-naming rule @@ -398,7 +441,11 @@ Before opening the PR, drive the provider through the running application agains a broken direct action - every monitoring panel, and each maintenance operation -Add a service to `database-compose.yml` so the next person can repeat this. +Add a service to `database-compose.yml` so the next person can repeat this — or a profile-gated set of +them, which is what a distributed engine needs. Druid has no single-container mode, so its seven +services all carry `profiles: ["druid"]`: a default `docker compose up -d` must not double for +everyone who is not working on Druid, and `docker compose --profile druid down` is then needed to +remove them again. --- @@ -516,27 +563,43 @@ For the authoritative, code-verified reference for each shipped provider (extend driver, pooling, capabilities, labels, `prepareQuery` behaviour, and limitations), see the prime docs — they are the single source of truth and are kept in sync with the code: -**[docs/providers/](./providers/README.md)** → postgres · mysql · oracle · mssql · sqlite · redis · mongodb · couchbase · clickhouse · libredb +**[docs/providers/](./providers/README.md)** → postgres · mysql · oracle · mssql · sqlite · redis · mongodb · couchbase · clickhouse · druid · libredb When implementing a new provider, the closest existing analogue is the best template: a pooled SQL provider (postgres/mysql), an embedded SQL provider (sqlite), a non-SQL provider (mongodb/redis), or -a driverless provider reached over HTTP (couchbase). +a driverless provider reached over HTTP (clickhouse or druid for SQL, couchbase for a document +store). ## Driver-free candidates Assessed against the rubric in [Prerequisites](#prerequisites). Anything not listed almost certainly needs a driver. +**Shipped since this list was written:** Couchbase +([#263](https://github.com/libredb/libredb-studio/issues/263)), ClickHouse +([#264](https://github.com/libredb/libredb-studio/issues/264)) and Apache Druid +([#265](https://github.com/libredb/libredb-studio/issues/265)). + +Druid is worth a paragraph, because it **corrected this table's own verdict**. The entry that stood +here rated it strong but predicted that `EXPLAIN PLAN FOR` "returns a native-query translation rather +than an operator tree, so it does not fit the existing tree render model". The live plan disproved +that: `query.dataSource` recurses — `join` carries `left` and `right`, `query` carries one child, +`union` carries a list — so the native query **is** a nested tree, and it renders as +`{ kind: "tree" }` with nothing forced. What keeps that honest is the omission: Druid's planner emits +no cost and no row estimate, so no node carries `metrics`, and node labels name Druid's own query +types (`groupBy`, `scan`, `timeseries`, `topN`) rather than borrowing a relational-plan vocabulary. +The lesson for the next candidate is to read the engine's real EXPLAIN output before predicting the +render model from its documentation. See [druid.md](./providers/druid.md). + | Candidate | Verdict | |---|---| -| **Apache Druid** ([#265](https://github.com/libredb/libredb-studio/issues/265)) | Strong, and it returns errors with real HTTP status codes. `EXPLAIN PLAN FOR` returns a native-query translation rather than an operator tree, so it does not fit the existing tree render model | | **Trino / Starburst** | Highest strategic value — one provider fronts S3, Iceberg, Delta and Hive. Unscheduled on purpose: a catalog is another *system*, so what a connection pins is a product question. Also a `nextUri` polling protocol and a fragmented auth matrix | | **OpenSearch / Elasticsearch** | HTTP is the only protocol. Needs a dialect decision first: the SQL endpoint is a subset, the native DSL is JSON. OpenSearch is Apache 2.0 and the cleaner primary target | | **Snowflake / BigQuery / Databricks SQL** | REST SQL APIs exist and the data model fits; auth is the wall (key-pair JWT, service-account signing, OAuth) and that is where the no-dependency promise ends | | **CouchDB, ArangoDB, SurrealDB, Qdrant, Weaviate** | All HTTP, all non-SQL or only partially SQL. Feasible, but each needs its own query grammar the way MongoDB and LibreDB do | Contributions are welcome for any of these. Open an issue with the rubric score first, so the design -decisions are settled before code exists — that is what let the Couchbase provider land as a single -reviewable PR. +decisions are settled before code exists — that is what let the Couchbase, ClickHouse and Druid +providers each land as a single reviewable PR. --- @@ -554,8 +617,11 @@ The integration points, all of which need an entry. This is the list the Strateg - [ ] `src/hooks/use-connection-form.ts` — **append** to `selectableTypes` (do not retype the array) - [ ] `src/components/icons/db-icons.tsx` — the engine's mark (`strokeWidth={1.5}`, no HTML size attrs) - [ ] `src/lib/seed/types.ts` — the seed-config `type` enum, or seeded connections fail validation -- [ ] `package.json` — the driver, **if** it needs one. A driver-free provider leaves it untouched -- [ ] `database-compose.yml` — a service, so the next person can repeat the live pass +- [ ] `package.json` — the driver, **if** it needs one. A driver-free provider leaves it untouched, and + three shipped ones do: `couchbase`, `clickhouse` and `druid` each add nothing here +- [ ] `database-compose.yml` — a service, so the next person can repeat the live pass. A distributed + engine contributes a `profiles: [...]` set instead, as Druid's seven services do, so the default + stack does not grow for everyone **Conditionally, and each one is easy to miss because the code still compiles without it:** @@ -578,7 +644,7 @@ connection-form test), so the compiler and those tests refuse to pass until each > **`git grep -l -- src/ tests/` is the authoritative checklist.** > This list is maintained by hand and has been wrong before: it long claimed "no other files should > need changes", while Couchbase (#263) and ClickHouse (#264) each touched 27 files under `src/` and -> `tests/`. Trust the grep over this list. +> `tests/`, and Druid (#265) roughly two dozen of its own. Trust the grep over this list. What the Strategy Pattern *does* spare you is **provider logic**: no route, no shared component and no existing provider needs to know your engine exists. If you find yourself adding a `=== ''` diff --git a/docs/API_DOCS.md b/docs/API_DOCS.md index 1c5217f6..6b55c88b 100644 --- a/docs/API_DOCS.md +++ b/docs/API_DOCS.md @@ -24,12 +24,12 @@ ## Overview -LibreDB Studio provides a RESTful API for database management operations. The API supports PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, and Redis. +LibreDB Studio provides a RESTful API for database management operations. The API supports PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Apache Druid, and Redis. ### Key Features - **JWT Authentication** - Secure token-based authentication stored in HTTP-only cookies -- **Multi-Database Support** - PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Redis +- **Multi-Database Support** - PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis - **AI-Powered Queries** - Natural language to SQL with streaming responses - **Real-time Health Monitoring** - Database metrics and performance insights @@ -417,6 +417,50 @@ providers: --- +##### Apache Druid Query Format + +Druid speaks SQL over `POST /druid/v2/sql`, so the `sql` field carries a plain statement. Three +things differ from the other SQL providers: + +- **There is no `database` and no `connectionString`.** `INFORMATION_SCHEMA.SCHEMATA` reports exactly + one catalog, always `druid`, so a connection is `host` + `port` alone. `user`/`password` are + optional and are sent as HTTP basic auth for a cluster running `druid-basic-security`; a default + install ignores the `Authorization` header entirely. `port` is the Router's `8888`; the Broker's + `8082` serves the identical endpoint and needs no other change. +- **Druid SQL cannot write.** `UPDATE` and `DELETE` are not in the grammar, `CREATE TABLE` is a + syntax error, and `INSERT` / `REPLACE` are refused by the native engine ("consider using MSQ"). + Each comes back as `400` / `QUERY_ERROR` carrying Druid's own message, which names the reason and + the alternative. `POST /api/db/maintenance` accepts no operation at all for a `druid` connection. +- **A statement ending in `OFFSET n` with no `LIMIT` is sent unchanged**, so `wasLimited` is `false`: + Druid rejects `OFFSET n LIMIT m` ("'OFFSET start LIMIT count' is not allowed under the current SQL + conformance level"), so the auto-limiter must not append one there. Every other statement is + limited normally. + +```json +{ + "connection": { + "type": "druid", + "host": "localhost", + "port": 8888 + }, + "sql": "SELECT * FROM \"libredb_demo\" LIMIT 50" +} +``` + +**Notes:** +- A duplicate output name (a join projecting two `id`s, say) is disambiguated rather than dropped: + `fields` carries `id` and `id (2)`, and both columns reach the grid. +- `ORDER BY` on a non-`__time` column of a plain table scan is refused by the planner ("SQL query + requires ordering a table by non-time column"). Order by `__time`, or aggregate with `GROUP BY`. +- Druid uses Calcite's reserved-word list, which is large and surprising — `SELECT 1 AS one` is a + syntax error — so every generated identifier is double-quoted. +- Integers wider than 2^53 are returned as exact strings rather than as JSON numbers, so no value is + silently rounded on the way to the grid. `ARRAY` columns arrive as JSON strings (`"[1,2]"`), which + is what Druid's own clients show. +- Full reference: [`docs/providers/druid.md`](providers/druid.md). + +--- + ##### Redis Query Format Redis is a key-value store, so the `sql` field carries a Redis command instead of SQL. Two interchangeable formats are accepted. @@ -615,6 +659,8 @@ Run database maintenance operations. The handler validates against the target provider's capabilities: `type` is required (`{ "error": "Maintenance type is required" }`), the provider must support maintenance at all, and the requested operation must be in that provider's supported set (see the matrix above) — otherwise a `400` is returned listing what the provider does support. +A `druid` connection fails the second check whatever the `type` is, with `{ "error": "Maintenance operations not supported for this database" }`: no maintenance operation is reachable from Druid SQL, so its supported set is empty by design. Compaction and retention are Coordinator and task concerns, and Druid publishes no catalog of running queries, so there is no id for `kill` to name. + --- ### AI API @@ -792,12 +838,12 @@ interface DatabaseConnection { port?: number; // Port number user?: string; // Username password?: string; // Password - database?: string; // Database name (Couchbase: the bucket) - connectionString?: string; // Full connection string (alternative) + database?: string; // Database name (Couchbase: the bucket; Druid: unused, it has one catalog) + connectionString?: string; // Full connection string (alternative; Druid has no URI form, host + port only) createdAt: Date; // Creation timestamp } -type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase' | 'clickhouse'; +type DatabaseType = 'postgres' | 'mysql' | 'sqlite' | 'mongodb' | 'redis' | 'oracle' | 'mssql' | 'libredb' | 'couchbase' | 'clickhouse' | 'druid'; ``` ### TableSchema diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a67a368b..747406ff 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -4,7 +4,7 @@ This document outlines the architectural patterns, tech stack, and system design ## System Overview -LibreDB Studio is a hybrid, cloud-native database management tool that provides an IDE-like experience in the browser. It supports **10 database backends** via a Strategy Pattern abstraction: PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Redis, LibreDB. +LibreDB Studio is a hybrid, cloud-native database management tool that provides an IDE-like experience in the browser. It supports **11 database backends** via a Strategy Pattern abstraction: PostgreSQL, MySQL, SQLite, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis, LibreDB. It runs in two modes: as a **standalone Next.js app** and as an **embedded npm package** (`@libredb/studio`) consumed by libredb-platform. See [§4.6](#46-workspace-abstraction-npm-package-embedding). @@ -48,6 +48,7 @@ graph TD SQL --> Oracle[(Oracle)] SQL --> MSSQL[(SQL Server)] SQL --> ClickHouse[(ClickHouse)] + SQL --> Druid[(Apache Druid)] Document --> MongoDB[(MongoDB)] Document --> Couchbase[(Couchbase)] KeyValue --> Redis[(Redis)] @@ -101,6 +102,7 @@ classDiagram SQLBaseProvider <|-- OracleProvider SQLBaseProvider <|-- MSSQLProvider SQLBaseProvider <|-- ClickHouseProvider + SQLBaseProvider <|-- DruidProvider ``` Each provider implements: @@ -110,7 +112,7 @@ Each provider implements: Adding a new database type requires: **1 provider class** + **1 entry in `db-ui-config.ts`**. -`CouchbaseProvider` extends `BaseDatabaseProvider` even though SQL++ is a SQL dialect: `SQLBaseProvider` owns pooled-driver mechanics (per-dialect escaping, placeholders, transactions, `cancelQuery`) that its stateless HTTP transport does not have, so the SQL-ness is declared through `queryLanguage: 'sql'` instead. It is also the only provider with no driver dependency — the cluster is reached over the documented Query and management REST APIs. See [`docs/providers/couchbase.md`](providers/couchbase.md). +`CouchbaseProvider` extends `BaseDatabaseProvider` even though SQL++ is a SQL dialect: SQL++ quotes identifiers with doubled backticks, which `escapeIdentifier()` produces for no existing type, so it owns its quoting and declares its SQL-ness through `queryLanguage: 'sql'` instead. Being reached over HTTP is **not** the reason — `ClickHouseProvider` and `DruidProvider` add no driver either, and both extend `SQLBaseProvider`, because double-quoted identifiers and `LIMIT n OFFSET m` are correct in both dialects. Each of the three is a directory rather than a single file, with its wire format behind a transport seam that provider logic never bypasses. See [`docs/providers/couchbase.md`](providers/couchbase.md), [`clickhouse.md`](providers/clickhouse.md) and [`druid.md`](providers/druid.md). ## 4. Key Architectural Patterns @@ -237,7 +239,7 @@ src/ └── lib/ ├── db/ # Database provider module │ ├── providers/ - │ │ ├── sql/ # postgres, mysql, sqlite (+ sqlite-driver runtime adapter), oracle, mssql, clickhouse/ (transport seam + SQL over HTTP) + │ │ ├── sql/ # postgres, mysql, sqlite (+ sqlite-driver runtime adapter), oracle, mssql, clickhouse/ (transport seam + SQL over HTTP), druid/ (transport seam + SQL over POST /druid/v2/sql) │ │ ├── document/ # mongodb, couchbase/ (transport seam + SQL++ over REST) │ │ ├── keyvalue/ # redis │ │ └── embedded/ # libredb (built-in embedded provider for the sample connection) diff --git a/docs/DATABASE_PROVIDERS.md b/docs/DATABASE_PROVIDERS.md index c366fa27..6b1ebec7 100644 --- a/docs/DATABASE_PROVIDERS.md +++ b/docs/DATABASE_PROVIDERS.md @@ -29,11 +29,16 @@ src/lib/db/ │ │ ├── sqlite-driver.ts # SQLite runtime driver adapter (bun:sqlite | node:sqlite) │ │ ├── oracle.ts # Oracle Strategy │ │ ├── mssql.ts # SQL Server Strategy -│ │ └── clickhouse/ # ClickHouse Strategy (SQL over HTTP, no driver) -│ │ ├── index.ts # ClickHouseProvider -│ │ ├── transport.ts # ClickHouseTransport seam + neutral result types +│ │ ├── clickhouse/ # ClickHouse Strategy (SQL over HTTP, no driver) +│ │ │ ├── index.ts # ClickHouseProvider +│ │ │ ├── transport.ts # ClickHouseTransport seam + neutral result types +│ │ │ ├── http-transport.ts # The one HTTP implementation (fetch) +│ │ │ └── introspect.ts # system.* catalogs (databases/tables/columns/data_skipping_indices) +│ │ └── druid/ # Apache Druid Strategy (SQL over POST /druid/v2/sql, no driver) +│ │ ├── index.ts # DruidProvider +│ │ ├── transport.ts # DruidTransport seam + neutral result types + error categories │ │ ├── http-transport.ts # The one HTTP implementation (fetch) -│ │ └── introspect.ts # system.* catalogs (databases/tables/columns/data_skipping_indices) +│ │ └── introspect.ts # INFORMATION_SCHEMA datasources + sys.servers/segments/tasks │ ├── document/ # Document Database Providers │ │ ├── mongodb.ts # MongoDB Strategy │ │ └── couchbase/ # Couchbase Strategy (SQL++ over REST, no driver) @@ -61,7 +66,8 @@ BaseDatabaseProvider (abstract) │ ├── SQLiteProvider │ (shared SQL utilities) │ ├── OracleProvider │ │ ├── MSSQLProvider │ -│ └── ClickHouseProvider │ +│ ├── ClickHouseProvider │ +│ └── DruidProvider │ ├── MongoDBProvider ────────────────────────┤ Document Database ├── CouchbaseProvider ──────────────────────┤ Document Database (SQL++ over REST) ├── RedisProvider ──────────────────────────┤ Key-Value Store @@ -70,7 +76,9 @@ BaseDatabaseProvider (abstract) `SQLBaseProvider` provides SQL-specific helpers (LIMIT injection, identifier escaping, placeholder generation). Non-SQL databases like MongoDB, Redis, and LibreDB extend `BaseDatabaseProvider` directly. LibreDB is embedded (opened in-process from a file, like SQLite) but, having no SQL, it is a key-value-style provider rather than a SQL one. -Couchbase is the one provider that speaks a SQL dialect (SQL++) without extending `SQLBaseProvider`: that base owns pooled-driver mechanics — per-dialect escaping, placeholder generation, transactions, `cancelQuery` — that a stateless HTTP transport does not have. Its SQL-ness is expressed through `queryLanguage: 'sql'` in the capabilities instead. See [providers/couchbase.md](./providers/couchbase.md). +Couchbase is the one provider that speaks a SQL dialect (SQL++) without extending `SQLBaseProvider`: SQL++ quotes identifiers with doubled backticks, which `escapeIdentifier()` produces for no existing type, so it owns its quoting and expresses its SQL-ness through `queryLanguage: 'sql'` in the capabilities instead. See [providers/couchbase.md](./providers/couchbase.md). + +Being driver-free and reached over HTTP is not what decides the base class. `ClickHouseProvider` and `DruidProvider` add no driver either, and both extend `SQLBaseProvider`: double-quoted identifiers and `LIMIT n OFFSET m` are correct in both dialects, so identifier escaping, the `LIMIT` builder and the placeholder style are inherited rather than rewritten. Druid overrides only `prepareQuery()`, and only because it rejects `OFFSET n LIMIT m` — a statement that already ends in an `OFFSET` is therefore sent unlimited instead of being rewritten into a syntax error. See [providers/clickhouse.md](./providers/clickhouse.md) and [providers/druid.md](./providers/druid.md). The `SQLiteProvider` loads its embedded driver at runtime through `sqlite-driver.ts`: `bun:sqlite` under Bun, `node:sqlite` under plain Node (Node >= 24 built-in). Set `LIBREDB_SQLITE_DRIVER=bun|node` to force a driver. `better-sqlite3` is **not** used by the DB provider — it is only the SQLite driver for the storage layer (`src/lib/storage/`). @@ -111,7 +119,7 @@ QueryEditor /api/db/query ## Supported Databases -Ten providers are supported. For the per-provider reference (driver, pooling, query format, +Eleven providers are supported. For the per-provider reference (driver, pooling, query format, monitoring, limitations, …) see the prime docs in **[`docs/providers/`](./providers/README.md)**: | Provider | type-id | Family | Reference | @@ -125,6 +133,7 @@ monitoring, limitations, …) see the prime docs in **[`docs/providers/`](./prov | MongoDB | `mongodb` | Document | [providers/mongodb.md](./providers/mongodb.md) | | Couchbase | `couchbase` | Document (SQL++) | [providers/couchbase.md](./providers/couchbase.md) | | ClickHouse | `clickhouse` | SQL | [providers/clickhouse.md](./providers/clickhouse.md) | +| Apache Druid | `druid` | SQL (read-only) | [providers/druid.md](./providers/druid.md) | | LibreDB | `libredb` | Embedded (key-value) | [providers/libredb.md](./providers/libredb.md) | ## Core Interface @@ -297,7 +306,14 @@ DatabaseError (base) Provider-specific behaviour — pooling model, SSL/encryption, pagination, monitoring sources, maintenance operations, and known limitations — is documented per provider under [`docs/providers/`](./providers/README.md). Start there for anything specific to PostgreSQL, MySQL, -Oracle, SQL Server, SQLite, Redis, MongoDB, Couchbase, ClickHouse, or LibreDB. +Oracle, SQL Server, SQLite, Redis, MongoDB, Couchbase, ClickHouse, Apache Druid, or LibreDB. + +Not every provider has every feature, and the docs record the absences rather than glossing over +them. Druid is the sharpest case: its SQL has no `UPDATE`, no `DELETE` and no `CREATE TABLE`, no +maintenance operation is reachable from SQL, it has no user-defined indexes and no foreign keys, and +it keeps no query log — so `supportsCreateTable` and `supportsMaintenance` are `false`, and +`getIndexStats()`, `getSlowQueries()` and `getPerformanceMetrics()` return empty or zeroed values +that are the truth about the engine rather than a fallback. ## Security Considerations diff --git a/docs/FEATURES.md b/docs/FEATURES.md index db5740b4..0194fe78 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -27,7 +27,7 @@ ### 4. Visual EXPLAIN (Query Analyzer) * **Performance Visualization:** Visual execution plan to identify performance bottlenecks. * **Detailed Metrics:** Graphical representation of database scan types, join operations, costs, and execution times. -* **Multi-DB Support:** PostgreSQL and MySQL JSON plans, SQLite `EXPLAIN QUERY PLAN`, Couchbase SQL++ plan trees, and ClickHouse JSON plan trees. Providers without a real analyze mode hide the toggle instead of degrading to an estimate. +* **Multi-DB Support:** PostgreSQL and MySQL JSON plans, SQLite `EXPLAIN QUERY PLAN`, Couchbase SQL++ plan trees, ClickHouse JSON plan trees, and Apache Druid native-query plan trees. Providers without a real analyze mode hide the toggle instead of degrading to an estimate. A plan also shows only the numbers its planner actually reports: Druid emits no cost and no row estimate, so its nodes carry structure and no metrics rather than invented ones. ### 5. AI Query Assistant (Multi-Provider LLM) * **Natural Language to SQL:** Convert natural language requests into high-precision SQL code. @@ -47,6 +47,7 @@ * **Oracle:** Full support with connection pooling (`oracledb`); introspection/monitoring via the `ALL_*`/`DBA_*` data-dictionary views. * **SQL Server:** Full support with connection pooling (`mssql`); monitoring via DMVs (`sys.dm_*`). * **ClickHouse:** Full support with **no driver dependency** — SQL over the documented HTTP interface, so the SQL editor, limiter and NL2SQL all apply. Column types read verbatim from `system.columns`, JSON EXPLAIN plan trees, and `OPTIMIZE TABLE` / table-statistics / query-kill maintenance. + * **Apache Druid:** Read-only support with **no driver dependency** — SQL over `POST /druid/v2/sql` on the Router (8888) or the Broker (8082), so the SQL editor, limiter and NL2SQL all apply. Datasources and column types from `INFORMATION_SCHEMA`, native-query EXPLAIN plan trees, and monitoring from `sys.segments` / `sys.servers` / `sys.tasks`. Read-only is the engine, not the integration: Druid SQL has no `UPDATE`, no `DELETE` and no `CREATE TABLE`, and no maintenance operation is reachable from SQL, so those controls are reported as unsupported instead of failing when used. * **Document Databases:** * **MongoDB:** Full support with official driver, JSON-based MQL queries, automatic schema inference, and aggregation pipelines. * **Couchbase:** Full support with **no driver dependency** — SQL++ over the documented Query and management REST APIs, so the SQL editor, limiter and NL2SQL all apply. Buckets/scopes/collections flattened into the schema explorer, `INFER`-based column inference, visual EXPLAIN plans, and read-your-writes query consistency by default. diff --git a/docs/SEED_CONNECTIONS.md b/docs/SEED_CONNECTIONS.md index 10d056db..803241b1 100644 --- a/docs/SEED_CONNECTIONS.md +++ b/docs/SEED_CONNECTIONS.md @@ -58,7 +58,7 @@ defaults: # Optional — merged into every connection connections: - id: "analytics-pg" # Required, unique, lowercase slug [a-z0-9-] name: "Analytics DB" # Required, display name in UI - type: postgres # Required: postgres|mysql|sqlite|mongodb|redis|oracle|mssql|libredb + type: postgres # Required: postgres|mysql|sqlite|mongodb|redis|oracle|mssql|libredb|couchbase|clickhouse|druid host: "${PG_HOST}" port: 5432 database: analytics @@ -86,6 +86,18 @@ connections: roles: ["*"] # Everyone can see this managed: false # User gets an editable copy environment: development + + - id: "events-druid" + name: "Druid Events" + type: druid + host: "${DRUID_HOST}" + port: 8888 # Router. The Broker's 8082 serves the same endpoint + roles: ["*"] + environment: production + # No `database`: Druid reports exactly one catalog, always `druid`, so there is + # nothing to select. No `connectionString` either - its HTTP SQL API has no URI + # convention, so host and port are the whole address. + # user/password are optional and only reach a cluster running druid-basic-security. ``` ### Field Reference @@ -100,13 +112,13 @@ connections: | `connections` | Yes | — | Array of connection definitions (min 1) | | `connections[].id` | Yes | — | Unique slug: `[a-z0-9-]+`, max 64 chars | | `connections[].name` | Yes | — | Display name, max 128 chars | -| `connections[].type` | Yes | — | Database type: `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse` | +| `connections[].type` | Yes | — | Database type: `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse`, `druid` | | `connections[].host` | No | — | Hostname or IP | | `connections[].port` | No | — | Port number (1-65535) | -| `connections[].database` | No | — | Database name | +| `connections[].database` | No | — | Database name (Couchbase: the bucket. Druid has one catalog and ignores it) | | `connections[].user` | No | — | Username | | `connections[].password` | No | — | Password (use `${ENV_VAR}` syntax) | -| `connections[].connectionString` | No | — | Full connection string (use `${ENV_VAR}`) | +| `connections[].connectionString` | No | — | Full connection string (use `${ENV_VAR}`). Druid has no URI form — a `druid` connection needs `host` and is addressed by host and port only | | `connections[].roles` | Yes | — | Access control: `["*"]`, `["admin"]`, `["user"]`, `["admin", "user"]` | | `connections[].managed` | No | from defaults | `true` = read-only, `false` = editable copy | | `connections[].environment` | No | from defaults | Environment badge | diff --git a/docs/providers/README.md b/docs/providers/README.md index cc94b7cf..e5fbfbf4 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -15,14 +15,15 @@ in lockstep with the code (see the tri-sync rule in [`../../CLAUDE.md`](../../CL | MongoDB | `mongodb` | Document | `mongodb` | JSON (MQL) | [mongodb.md](./mongodb.md) | | Couchbase | `couchbase` | Document | none (HTTP: Query + management REST) | SQL (SQL++) | [couchbase.md](./couchbase.md) | | ClickHouse | `clickhouse` | SQL | none (HTTP interface) | SQL | [clickhouse.md](./clickhouse.md) | +| Apache Druid | `druid` | SQL (analytics) | none (HTTP: SQL endpoint) | SQL (Calcite) | [druid.md](./druid.md) | | LibreDB | `libredb` | Embedded (Key-Value) | `@libredb/libredb` | JSON (command grammar) | [libredb.md](./libredb.md) | ## Conventions - **Filename = canonical type-id** (`postgres.md`, `mssql.md`, …), mirroring the source file (`src/lib/db/providers//.ts`, or a `/` directory when a provider is - split across modules, as Couchbase is). The official product name (e.g. "SQL Server") is used only - in each doc's title and prose. + split across modules, as Couchbase, ClickHouse and Druid are). The official product name (e.g. + "SQL Server") is used only in each doc's title and prose. - **Each doc mirrors the code.** Every `file:line` citation is verified, and the per-provider triad — code, this doc, and `tests/integration/db/-provider.test.ts` — must stay in sync in the same PR (the *provider tri-sync invariant*). diff --git a/docs/providers/druid.md b/docs/providers/druid.md new file mode 100644 index 00000000..727387a4 --- /dev/null +++ b/docs/providers/druid.md @@ -0,0 +1,1522 @@ +# Apache Druid Provider + +> Apache Druid support for LibreDB Studio, built on Druid's SQL HTTP endpoint +> (`POST /druid/v2/sql`, port `8888` on the Router or `8082` on the Broker) with **no driver +> dependency of any kind**: every statement is a JSON body and the answer comes back through the +> runtime's own `fetch`. This document is the single reference point for the Druid provider: design, +> architecture, usage, and tests. If you are reading the code, extending Druid support, or authoring +> a new provider over HTTP, start here. + +| | | +|---|---| +| **Status** | Implemented & shipped | +| **Database type id** | `druid` | +| **Family** | SQL (`src/lib/db/providers/sql/druid/`) | +| **Driver** | None — HTTP only (`fetch`, a runtime built-in) | +| **Query language** | `sql` (Apache Calcite dialect) | +| **Default port** | `8888` (the Router). `8082` (the Broker) serves the identical endpoint and needs no different configuration — see [§3.3](#33-router-8888-or-broker-8082--both-work-identically) | +| **Connection pooling** | None — each statement is one stateless HTTP request | +| **Connection string** | **Not supported** — Druid has no URI convention for its SQL API ([§4.2](#42-there-is-no-connection-string-and-that-is-deliberate)) | +| **EXPLAIN** | `druid-native` — the native query plan as a tree. Druid's `EXPLAIN` never executes the statement, so there is no separate analyze mode | +| **Writes** | **None are possible.** Druid SQL has no `UPDATE`, no `DELETE` and no `CREATE TABLE`; `INSERT`/`REPLACE` need the MSQ task engine ([§5.5](#55-druid-sql-cannot-write-and-the-server-says-so-clearly)) | +| **Transactions** | Not exposed (Druid has none) | +| **Maintenance** | None — nothing in `MaintenanceType` has a SQL-reachable Druid analogue ([§8](#8-maintenance)) | +| **Query cancellation** | No `cancelQuery`; the server-side statement deadline is what stops a runaway query ([§13](#13-known-limitations--future-work)) | +| **Verified against** | **Apache Druid 37.0.0**, datasources `libredb_demo` (50 rows) and `libredb_rollup` (20 rows) | +| **Source** | [`src/lib/db/providers/sql/druid/`](../../src/lib/db/providers/sql/druid/) | +| **Tests** | [`tests/integration/db/druid-provider.test.ts`](../../tests/integration/db/druid-provider.test.ts) + [`tests/unit/db/druid/`](../../tests/unit/db/druid/) + [`tests/unit/lib/explain/druid-native.test.ts`](../../tests/unit/lib/explain/druid-native.test.ts) | +| **Tracking issue** | [#265 — Add Apache Druid provider](https://github.com/libredb/libredb-studio/issues/265) | + +--- + +## 1. Overview + +Apache Druid is a distributed real-time analytics database. Its **SQL layer is Apache Calcite over +Druid's own native query engine**, exposed as a plain JSON HTTP endpoint — the same one Druid's own +web console uses, and the one this provider speaks. That endpoint is the whole surface the provider +needs: querying, catalog introspection and monitoring are all SQL, over one path. + +Three things are Druid-shaped, and nearly every decision below flows from one of them: + +1. **Druid SQL is read-only.** Not "writes are unimplemented here" — `UPDATE` and `DELETE` are not in + the grammar, `CREATE TABLE` is not in the grammar, and `INSERT`/`REPLACE` are rejected by the + engine that answers this endpoint. A datasource comes into existence by being **ingested into**, + and data is removed by marking segments unused and running a kill task + ([§5.5](#55-druid-sql-cannot-write-and-the-server-says-so-clearly)). +2. **The JSON on the wire is not the JSON you would design.** The result is a *positional array + behind three header rows*, because the obvious object form silently drops duplicate columns + ([§3.4](#34-resultformat-array-because-the-object-form-loses-columns)); a 64-bit integer arrives as + an unquoted number that `JSON.parse` rounds, with no server-side setting to fix it + ([§3.6](#36-64-bit-integers-arrive-unquoted-and-druid-offers-no-server-side-fix)); and the error + body's `error` field is a *discriminator* rather than a message + ([§3.7](#37-two-error-envelopes-and-the-http-status-is-not-enough)). +3. **The HTTP status misclassifies, in both directions.** `SELECT 1/0` — an ordinary typo — answers + **HTTP 500** with `persona: "ADMIN"`. Every failure in this provider is classified by the + `category` Druid reports, never by the status code. + +### Concept mapping + +| `DatabaseProvider` slot | Druid realisation | Mechanism | +|---|---|---| +| "Table" (`TableSchema`) | A **datasource**, displayed by its bare name | `INFORMATION_SCHEMA.TABLES` where `TABLE_SCHEMA = 'druid'` | +| "Row" | One result row | One positional array element behind the header rows | +| Columns | The datasource's column list, SQL types verbatim | `INFORMATION_SCHEMA.COLUMNS` / the query response's header rows | +| Primary key | none — nothing in a datasource is unique | `isPrimary: false` on every column, `__time` included ([§6](#6-schema-introspection)) | +| `query(sql)` | One SQL statement, with `?` parameters bound | `POST /druid/v2/sql` | +| Indexes | none — every dimension is indexed inside its segment, with no index *object* | always `[]` | +| Foreign keys | none (Druid has none) | always `[]` | +| `getOverview()` / storage | Process identity, segment bytes, datasource count, running tasks | `sys.servers`, `sys.segments`, `INFORMATION_SCHEMA.TABLES`, `sys.tasks` | +| `getActiveSessions()` | **Ingestion tasks** — Druid has no query sessions | `sys.tasks` where `status IN ('RUNNING','PENDING')` | +| `getSlowQueries()` | nothing — Druid keeps no query log | always `[]` | +| Maintenance | nothing SQL can reach | `runMaintenance()` throws with the reason | + +--- + +## 2. Architecture + +### 2.1 Where it sits + +The database layer uses the **Strategy Pattern**. SQL providers add an intermediate abstract layer, +`SQLBaseProvider`, between the generic base and each concrete provider. Druid is a *directory* rather +than a single file, because the HTTP transport is a seam +([§3.2](#32-the-transport-seam-one-interface-one-implementation)) — the same layout ClickHouse uses: + +``` +src/lib/db/providers/sql/ +├── postgres.ts +├── sql-base.ts +├── clickhouse/ +└── druid/ + ├── index.ts # DruidProvider - the SQLBaseProvider subclass + ├── transport.ts # DruidTransport interface + neutral result/error types (no I/O) + ├── http-transport.ts # the one implementation: POST /druid/v2/sql + └── introspect.ts # INFORMATION_SCHEMA + sys.* reads +``` + +The explain strategy lives with the other strategies, not with the provider: +[`src/lib/explain/druid-native.ts`](../../src/lib/explain/druid-native.ts). + +### 2.2 Class hierarchy + +``` +DatabaseProvider (interface, types.ts) + ^ + | implements +BaseDatabaseProvider (abstract, base-provider.ts) + ^ + | extends +SQLBaseProvider (abstract, sql-base.ts) + ^ + | extends +DruidProvider (druid/index.ts) +``` + +`DruidProvider` extends `SQLBaseProvider` — not `BaseDatabaseProvider` directly, the way Couchbase +does — because the dialect really is standard on the points the shared helpers care about: +double-quoted identifiers and `LIMIT n OFFSET m` are both correct Druid SQL, live-verified. This is +the case [`docs/ADDING_A_PROVIDER.md`](../ADDING_A_PROVIDER.md) names ClickHouse for. Only +`prepareQuery()` is overridden, for the one trap in +[§3.9](#39-the-preparequery-override-offset-with-no-limit). + +### 2.3 What `SQLBaseProvider` gives for free + +| Member | Purpose | +|---|---| +| `escapeIdentifier()` | Double-quoted, since `this.type` (`druid`) falls through to the default branch — the same quoting PostgreSQL uses, and correct here: `SELECT "id" FROM "libredb_demo"` parses. Quoting is not optional in generated SQL — see the reserved-word trap in [§5.4](#54-dialect-traps-a-user-will-hit) | +| `buildLimitClause()` | `LIMIT n` / `LIMIT n OFFSET m`, both accepted by Druid | +| `getPlaceholder()` | Returns `?`, which is exactly what Druid's positional parameters use ([§3.10](#310-positional-parameters-really-execute)) | +| `shouldEnableSSL()` | Inherited but **never called**, deliberately. It infers TLS from substrings in the host name, which would silently switch a self-hosted cluster whose hostname merely contains one. TLS here comes from the connection's own `ssl` config only ([§4.3](#43-tls)) | +| `prepareQuery()` (base) | The shared query limiter; `DruidProvider` calls it first and only overrides the `OFFSET`-with-no-`LIMIT` case | +| `getLabels()` (base) | Everything but the two entity labels ([§9](#9-capabilities--labels)) | + +### 2.4 Registration & lifecycle + +The factory wires Druid in via a dynamic import ([`factory.ts:95`](../../src/lib/db/factory.ts)): + +```ts +case "druid": { + // The explicit /index specifier keeps this dynamic import statically + // analysable: a bare directory resolves only at runtime, which the bundler + // cannot trace into a chunk. + const { DruidProvider } = await import("./providers/sql/druid/index"); + return new DruidProvider(connection, options); +} +``` + +`connect()` proves the endpoint with one `SELECT 1`. That statement is live-verified as valid Druid +SQL — the planner answers it from a one-row inline datasource and names the column `EXPR$0` — so it +needs no datasource and succeeds on a cluster that has not ingested anything yet. Sending it at +connect time is what makes a wrong port, a proxy in front of the Broker, a Druid process that is not +a query endpoint, and a rejected credential surface while the user is still looking at the connection +form. `disconnect()` has nothing to release — there is no pool and no session — so it only clears the +cached transport reference. API routes use `getOrCreateProvider()`, which caches the connected +provider per `connection.id` and evicts it after 30 minutes idle. + +--- + +## 3. Design decisions + +These are the non-obvious choices. Read this section before changing the provider. + +### 3.1 HTTP only — no driver, and what that costs + +Druid ships a JDBC driver, but it addresses **Avatica** +(`jdbc:avatica:remote:url=http://host:8888/druid/v2/sql/avatica/`), which needs a JVM client library; +there is no Node client in Druid's own distribution. Everything this provider needs — querying, +`INFORMATION_SCHEMA`, `sys.*` monitoring, `EXPLAIN` — is reachable over the documented SQL HTTP +endpoint, so the provider speaks that and nothing else. `package.json` is untouched: there is no +install step to fail, no native module in the Docker image or in any distribution channel, and no +N-API compatibility question for the Bun runtime. This is the rubric in +[`docs/ADDING_A_PROVIDER.md`](../ADDING_A_PROVIDER.md) applied unchanged. + +**What it costs, stated plainly**, because a driver is not free of charge in the other direction +either: + +- **No failover and no retry.** One statement is one `fetch` to one host. A refused socket or a + Broker restart surfaces as an error rather than being retried against a second Broker. For an + interactive editor this is the right trade: the user sees the failure and presses the button again, + which is more honest than a silent retry that hides a degraded cluster. A Druid deployment that + wants failover puts the **Router** or a load balancer in front of its Brokers anyway, which is + exactly the host a Studio connection points at. +- **No cursor paging.** The response body is read to the end (`await response.text()`), scanned once + for unsafe integers, then parsed — so a result set is materialised as text and again as objects. + The editor's `LIMIT 500` injection ([§5.1](#51-execution)) is what keeps that bounded; a statement + that deliberately asks for millions of rows will be expensive here in a way a streaming client + would not be. +- **No prepared statements, no session state.** Avatica offers both; the SQL endpoint offers neither, + and neither is reachable from the editor's one-statement-per-execution model regardless. +- **No cancellation.** Abandoning the request client-side does not stop the query on the cluster, + which is why a server-side deadline is always sent alongside the client one + ([§3.8](#38-both-halves-of-the-timeout)). Druid *does* expose + `DELETE /druid/v2/sql/{sqlQueryId}`, so a real `cancelQuery` is a concrete follow-up rather than an + impossibility ([§13](#13-known-limitations--future-work)). + +### 3.2 The transport seam: one interface, one implementation + +Provider logic never calls `fetch`. It goes through `DruidTransport` +([transport.ts:156](../../src/lib/db/providers/sql/druid/transport.ts)), so adopting the Avatica +driver later — or any client that is not this endpoint — is one new file implementing the same +contract rather than a rewrite of the provider, the introspection and the explain strategy: + +```ts +interface DruidTransport { + readonly kind: "http"; + query(sql: string, opts?: DruidQueryOptions): Promise; + close(): Promise; +} +``` + +There is no second entry point next to `query()`, unlike Couchbase's `manage()`: every Druid metric, +task and storage statistic the provider needs is a `sys.*` table reachable by SQL +([§7](#7-monitoring--health)), so a permanent second HTTP surface would buy nothing. + +The result type is deliberately **neutral** rather than the wire envelope +([transport.ts:47](../../src/lib/db/providers/sql/druid/transport.ts)): + +```ts +interface DruidQueryResult { + rows: Record[]; + fieldNames: string[] | null; // declared order, made UNIQUE by the implementation + sqlTypes: Record | null; // BIGINT, VARCHAR, TIMESTAMP, ARRAY, ... - the trustworthy pair + nativeTypes: Record | null; // LONG, DOUBLE, ARRAY, COMPLEX, ... + executionTimeMs: number; // MEASURED here, never reported by the server +} +``` + +Three things this type says by what it omits: + +- **No mutation count.** Druid SQL has no statement that mutates + ([§5.5](#55-druid-sql-cannot-write-and-the-server-says-so-clearly)), so a count here could only ever + be zero — and a field that is always zero reads as "nothing changed" rather than "this cannot + happen". +- **No server-reported duration.** Live-verified: the endpoint answers with the rows and nothing + else — no timing in the body, none in the response headers, only query ids. The transport times its + own exchange, and the type's comment says so, so no reader mistakes it for the server's number. +- **`fieldNames` is required to be unique**, which is the transport's obligation rather than the + wire's ([§3.4](#34-resultformat-array-because-the-object-form-loses-columns)). + +> **Seam rule.** The wire vocabulary (`resultFormat`, `typesHeader`, `sqlTypesHeader`, +> `/druid/v2/sql`, `druidException`, `errorMessage`, `errorClass`, the `authorization` header, and +> `fetch` itself) must appear **only** in `http-transport.ts`. +> [`seam-guard.test.ts`](../../tests/unit/db/druid/seam-guard.test.ts) parses every source file in the +> directory with the TypeScript compiler API — not a grep — and fails the build the moment any of that +> vocabulary appears elsewhere, whether as a string, a property or a template. One token needs +> narrower treatment than ClickHouse's guard needed: the **neutral** error deliberately borrows +> Druid's own word `persona`, so `error.persona` is a legitimate read of `DruidTransportError` and is +> flagged only when spelled as a *string* (`body["persona"]`), which is what envelope parsing looks +> like. That hole is deliberate and is cheaper than a guard that cries wolf. + +### 3.3 Router 8888 or Broker 8082 — both work identically + +Live-verified on both ports of the same cluster: the same `POST /druid/v2/sql`, the same request +body, the same three-header-row envelope, the same error envelopes, and `sys.servers` returns the +same six rows from either. + +``` +$ curl -s -XPOST -H 'content-type: application/json' \ + -d '{"query":"SELECT COUNT(*) AS c FROM sys.servers","resultFormat":"array", + "header":true,"typesHeader":true,"sqlTypesHeader":true}' \ + http://localhost:8082/druid/v2/sql +[["c"],["LONG"],["BIGINT"],[6]] +``` + +**The Router (8888) is the default only because it fronts more**: the SQL API, the web console, and — +when `druid.router.managementProxy.enabled` is set — the Coordinator and Overlord APIs, so one port +is enough for both querying and loading data. **A Broker-only deployment needs no different +configuration**: point the connection at the Broker's port and everything in this document works, +including every monitoring panel. Nothing in the provider knows which of the two it is talking to. + +### 3.4 `resultFormat: "array"`, because the object form loses columns + +**This is a correctness decision, not a preference.** Druid's `resultFormat: "object"` returns rows +as JSON objects, and duplicate output names — legal SQL, and what a join projecting two `id`s +produces — collide. Live-verified, the raw response text for `SELECT 1 AS c, 2 AS c`: + +``` +resultFormat "object" -> [{"c":null,"c":null},{"c":1,"c":2}] +resultFormat "array" -> [["c","c"],["LONG","LONG"],["INTEGER","INTEGER"],[1,2]] +``` + +The object form puts the duplicate key in the text, where **every** JSON parser keeps only the last +occurrence: the first column disappears before any code can see it. The array form is positional, so +it keeps both, and column order becomes authoritative in a way object keys never are. + +The transport therefore always sends: + +```json +{ "query": "...", "resultFormat": "array", + "header": true, "typesHeader": true, "sqlTypesHeader": true } +``` + +and reads back **exactly three** header rows before the data — names, native types, SQL types, in +that order: + +```json +[["__time","snowflake_id","id","name","region","qty","amount","row_count"], + ["LONG","LONG","LONG","STRING","STRING","LONG","DOUBLE","LONG"], + ["TIMESTAMP","BIGINT","BIGINT","VARCHAR","VARCHAR","BIGINT","DOUBLE","BIGINT"], + ["2026-08-01T00:15:00.000Z",9007199254740993,1000,"alpha","emea",0,10.5,1]] +``` + +**A result set with no rows still carries all three header rows** (live-verified: +`SELECT id, name FROM libredb_demo WHERE 1=0` answers exactly +`[["id","name"],["LONG","STRING"],["BIGINT","VARCHAR"]]`), and a bare `SET` — the only other statement +form the grammar accepts — is rejected outright rather than answering short. So there is **no +legitimate way to receive fewer than three rows**, and the transport treats a shorter payload as a +failure rather than as an empty result. + +That distinction matters more than it looks. What actually produces a short payload is a truncated +body or a proxy that rewrote the response, and answering `{ rows: [] }` there would render the most +convincing possible lie: a successful query, over the right datasource, that happens to have found +nothing. Data loss has to surface as an error, which is the same reason the mid-response truncation +case above raises instead of returning what it managed to read. + +Rows are rebuilt from the declared names, and a repeat is **disambiguated rather than overwritten**: +`SELECT 1 AS c, 2 AS c` reaches the grid as columns `c` and `c (2)`, both with their values. The +suffix keeps climbing (`c (3)`, …) because `SELECT 1 AS c, 2 AS "c (2)", 3 AS c` is legal too. Without +this the array format would have been chosen and then thrown away one step later, since a row is a +`Record`. + +### 3.5 The SQL type labels the column, because the native type lies + +Druid publishes two type names per column and they disagree. Live-verified: + +| expression | native type | SQL type | value on the wire | +|---|---|---|---| +| `CURRENT_TIMESTAMP` | `LONG` | `TIMESTAMP` | `"2026-08-03T16:06:07.520Z"` | +| `(1 = 1)` | `LONG` | `BOOLEAN` | `true` | +| `ARRAY[1,2]` | `ARRAY` | `ARRAY` | `"[1,2]"` | +| `ARRAY['alpha']` | `ARRAY` | `ARRAY` | `"[\"alpha\"]"` | + +The native type is `LONG` for a value that is an ISO timestamp string and for a value that is +`true`, so **the SQL type is the trustworthy one of the pair**. The native type is kept alongside it +in the neutral result rather than dropped: it is the vocabulary a Druid user reads in the web console +and in a segment's dimension list, so discarding it would make the editor describe columns in words +the user's other tools never use. It is *carried*, not trusted. + +**Where each type actually reaches the screen**, because the two halves differ and it is easy to +assume otherwise: + +- **The schema tree** shows `INFORMATION_SCHEMA.COLUMNS.DATA_TYPE` — also a SQL type — which + introspection puts on `ColumnSchema.type`. This is the type a user sees today. +- **The result grid shows no per-column type at all**, for any provider. `QueryResult` + ([src/lib/types.ts](../../src/lib/types.ts)) is `{ rows, fields, rowCount, executionTime, + explainPlan?, pagination? }` — there is no channel a column type could travel in — so + `toQueryResult` drops both maps. They are collected because every Druid client knows them and + because the transport is the only place that *can* know them, not because something consumes them + yet. Giving `QueryResult` a type channel would serve every provider and is a follow-up, not part of + this one. + +Full observed surface: native `LONG`, `DOUBLE`, `FLOAT`, `STRING`, `ARRAY`, `ARRAY`, +`COMPLEX` against SQL `BIGINT`, `INTEGER`, `DOUBLE`, `FLOAT`, `DECIMAL`, `VARCHAR`, `CHAR`, +`TIMESTAMP`, `BOOLEAN`, `ARRAY`, `OTHER`, `NULL`. Note that `SELECT 1` is SQL `INTEGER` while a +`BIGINT` column is `BIGINT`: the SQL type is the *expression's* type, not a normalised family. + +### 3.6 64-bit integers arrive unquoted, and Druid offers no server-side fix + +Live-verified on real ingested data. `libredb_demo.snowflake_id` holds **9007199254740993** — that is +253 + 1, the first integer JavaScript cannot represent — and it comes back as the +**unquoted JSON number** `9007199254740993`: + +``` +[... ,["2026-08-01T00:15:00.000Z",9007199254740993,1000,"alpha","emea",0,10.5,1]] +``` + +`JSON.parse` turns that into `9007199254740992` with no error and no warning. A displayed id that is +off by one is worse than an error, because nothing signals it. + +**Druid has no server-side "quote longs" setting.** This is the one place the provider genuinely +diverges from ClickHouse (#264), which could push the problem to the server with +`output_format_json_quote_64bit_integers=1`. Here the only place left to fix it is the raw body, +before it is parsed — so `http-transport.ts` owns `quoteUnsafeIntegers()` +([http-transport.ts:264](../../src/lib/db/providers/sql/druid/http-transport.ts)), a single-pass, +**string-aware** scanner that wraps any integer literal outside +`Number.MIN_SAFE_INTEGER … Number.MAX_SAFE_INTEGER` in quotes. The value then reaches the UI as an +exact string — the same thing the `pg` driver already does for `int8`, so the grid renders it +correctly with no further change. + +Properties it upholds, each with its own test: + +- **Never rewrites inside a string literal.** A digit run in `"id: 9007199254740993"` is data the + user is reading. This includes a string containing an escaped quote (`"a\"9007199254740993"`) — the + escape must consume the next character, or the scanner desyncs and starts rewriting *inside* a + string, producing invalid JSON. +- **Leaves safe integers, floats, exponent forms and `-0` alone.** A float is a double on both sides, + so quoting one would turn a number the grid can sort into a string it cannot. Only integers lose + exactness. +- **Handles the literal in every position** a JSON body can put it: negative, as an object value, as + an array element, adjacent to `,` / `]` / `}` with and without whitespace. +- **Is a no-op on a body with no unsafe literal** — the common case returns the original string + without allocating a copy. +- **The comparison is on digits, not numbers.** Converting the run to a number to find out whether + converting it to a number is safe is the bug. A longer digit run always exceeds the range, and for + two runs of equal length a lexical comparison *is* the numeric one — so the check needs no + arithmetic at all. The sign never matters either, because the safe range is symmetric + (`MIN_SAFE_INTEGER === -MAX_SAFE_INTEGER`). + +Both encodings then reach every reader in the provider, because a `LONG` small enough stays an +unquoted number in the same response where a large `SUM(size)` arrives quoted. + +### 3.7 Two error envelopes, and the HTTP status is not enough + +Live-verified on 37.0.0. Both of these arrive from the same cluster, and the difference is not a +version skew — the modern shape is planning and validation, the legacy wrapper is a runtime failure a +data server reported. + +**`druidException`** — planning, validation, unsupported statements: + +```json +{ "error": "druidException", "errorCode": "invalidInput", "persona": "USER", + "category": "INVALID_INPUT", + "errorMessage": "Object 'nope' not found (line [1], column [15])", + "context": { "sourceType": "sql", "line": "1", "column": "15", + "endLine": "1", "endColumn": "18" } } +``` + +**Legacy-wrapped** — a runtime failure from a data server (here, a 1 ms deadline): + +```json +{ "error": "Query timeout", + "errorClass": "org.apache.druid.query.QueryTimeoutException", + "host": "172.18.0.5:8083", "errorCode": "legacyQueryException", + "persona": "OPERATOR", "category": "TIMEOUT", + "errorMessage": "url[http://172.18.0.5:8083/druid/v2/] timed out", + "context": { "host": "172.18.0.5:8083", + "errorClass": "org.apache.druid.query.QueryTimeoutException", + "legacyErrorCode": "Query timeout" } } +``` + +Four consequences the implementation honours: + +1. **`error` is a discriminator, not a message.** In the modern shape its value is the literal string + `"druidException"`. Showing `error` to a user prints *"druidException"* to the person who mistyped + a datasource name. The transport always reads **`errorMessage`**, and falls back to `error` only + when it is *not* that token (the legacy shape puts a real message in both). +2. **Classification is on `category`.** It is present in both shapes and is a closed enum, exported + frozen as `DRUID_ERROR_CATEGORIES` + ([transport.ts](../../src/lib/db/providers/sql/druid/transport.ts)) so no call site spells a token: + `INVALID_INPUT`, `UNAUTHORIZED`, `FORBIDDEN`, `CAPACITY_EXCEEDED`, `CANCELED`, `RUNTIME_FAILURE`, + `TIMEOUT`, `UNSUPPORTED`, `NOT_FOUND`, `UNCATEGORIZED`, `DEFENSIVE`. `errorCode` is secondary and + coarser (`invalidInput`, `general`, `legacyQueryException` — the same code arrives with different + categories), and the `errorClass` / `host` **fields** exist only in the legacy shape and are never + read: a Java class name adds nothing for the person who wrote the statement. + + One honest caveat, because the distinction is easy to misread: not reading the `host` field is not + the same as never showing the address. Druid writes it into `errorMessage` itself — the legacy + timeout above reads `url[http://172.18.0.5:8083/druid/v2/] timed out` — and that message is + surfaced verbatim. That is deliberate: which server timed out is the single most useful fact when + one Historical is slow, and it is an address inside a cluster the user is already connected to. + The alternative, preferring the legacy `error` value (`"Query timeout"`), is tidier and less + useful. +3. **HTTP 500 can be a plain user error.** Live-verified: + + ``` + $ curl -s -XPOST -H 'content-type: application/json' \ + -d '{"query":"SELECT 1/0"}' http://localhost:8888/druid/v2/sql + {"error":"druidException","errorCode":"general","persona":"ADMIN", + "category":"UNCATEGORIZED","errorMessage":"/ by zero","context":{}} + HTTP 500 + ``` + + Dividing by zero is reported as a 500 with `persona: "ADMIN"`. Reading 5xx as "the cluster is + broken" would tell the user something false and send them to check their host. `persona` is carried + for display only and **never branched on**, for exactly this reason — Druid's own guess at who + should read the message is wrong in the case that matters most. Timeout is **504** / + `category: TIMEOUT`; everything else observed is **400**. +4. **A body that is not JSON at all** — a proxy's HTML error page, an empty body, a body that stopped + arriving — still produces the seam's own error type, with a stand-in category + (`TRANSPORT_FAILURE`) that is deliberately **not** one of Druid's. Reusing `UNCATEGORIZED` for it, + tempting because that is what it means in English, would let a caller believe the server had + spoken and classified the failure when nothing ever answered. + +`DruidTransportError` +([transport.ts:256](../../src/lib/db/providers/sql/druid/transport.ts)) carries `category`, +`errorCode`, `persona` and the resolved message, and offers two predicates so no call site spells a +literal: + +- `is(category)` — takes the closed union, so a misspelling does not compile. +- `isMonitoringUnavailable()` — true for `UNAUTHORIZED`, `FORBIDDEN` and `NOT_FOUND` **only**. Those + three are the ordinary configurations of a locked-down cluster (`druid-basic-security` grants the + `sys` schema table by table) or of a build where the table is simply absent. Anything else must keep + propagating, or an empty monitoring panel hides the user's own mistake forever. + +`category` is typed as a plain `string` on the error rather than the closed union, so a category a +later Druid adds arrives verbatim instead of being flattened onto `UNCATEGORIZED` — which is itself a +real category. + +### 3.8 Both halves of the timeout + +Two deadlines, both derived from `queryTimeout` (default 60 s), because neither covers the other: + +- **Server side**: `context: { timeout: }`. Verified — `timeout: 1` answers 504 / + `category: TIMEOUT` on a statement that otherwise takes milliseconds. Asking the server to stop is + what actually frees the cluster's resources; abandoning the request client-side leaves the query + running. +- **Client side**: an `AbortSignal.timeout(...)` on the `fetch`, which also bounds the **body read**. + A server-side deadline only starts counting once the server has *accepted* the statement, so it + cannot bound a stalled DNS lookup, TCP connect or TLS handshake, or a response body that stops + arriving part-way. This is the #264 lesson applied. + +The client deadline is deliberately the **later** of the two (`queryTimeout + 5 s`, +`CLIENT_DEADLINE_GRACE_MS`): a client that gave up first would abandon a query that is still running +and report a bare abort instead of the 504 Druid was about to send, with its category and its +message. Catalog and `sys` reads use a shorter 15 s bound on both halves +([`DRUID_SYSTEM_READ_TIMEOUT_MS`](../../src/lib/db/providers/sql/druid/introspect.ts)) — a hanging +panel is worse than an empty one. + +### 3.9 The `prepareQuery()` override: `OFFSET` with no `LIMIT` + +The inherited limiter is otherwise correct for Druid: `LIMIT 500` appends cleanly, an existing `LIMIT` +is left alone, and the shared `applyQueryLimit` already strips and re-appends a trailing semicolon — +which matters, because Druid accepts `SELECT 1 AS c1;` but rejects `SELECT 1 AS c1; LIMIT 2`, and the +shared limiter never produces the latter. + +The one failure is a statement ending in **`OFFSET n` with no `LIMIT`**. The limiter appends `LIMIT` +after it, and Druid rejects the result outright (live-verified): + +``` +SELECT id FROM libredb_demo OFFSET 2 LIMIT 3 +-> 400 "'OFFSET start LIMIT count' is not allowed under the current SQL conformance level" +``` + +`DruidProvider.prepareQuery()` +([index.ts:218](../../src/lib/db/providers/sql/druid/index.ts)) asks `analyzeQuery()` — the shared +analyzer, not a regex of its own, because it already handles the trailing semicolon and already +distinguishes an `OFFSET` that follows a `LIMIT` (the ordinary paginated form, which the limiter +leaves alone anyway) from one that stands alone. When the statement has an `OFFSET` and no `LIMIT`, it +returns the query **untouched** with `wasLimited: false`. + +The bias is the same one ClickHouse's trailing-clause case takes, for the same reason: **rewriting +wrongly turns a working statement into a syntax error, while leaving it alone at worst returns more +rows than the page size.** + +### 3.10 Positional parameters really execute + +Unlike ClickHouse — whose HTTP interface binds only named `{name:Type}` parameters, so its provider +throws on positional ones — Druid takes `?` placeholders with a typed parameter list, live-verified: + +``` +{"query":"SELECT COUNT(*) AS c FROM libredb_demo WHERE region = ?", + "parameters":[{"type":"VARCHAR","value":"emea"}]} +-> [["c"],[20]] +``` + +So `query(sql, params)` binds rather than refuses, and `getPlaceholder()` needs no override. The +mapping: + +| JS value | Druid parameter type | Note | +|---|---|---| +| `string` | `VARCHAR` | | +| `number`, integral and safe | `BIGINT` | refused outside `Number.MAX_SAFE_INTEGER` — see below | +| `number`, non-integral | `DOUBLE` | | +| `bigint` | `BIGINT` | sent as a **raw unquoted literal** — see below | +| `boolean` | `BOOLEAN` | | +| `Date` | `TIMESTAMP` | epoch millis; verified, `{"type":"TIMESTAMP","value":0}` against `__time > ?` matches every row | +| `null` / `undefined` | `VARCHAR` with a `null` value | verified to execute and match the rows a null comparison should | + +Anything else (a plain object, an array, a symbol) raises an error naming the unsupported type, and +`Infinity` / `NaN` are refused explicitly because `JSON.stringify` turns both into `null`, which the +server would read as a null comparison. Refusing beats sending a value the server will misread. + +**The `bigint` case contradicted the plan, and the cluster won.** A `bigint` cannot be +`JSON.stringify`d, and the obvious encoding — send the digits as a string — is rejected: + +``` +{"type":"BIGINT","value":"9007199254740993"} -> 500 RUNTIME_FAILURE "Cannot handle query" +{"type":"BIGINT","value":9007199254740993} -> matches the row exactly +``` + +So the literal has to reach the body unquoted. `JSON.rawJSON` would do it, but it is the ES2025 JSON +source-text proposal — V8 12.4 / Node 22.2 — while `package.json` declares +`engines.node: ">=20.9.0"`, so depending on it would throw a bare `TypeError` on a runtime the +package claims to support. Instead the **parameters array is serialized by hand** and spliced into the +envelope at its closing brace, whose position is known because `JSON.stringify` just produced it. +Everything that is not a `bigint` still goes through `JSON.stringify`, so user strings are escaped by +the runtime rather than by us. + +That is deliberately structural rather than a marker-and-substitute pass. An earlier revision wrapped +the digits in a NUL sentinel and unquoted it with a regex over the finished body, which is unsound: a +sentinel is only as private as the values flowing through it, so a caller whose `VARCHAR` parameter +happened to contain that sentinel would have had their string silently unquoted into a number. +Emitting the literal in the first place cannot collide with anything, because no marker ever exists. + +**An integral `number` outside the safe range is refused,** which looks strict until you notice the +value is already wrong. A caller writing `9007199254740993` as a number literal handed the transport +`9007199254740992` — JavaScript rounded it before any of this code ran — and nothing here can recover +the digit. Sending it would filter on a value the user never wrote and return a plausible wrong row +set, so the error names the fix: pass a `bigint`, which binds exactly. The same check catches an +integral double past Druid's own `BIGINT` range. + +The design plan for this provider said a bigint travels "as a string value"; that is the one line of +it the live cluster disproved, and this is the record of why the code differs. + +### 3.11 The three `false` capabilities are each impossible, not merely unimplemented + +- **`supportsCreateTable: false`.** `CREATE TABLE t (id BIGINT)` is not unsupported — it is not in the + grammar. Live: `400`, *"Incorrect syntax near the keyword 'CREATE' at line 1, column 1."*, and the + parser then lists what it expected: `"INSERT"`, `"UPSERT"`, `"EXPLAIN"`, `"SET"`, `"RESET"`, … with + no form of `CREATE` among them. A datasource comes into existence by being ingested into. Per the + capability-honesty rule in [`docs/ADDING_A_PROVIDER.md`](../ADDING_A_PROVIDER.md), a flag that is + `true` but produces a control that can only emit invalid input is a defect, so it stays `false` and + the Create Table modal never appears. +- **`supportsMaintenance: false`, `maintenanceOperations: []`.** Nothing in `MaintenanceType` + (`vacuum`, `analyze`, `reindex`, `optimize`, `check`, `kill`) has a Druid analogue reachable from + SQL. Compaction and retention are **Coordinator and task** concerns, out of scope for #265. `kill` + is impossible for a second, independent reason: there is no `sys.queries` catalog, so there is + nowhere honest for a user to read a cancellable query id from. +- **`supportsConnectionString: false`.** See + [§4.2](#42-there-is-no-connection-string-and-that-is-deliberate). + +`schemaRefreshPattern` is `\b(INSERT|REPLACE)\b` — the only two statements that could change a +datasource. The native engine rejects both, so in practice a query never refreshes the schema, which +is *correct*: a Druid schema changes through ingestion, not through the editor. + +### 3.12 EXPLAIN: the native plan is genuinely a tree + +`ExplainFormat` gains `"druid-native"`, with the strategy in +[`src/lib/explain/druid-native.ts`](../../src/lib/explain/druid-native.ts). + +**What the plan is.** `EXPLAIN PLAN FOR