Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn

<p align="center">
<img src="public/screenshots/connection-modal.png" alt="Multi-Database Connection Manager" width="100%" />
<br/><em>Connect to PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Redis, or SQLite with SSL/TLS and SSH Tunnel support.</em>
<br/><em>Connect to PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis, or SQLite with SSL/TLS and SSH Tunnel support.</em>
</p>

---
Expand Down Expand Up @@ -185,9 +185,10 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn
| **MongoDB** | `mongodb` | JSON query editor, collection operations (find, aggregate, insert, update, delete) |
| **Couchbase** | none — HTTP (Query + management REST) | Full SQL++ IDE, EXPLAIN plans, bucket/scope/collection explorer, `INFER` column inference, read-your-writes consistency, `UPDATE STATISTICS` / `BUILD INDEX` / request kill |
| **ClickHouse** | none — HTTP (SQL interface, port 8123) | Full SQL IDE, JSON EXPLAIN plan trees, system-table schema introspection, `OPTIMIZE TABLE` / table statistics / query kill maintenance |
| **Apache Druid** | none — HTTP (`POST /druid/v2/sql`, Router port 8888 or Broker 8082) | Read-only SQL IDE, native-query EXPLAIN plan trees, `INFORMATION_SCHEMA` datasource introspection, `sys.*` monitoring (segments, servers, ingestion tasks). Druid SQL has no `UPDATE`, no `DELETE` and no `CREATE TABLE`, and nothing it can do counts as a maintenance operation — a datasource changes through ingestion, not from the editor |
| **Redis** | `ioredis` | Command editor, key browser, INFO-based monitoring |

> All SQL databases share: schema explorer, ER diagrams, schema diff & migration, display masking (preview), monitoring dashboard, and connection string import.
> All SQL databases share: schema explorer, ER diagrams, schema diff & migration, display masking (preview), monitoring dashboard, and connection string import. Druid is the exception twice over: its HTTP SQL API has no URI convention to paste, so it is configured by host and port only, and generated migration SQL has nothing to apply against an engine whose SQL contains no DDL.

> **Provider reference docs:** each database has an in-depth reference (design, connection, query format, monitoring, limitations) under [`docs/providers/`](docs/providers/README.md). For the provider architecture see [`docs/DATABASE_PROVIDERS.md`](docs/DATABASE_PROVIDERS.md), and to add a new database see [`docs/ADDING_A_PROVIDER.md`](docs/ADDING_A_PROVIDER.md).

Expand All @@ -203,7 +204,7 @@ The test instance comes with a pre-configured PostgreSQL database via [Seed Conn
| **Editor** | Monaco Editor (VS Code Engine) | Web |
| **AI** | Multi-Model (Gemini, OpenAI, Ollama, Custom) | Web, Mobile |
| **Auth** | JWT (`jose`) + OIDC (`openid-client`), PKCE, Role Mapping | Web, Mobile |
| **Database** | PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Redis | Web, Mobile |
| **Database** | PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Apache Druid, Redis | Web, Mobile |
| **Charts** | Recharts (Bar, Line, Pie, Area, Scatter, Histogram, Stacked) | Web, Mobile |
| **ERD** | React Flow, ELK.js (auto-layout) | Web |
| **State/Grid** | TanStack Table & Virtual | Web, Mobile |
Expand Down Expand Up @@ -294,7 +295,7 @@ journalctl -u libredb-studio

### Prerequisites
- [Bun](https://bun.sh/) (Recommended) or Node.js 24+
- A target database to query (PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, or Redis)
- A target database to query (PostgreSQL, MySQL, Oracle, SQL Server, SQLite, MongoDB, Couchbase, ClickHouse, Apache Druid, or Redis)

### Quick Start (Local)
1. **Clone & Install**
Expand Down Expand Up @@ -340,14 +341,21 @@ journalctl -u libredb-studio
Need databases to test with? We provide ready-to-use containers for all supported engines:

```bash
# Start all development databases (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle)
# Start every default-profile database (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, ...)
docker compose -f database-compose.yml up -d

# Or start a specific database
docker compose -f database-compose.yml up -d postgres
docker compose -f database-compose.yml up -d mssql
docker compose -f database-compose.yml up -d oracle

# Apache Druid: profile-gated, so a bare `up -d` does NOT start it. Druid is a distributed
# system with no single-container mode - five Druid processes plus ZooKeeper plus its own
# metadata database is the minimum that can answer a SQL query, so all seven services carry
# `profiles: [druid]` rather than doubling the default stack. Connect to the Router on 8888
# (or the Broker on 8082 - the same endpoint, no different configuration).
docker compose -f database-compose.yml --profile druid up -d

# Start PostgreSQL with sample e-commerce data
docker compose -f docker/postgres.yml up -d

Expand All @@ -356,6 +364,9 @@ docker compose -f database-compose.yml down

# Stop and remove all data
docker compose -f database-compose.yml down -v

# The Druid containers need the profile flag here too - without it `down` leaves them running
docker compose -f database-compose.yml --profile druid down -v
```

### Connection Details
Expand All @@ -367,6 +378,7 @@ docker compose -f database-compose.yml down -v
| **SQL Server** | localhost | 1433 | sa | Password123! | master |
| **Oracle** | localhost | 1521 | system | Password123! | freepdb1 |
| **MongoDB** | localhost | 27017 | admin | admin | — |
| **Apache Druid** | localhost | 8888 (Router) or 8082 (Broker) | — | — | — (one catalog, always `druid`) |

### PostgreSQL Sample Data

Expand Down Expand Up @@ -416,7 +428,7 @@ bun run test:coverage
|-------|-----------|--------|-------|----------------|
| **Unit** | `tests/unit/` | `bun:test` | ~1,609 | Pure functions: SQL parser, connection strings, data masking, query limiter, schema diff, error classes, DB icons, showcase queries |
| **API** | `tests/api/` | `bun:test` | ~279 | Route handlers: auth, query, transaction, maintenance, AI endpoints, middleware |
| **Integration** | `tests/integration/` | `bun:test` | ~346 | Database providers: PG, MySQL, SQLite, MongoDB, Couchbase, Redis, Oracle, MSSQL, ClickHouse|
| **Integration** | `tests/integration/` | `bun:test` | ~346 | Database providers: PG, MySQL, SQLite, MongoDB, Couchbase, Redis, Oracle, MSSQL, ClickHouse, Druid|
| **Hooks** | `tests/hooks/` | `bun:test` | ~251 | React hooks: auth, connections, tabs, query execution, transactions, inline editing, AI chat, monitoring |
| **Components** | `tests/components/` | `bun:test` + happy-dom | ~570 | UI components: Studio, Sidebar, QueryEditor, ResultsGrid, Admin Dashboard, Charts, ERD |
| **E2E** | `e2e/` | Playwright | ~32 | Full browser flows: login, connections, query execution, tabs, export, admin |
Expand Down Expand Up @@ -707,7 +719,7 @@ extraEnvFrom:
| `defaults` | No | Default values merged into all connections |
| `connections[].id` | Yes | Unique slug (`[a-z0-9-]+`, max 64 chars) |
| `connections[].name` | Yes | Display name in UI |
| `connections[].type` | Yes | `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse` |
| `connections[].type` | Yes | `postgres`, `mysql`, `sqlite`, `mongodb`, `redis`, `oracle`, `mssql`, `libredb`, `couchbase`, `clickhouse`, `druid` |
| `connections[].roles` | Yes | `["*"]` (everyone), `["admin"]`, `["user"]`, or `["admin", "user"]` |
| `connections[].managed` | No | `true` = read-only (default), `false` = editable copy for user |
| `connections[].password` | No | Use `${ENV_VAR}` syntax for secrets |
Expand Down Expand Up @@ -745,7 +757,7 @@ extraEnvFrom:
- [ ] **Phase 17**: Enterprise Collaboration (User Identity, Shared Workspaces, SAML 2.0).
- [ ] **Phase 18**: Server-Enforced Data Masking (SQL output-lineage, deployment-global policy, fail-closed API masking, alias/aggregate coverage).
- [x] **Phase 19**: Driver-Free Providers — Couchbase (SQL++ over the Query REST API), the first provider that adds no runtime dependency. Pattern documented in [Adding a Provider](docs/ADDING_A_PROVIDER.md).
- [ ] **Phase 20**: Analytics Databases — ClickHouse ([#264](https://github.com/libredb/libredb-studio/issues/264)) and Apache Druid ([#265](https://github.com/libredb/libredb-studio/issues/265)), both driver-free over HTTP.
- [x] **Phase 20**: Analytics Databases — ClickHouse ([#264](https://github.com/libredb/libredb-studio/issues/264)) and Apache Druid ([#265](https://github.com/libredb/libredb-studio/issues/265)), both driver-free over HTTP. Druid is read-only by nature — no `UPDATE`, no `DELETE`, no `CREATE TABLE` — so it also demonstrates a provider that reports absent capabilities honestly instead of offering controls that can only fail.
- [ ] **Phase 21**: Federated Query — Trino/Starburst. Deliberately unscheduled: a Trino catalog is another *system*, so what a connection pins is a product question that has to be answered before the work can be specified.

---
Expand Down
205 changes: 205 additions & 0 deletions database-compose.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,39 @@
# Shared configuration for every Druid process. Druid is configured entirely through
# `druid_*` environment variables (the image's entrypoint translates them into
# runtime.properties), so one anchor keeps the five processes provably identical - a cluster
# whose processes disagree about ZooKeeper or the metadata store fails in confusing ways.
x-druid-env: &druid-env
# Sizes every process for a laptop. Without it each one claims cluster-scale heap and
# direct memory, and the five together will not fit.
DRUID_SINGLE_NODE_CONF: micro-quickstart
# druid-multi-stage-query is included on purpose: it is what makes `INSERT INTO ... SELECT`
# (MSQ) available on /druid/v2/sql/task. The provider does not use that endpoint - the point
# is that the live pass can prove the *synchronous* endpoint rejects INSERT even when the
# engine is fully capable of it, which is the honest reason supportsCreateTable is false.
druid_extensions_loadList: '["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]'
druid_zk_service_host: druid-zookeeper
druid_metadata_storage_host: ""
druid_metadata_storage_type: postgresql
druid_metadata_storage_connector_connectURI: jdbc:postgresql://druid-metadata:5432/druid
druid_metadata_storage_connector_user: druid
druid_metadata_storage_connector_password: druid
# Deep storage on a shared local volume. Real deployments use S3/HDFS; local is what makes
# this a single-machine fixture, and the historical and middlemanager must see the same files.
druid_storage_type: local
druid_storage_storageDirectory: /opt/shared/segments
druid_indexer_logs_type: file
druid_indexer_logs_directory: /opt/shared/indexing-logs
druid_indexer_runner_javaOptsArray: '["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=1g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'
druid_indexer_fork_property_druid_processing_buffer_sizeBytes: 100MiB
druid_processing_numThreads: "2"
druid_processing_numMergeBuffers: "2"
# Lets the Router at 8888 proxy Coordinator and Overlord APIs, so the whole cluster is
# reachable through the one port the provider targets.
druid_router_managementProxy_enabled: "true"
# info, not the upstream compose's debug: the debug logger emits one line per internal
# request and buries the errors a live pass is looking for.
druid_emitter_logging_logLevel: info

services:
postgres:
image: postgres:18
Expand Down Expand Up @@ -163,3 +199,172 @@ services:
# No seed sidecar: CLICKHOUSE_DB above creates `demo` on first boot, and unlike Couchbase
# there is no index prerequisite - a SELECT against a freshly created table works
# immediately. Do not add one.

# ---------------------------------------------------------------------------------------
# Apache Druid. Unlike every other service here, Druid is a distributed system and has no
# single-container mode: five Druid processes plus ZooKeeper plus a metadata database is
# the minimum that can answer a SQL query. That is why all seven carry `profiles: [druid]`
# - a bare `docker compose -f database-compose.yml up -d` would otherwise grow from 7
# containers to 14 and from roughly 6 GB of RAM to 12. Start it explicitly:
#
# docker compose -f database-compose.yml --profile druid up -d
#
# The provider only ever talks to the Router (8888) or the Broker (8082); the other five
# processes are cluster internals and deliberately publish no host port. 8091, the
# MiddleManager's usual port, is already taken by couchbase above - another reason not to.
# ---------------------------------------------------------------------------------------
druid-metadata:
# Druid's metadata store (segment table, task history, supervisors, config). Distinct
# from the `postgres` service above on purpose: sharing it would put Druid's ~20 internal
# tables into the demo database the studio's own postgres connection browses.
image: postgres:17.6
container_name: libredb-druid-metadata
profiles: ["druid"]
restart: unless-stopped
environment:
POSTGRES_USER: druid
POSTGRES_PASSWORD: druid
POSTGRES_DB: druid
# No published port: only the Druid Coordinator/Overlord reads it, over the compose
# network. Publishing it would collide with the `postgres` service on 5432.
healthcheck:
test: ["CMD-SHELL", "pg_isready -U druid -d druid"]
interval: 5s
timeout: 5s
retries: 30
druid-zookeeper:
# Druid's cluster coordination: process discovery, segment load queues, task assignment.
image: zookeeper:3.9.3
container_name: libredb-druid-zookeeper
profiles: ["druid"]
restart: unless-stopped
environment:
ZOO_MY_ID: "1"
# zkServer.sh's own status check uses the `srvr` four-letter word, which 3.9 refuses
# unless it is whitelisted.
ZOO_4LW_COMMANDS_WHITELIST: "srvr,ruok,conf"
healthcheck:
test: ["CMD-SHELL", "echo ruok | nc -w 2 localhost 2181 | grep -q imok"]
interval: 5s
timeout: 5s
retries: 30
druid-coordinator:
# Coordinator + Overlord in one process (the image's coordinator config sets
# druid.coordinator.asOverlord.enabled), so segment balancing and task management both
# live here. This is what accepts an ingestion task, which is how a datasource is
# created at all - Druid has no CREATE TABLE.
image: apache/druid:37.0.0
container_name: libredb-druid-coordinator
profiles: ["druid"]
restart: unless-stopped
command: ["coordinator"]
environment: *druid-env
volumes:
- druid_shared:/opt/shared
- druid_coordinator_var:/opt/druid/var
depends_on:
druid-metadata:
condition: service_healthy
druid-zookeeper:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8081/status/health"]
interval: 10s
timeout: 5s
retries: 40
druid-historical:
# Serves published segments. Without it a datasource exists in the metadata store but
# returns no rows, which is the "datasource with no segments" case the provider has to
# render honestly.
image: apache/druid:37.0.0
container_name: libredb-druid-historical
profiles: ["druid"]
restart: unless-stopped
command: ["historical"]
environment: *druid-env
volumes:
- druid_shared:/opt/shared
- druid_historical_var:/opt/druid/var
depends_on:
druid-coordinator:
condition: service_started
druid-zookeeper:
condition: service_healthy
druid-middlemanager:
# Runs ingestion tasks as forked peon processes. Needed for the live pass to load data.
image: apache/druid:37.0.0
container_name: libredb-druid-middlemanager
profiles: ["druid"]
restart: unless-stopped
command: ["middleManager"]
environment: *druid-env
volumes:
- druid_shared:/opt/shared
- druid_middlemanager_var:/opt/druid/var
depends_on:
druid-coordinator:
condition: service_started
druid-zookeeper:
condition: service_healthy
druid-broker:
# Plans and merges queries. POST /druid/v2/sql here is the same API the Router proxies,
# so the provider works against 8082 directly too - published so that can be proven
# rather than assumed.
image: apache/druid:37.0.0
container_name: libredb-druid-broker
profiles: ["druid"]
restart: unless-stopped
command: ["broker"]
environment: *druid-env
volumes:
- druid_broker_var:/opt/druid/var
ports:
- "8082:8082"
depends_on:
druid-coordinator:
condition: service_started
druid-zookeeper:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8082/status/health"]
interval: 10s
timeout: 5s
retries: 40
druid-router:
# The provider's default target. It fronts the SQL API, the web console and - because
# druid_router_managementProxy_enabled is set - the Coordinator and Overlord APIs, so a
# single port is enough for both querying and loading data.
image: apache/druid:37.0.0
container_name: libredb-druid-router
profiles: ["druid"]
restart: unless-stopped
command: ["router"]
environment: *druid-env
volumes:
- druid_router_var:/opt/druid/var
ports:
- "8888:8888"
depends_on:
druid-broker:
condition: service_started
druid-coordinator:
condition: service_started
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8888/status/health"]
interval: 10s
timeout: 5s
retries: 40

volumes:
# Named volumes only for Druid: it is the one service here whose processes must share
# state (deep storage) and whose task history is worth surviving a restart. Everything
# above uses the image's own anonymous volumes.
#
# druid_shared is the deep-storage + indexing-log volume, mounted by the three processes
# that write or read segments. A `down -v` resets the cluster to empty.
druid_shared: {}
druid_coordinator_var: {}
druid_historical_var: {}
druid_middlemanager_var: {}
druid_broker_var: {}
druid_router_var: {}
Loading
Loading