Skip to content

deployment.md

Latest commit

 

History

History
252 lines (180 loc) · 7.11 KB

deployment.md

File metadata and controls

252 lines (180 loc) · 7.11 KB
Raw

Manifold Deployment Guide

This guide describes the current deployment model for Manifold as shipped in this repository.

Manifold stores durable runtime state in SQLite by default and stores projects on the local filesystem under the workdir configured in config.yaml.

Deployment Modes

Recommended: Docker Compose

For a fresh clone, the supported first-run path is:

  • manifold for the Manifold server and the embedded frontend

Optional services can be added later:

  • keycloak-db and keycloak for authentication testing
  • clickhouse and otel-collector for persistent observability

The default dashboard does not require ClickHouse. When obs.local.enabled is true, Manifold serves metrics, logs, and traces from bounded process-local buffers.

Local Host Builds

Local host builds are supported through the Makefile, but they are a developer workflow, not the simplest deployment path. Use them when you want to:

  • build the standard Forge-backed Manifold binary with make build-manifold, which writes dist/manifold
  • iterate on server-only Go code with make build-agentd when you do not need to refresh embedded frontend assets
  • run the frontend separately with pnpm -C web/agentd-ui dev

Host builds use SQLite by default and do not require an external database service. Postgres remains supported only through an explicitly configured external endpoint.

Prerequisites

Required for Docker deployment:

  • Docker with Docker Compose support
  • A valid LLM API key or a reachable OpenAI-compatible endpoint
  • A writable absolute host path for workdir

Required only for local development outside Docker:

  • Go 1.26.3
  • Node 22
  • pnpm
  • Chrome or another Chromium-compatible browser for browser-driven tools

First Run

  1. Copy the templates:
cp example.env .env
cp config.yaml.example config.yaml
cp specialists.yaml.example specialists.yaml
cp mcp.yaml.example mcp.yaml
mkdir -p ./tmp/manifold-workdir
  1. Edit config.yaml and set the main runtime values. At minimum, ensure:
workdir: /absolute/path/to/your/manifold/tmp/manifold-workdir

llm_client:
  provider: openai
  openai:
    apiKey: "${OPENAI_API_KEY}"

databases:
  backend: sqlite
  defaultDSN: ""
  sqlite:
    path: "~/.manifold/manifold.db"
  1. Edit .env and set the secret values referenced by your YAML:
OPENAI_API_KEY="your_real_api_key"
DATABASE_URL=""
CLICKHOUSE_DSN=""
  1. Start the required services:
docker compose up -d manifold
  1. Open the UI at http://localhost:32180.

Self-Contained Host Deployment

This mode runs without Compose services, external Postgres, ClickHouse, or an OpenTelemetry Collector. It is useful for local single-machine installs and development environments where you want durable Manifold state without managing a separate database server.

Prerequisites:

  • Go toolchain for building manifold
  • An LLM provider, either remote or local OpenAI-compatible

Build:

make build-manifold

Set .env values so no external database or telemetry service is selected:

DATABASE_URL=""
CLICKHOUSE_DSN=""
OPENAI_API_KEY="your_real_api_key"

Configure config.yaml:

databases:
  backend: sqlite
  defaultDSN: ""
  sqlite:
    path: "~/.manifold/manifold.db"
  chat:
    backend: sqlite
  search:
    backend: sqlite
  vector:
    backend: sqlite
  graph:
    backend: sqlite

obs:
  otlp: ""
  local:
    enabled: true
  clickhouse:
    dsn: ""

Run:

./dist/manifold storage doctor --json
./dist/manifold

SQLite durable state is stored at ~/.manifold/manifold.db unless databases.sqlite.path is set. The storage doctor verifies SQLite open, FTS5, Vec1 registration, WAL mode, and a temporary vector query.

Optional Postgres Deployment

Use Postgres for multi-host deployments or when keeping existing Postgres data:

databases:
  backend: postgres
  defaultDSN: "${DATABASE_URL}"
  chat:
    backend: postgres
    dsn: "${DATABASE_URL}"
  search:
    backend: postgres
    dsn: "${DATABASE_URL}"
  vector:
    backend: postgres
    dsn: "${DATABASE_URL}"
  graph:
    backend: postgres
    dsn: "${DATABASE_URL}"
DATABASE_URL="postgres://manifold:manifold@pg-manifold:5432/manifold?sslmode=disable"

Then start Postgres with Manifold:

docker compose up -d pg-manifold manifold

Service Map

Core services:

  • manifold: the main Manifold container with the embedded web UI

Optional services:

  • pg-manifold: PostgreSQL with pgvector, PostGIS, and pgRouting
  • clickhouse: optional persistent metrics, traces, and logs query backend
  • otel-collector: optional OTLP ingestion pipeline
  • keycloak-db: Postgres for Keycloak
  • keycloak: local OIDC provider for auth testing

Ports

  • 32180: Manifold UI and API
  • 5433: PostgreSQL exposed on the host
  • 8083: Keycloak admin and auth UI when enabled
  • 8123 and 9000: ClickHouse HTTP and native ports when enabled
  • 4417 and 4418: OTLP collector ports when enabled

When optional Postgres is enabled, Manifold connects to Postgres at pg-manifold:5432 inside the compose network. Host tools should use localhost:5433.

Configuration Notes

  • config.yaml is the primary runtime configuration file. .env is only used to supply values for ${VAR} interpolation inside YAML.
  • The runtime validates that workdir exists and is a directory.
  • config.yaml.example is the full runtime reference. specialists.yaml.example and mcp.yaml.example document the optional external specialist and MCP config files.
  • Leave DATABASE_URL empty for the default SQLite backend.
  • For optional Postgres, set databases.backend: postgres, wire databases.defaultDSN and per-subsystem DSNs to ${DATABASE_URL}, and start pg-manifold.
  • obs.local.enabled: true gives the dashboard local process telemetry without ClickHouse or an OpenTelemetry Collector. obs.clickhouse.dsn upgrades the dashboard to persistent telemetry when ClickHouse is available.
  • If you use llm_client.openai.api: responses, you can tune context-management behavior through llm_client.openai.extraParams in config.yaml.example.
  • Voice input requires an OpenAI-compatible transcription endpoint through the stt section in config.yaml.example.

Storage Model

Projects are stored directly on disk under:

$workdir/users/<user-id>/projects/<project-id>

Metadata is stored at:

$workdir/users/<user-id>/projects/<project-id>/.meta/project.json

See storage.md for details.

Auth And Observability

  • Authentication is optional and disabled by default. See auth.md.
  • Local dashboard observability is enabled by default through process-local buffers. ClickHouse and OTLP export are optional for persistent or external observability. See observability.md.

Backup And Recovery

Back up:

  • SQLite database file from databases.sqlite.path, default ~/.manifold/manifold.db
  • PostgreSQL data from pg-manifold when Postgres is explicitly configured
  • the entire workdir

At minimum, a recoverable local deployment needs both the database state and the project filesystem.