diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..0ac1077 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,57 @@ +# Inference Bazaar — agent rules + +## ⛔ Deploy the Blueprint Manager, NOT the operator binary + +This is the #1 recurring mistake. **Do not** run or `ExecStart` `inference-bazaar-operator run` +directly, with a hardcoded `SERVICE_ID` / `TEST_MODE=true`. That bypasses the entire +on-chain lifecycle and is a dev shortcut only. + +A production operator box runs the **Blueprint Manager daemon**: + +``` +ExecStart=/root/.cargo/bin/cargo-tangle blueprint run -t --pretty \ + --http-rpc-url --ws-rpc-url \ + --keystore-uri --data-dir /bpm-data \ + --chain testnet --protocol tangle +# systemd: SyslogIdentifier=blueprint-manager +``` + +The manager watches the chain and **spawns the per-service `inference-bazaar-operator run` +instance itself** when a service request is approved. `inference-bazaar-operator run` +(`operator/src/bin/blueprint.rs`) is the INSTANCE the manager spawns — it reads `SERVICE_ID` +from the env the manager hands it; it is never launched by hand. + +**The lifecycle is two-sided and on-chain — honor every step:** + +1. Deploy the Blueprint Manager (the daemon above) on each operator box. +2. The operator **registers** for the blueprint on-chain (`cargo tangle blueprint register`). +3. A **user requests a service** selecting registered operators (`request-service`). +4. Operators **approve**. +5. The already-running Blueprint Manager **spawns the instance** with the assigned service id. + +Reference that does this correctly: `~/code/ai-trading-blueprint/deploy/go-live.sh` + +`trading-blueprint.service`. When you write any deploy unit / script / doc, the `ExecStart` +MUST be `cargo-tangle blueprint run`, and the runbook MUST include register → request → approve. + +`InferenceBazaarBSM.sol` (`[blueprint.manager]`) is the ON-CHAIN Blueprint Service Manager +(lifecycle hooks/slashing) — a different thing from the off-chain manager daemon. Don't conflate. + +## Build / toolchain + +- Rust is pinned to **1.91** (`rust-toolchain.toml`). The `--features blueprint` build needs + **`protoc`** (a transitive dep): `PROTOC=/usr/bin/protoc cargo build … --features blueprint`. +- CI gates: `cargo fmt --all --check`, core-crate `clippy -- -D warnings` + (`orderbook/settlement-core/matcher/settlement`), `cargo test --workspace`, `forge test`. + +## Operator inference backend (the "resell Tangle models" flow) + +To resell the Tangle Router's models, set `INFERENCE_BAZAAR_INFERENCE_URL=https://router.tangle.tools/v1` ++ `INFERENCE_BAZAAR_INFERENCE_API_KEY=` → backend `mode="external"`, which passes the +bonded-issuer check. The naked `INFERENCE_BAZAAR_ROUTER_URL` fallback is `mode="router"` and is +**refused** for a bonded issuer (`operator/src/venue.rs`). + +## Other gotchas + +- **Local git branch refs go stale.** Always `git fetch` and check `origin/` before + claiming something is unmerged — this has produced wrong "not merged" conclusions repeatedly. +- The fleet is on Base **Sepolia**, not mainnet. The "USDC rail" RPC still points at testnet. diff --git a/deploy/gen-resell-operator.sh b/deploy/gen-resell-operator.sh new file mode 100755 index 0000000..7766002 --- /dev/null +++ b/deploy/gen-resell-operator.sh @@ -0,0 +1,119 @@ +#!/usr/bin/env bash +# Generate a full-blueprint "resell" operator for the Hetzner fleet: the +# Blueprint Manager systemd unit + the operator's settings.env. The operator +# resells the Tangle Router's centralized models at a discount. +# +# ⛔ PRODUCTION = THE BLUEPRINT MANAGER, NOT THE INSTANCE BINARY. +# This does NOT ExecStart `inference-bazaar-operator run` — that (with a hardcoded +# SERVICE_ID / TEST_MODE) is for LOCAL TESTING ONLY. The unit runs the Blueprint +# Manager daemon (`cargo-tangle blueprint run`); the manager watches the chain and +# SPAWNS the per-service instance itself when a user's service request is approved. +# See /home/drew/code/bazaar/CLAUDE.md and ai-trading-blueprint/deploy/go-live.sh. +# +# WHY the resell backend is valid (not the forbidden mode): the settings.env points +# INFERENCE_BAZAAR_INFERENCE_URL at the router with the operator's own key → backend +# mode="external", which passes the bonded-issuer check (operator/src/venue.rs). The +# naked ROUTER_URL fallback would be mode="router" and is refused for an issuer. +# +# What you must provide (NOT in the repo — human-gated): +# OP_N unique index for paths/unit name (e.g. 5; 3/4 already live) +# OPERATOR_KEY EVM attester/signing key, 0x-hex (settlement attester; the +# Tangle operator identity is imported into the keystore separately) +# SUBMITTER_KEY gas/submitter key, 0x-hex (keep distinct from OPERATOR_KEY) +# ROUTER_API_KEY this operator's OWN Tangle Router API key (the resell credential) +# INSTRUMENT market to make, e.g. anthropic/claude-opus-4-8:output +# +# Usage: +# OP_N=5 OPERATOR_KEY=0x.. SUBMITTER_KEY=0x.. ROUTER_API_KEY=tngl-.. \ +# INSTRUMENT=anthropic/claude-opus-4-8:output deploy/gen-resell-operator.sh +set -euo pipefail +cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +: "${OP_N:?set OP_N -- unique operator index, e.g. 5}" +: "${OPERATOR_KEY:?set OPERATOR_KEY -- 0x-hex EVM attester key}" +: "${ROUTER_API_KEY:?set ROUTER_API_KEY -- this operator Tangle Router API key}" +SUBMITTER_KEY="${SUBMITTER_KEY:-$OPERATOR_KEY}" +INSTRUMENT="${INSTRUMENT:-anthropic/claude-opus-4-8:output}" +ROUTER_URL="${ROUTER_URL:-https://router.tangle.tools}" +SETTLEMENT_ADDR="${INFERENCE_BAZAAR_SETTLEMENT_ADDR:-0x64867eacf2e4581d182c2Be634cfD7fF3D3d9f83}" +BLUEPRINT_ID="${BLUEPRINT_ID:-17}" +HTTP_RPC="${HTTP_RPC_URL:-https://base-sepolia-rpc.publicnode.com/}" +WS_RPC="${WS_RPC_URL:-wss://base-sepolia.drpc.org/}" +ROOT="/opt/inference-bazaar${OP_N}" +SETTINGS="deploy/hetzner/inference-bazaar${OP_N}.settings.env" +UNIT="deploy/hetzner/inference-bazaar${OP_N}-blueprint-manager.service" + +# ── the operator's settings.env — the env the manager hands the spawned instance. +# NO SERVICE_ID here: the manager assigns it when the service request is approved. +cat > "$SETTINGS" < "$UNIT" </dev/null && cast wallet address --private-key "$OPERATOR_KEY" 2>/dev/null || echo "") + +cat < wrote ${UNIT} +==> wrote ${SETTINGS} (operator ${OP_N}, reselling ${ROUTER_URL} as ${INSTRUMENT}) + operator address: ${OP_ADDR} + +Runbook (⚠ = human-gated). The manager spawns the instance — you never run it by hand: + +1. ⚠ Fund ${OP_ADDR} (gas + collateral token) and post collateral: + COLLATERAL_AMOUNT=20000000 OPERATOR_KEY=${OPERATOR_KEY} deploy/onboard-operator.sh +2. Install on 178.104.232.124 (the manager + its keystore/settings): + ssh root@178.104.232.124 'mkdir -p ${ROOT}/keystore ${ROOT}/bpm-data' + scp ${SETTINGS} root@178.104.232.124:${ROOT}/settings.env + scp ${UNIT} root@178.104.232.124:/etc/systemd/system/ +3. ⚠ Import the operator's Tangle identity into the keystore + register for the blueprint: + cargo tangle blueprint register --blueprint-id ${BLUEPRINT_ID} --keystore-uri ${ROOT}/keystore +4. Start the manager (it now waits for a service request): + ssh root@178.104.232.124 'systemctl daemon-reload && systemctl enable --now inference-bazaar${OP_N}-blueprint-manager' +5. ⚠ A USER requests a service selecting these operators; operators APPROVE: + cargo tangle blueprint request-service --blueprint-id ${BLUEPRINT_ID} # the buyer/demand side + # operators approve -> the running manager SPAWNS the instance with the assigned service id. + +RUNBOOK diff --git a/deploy/hetzner/inference-bazaar-blueprint-runtime.service b/deploy/hetzner/inference-bazaar-blueprint-runtime.service index ed7db03..f66692e 100644 --- a/deploy/hetzner/inference-bazaar-blueprint-runtime.service +++ b/deploy/hetzner/inference-bazaar-blueprint-runtime.service @@ -1,5 +1,9 @@ +# ⛔ TEST/DEV ONLY — this runs the instance binary directly (TEST_MODE + hardcoded +# SERVICE_ID), bypassing the on-chain lifecycle. PRODUCTION runs the Blueprint +# Manager (cargo-tangle blueprint run), which spawns the instance on an approved +# service request — see ../gen-resell-operator.sh and ../../CLAUDE.md. [Unit] -Description=InferenceBazaar blueprint runtime (Base Sepolia blueprint 17 service 3) +Description=InferenceBazaar blueprint runtime — TEST_MODE direct-run (Base Sepolia blueprint 17 service 3) After=network.target inference-bazaar-mm-sidecar.service inference-bazaar-arti.service Wants=inference-bazaar-mm-sidecar.service inference-bazaar-arti.service diff --git a/deploy/hetzner/inference-bazaar3-blueprint-runtime.service b/deploy/hetzner/inference-bazaar3-blueprint-runtime.service index 09adef4..0043163 100644 --- a/deploy/hetzner/inference-bazaar3-blueprint-runtime.service +++ b/deploy/hetzner/inference-bazaar3-blueprint-runtime.service @@ -1,5 +1,8 @@ +# ⛔ TEST/DEV ONLY — runs the instance binary directly, bypassing the on-chain +# lifecycle. PRODUCTION runs the Blueprint Manager (cargo-tangle blueprint run), +# which spawns the instance on an approved service request. See ../gen-resell-operator.sh. [Unit] -Description=InferenceBazaar venue 2 (Base Sepolia blueprint 17 service 3, operator 0x2420) +Description=InferenceBazaar venue 2 — TEST_MODE direct-run (Base Sepolia blueprint 17 service 3, operator 0x2420) After=network.target inference-bazaar-mm-sidecar.service Wants=inference-bazaar-mm-sidecar.service diff --git a/docs/OPERATOR_ONBOARDING.md b/docs/OPERATOR_ONBOARDING.md index ed1b20c..4d2d211 100644 --- a/docs/OPERATOR_ONBOARDING.md +++ b/docs/OPERATOR_ONBOARDING.md @@ -169,20 +169,39 @@ A ready-to-edit template lives at [`.env.operator.example`](../.env.operator.exa ## 7. Run -**Standalone bonded issuer (HTTP venue + on-chain settlement):** +### Local testing only — run the binary directly + +For kicking the tires on your own box, run the lite venue directly. It goes +on-chain the moment `CHAIN_ID` + `SETTLEMENT_ADDR` are set: ```bash cargo run -p inference-bazaar-operator --bin inference-bazaar-operator-lite -# (same lite binary; it goes on-chain the moment CHAIN_ID + SETTLEMENT_ADDR are set) ``` -**Full blueprint runner (registered on Tangle, multi-operator CLOB):** +> ⛔ **Do NOT do this in production.** Running `inference-bazaar-operator run` +> directly (with a hardcoded `SERVICE_ID` / `TEST_MODE=true`) bypasses the entire +> on-chain lifecycle. It's a dev shortcut only. + +### Production — run the Blueprint Manager (it spawns the instance) + +Each production operator box runs the **Blueprint Manager daemon**. The manager +watches the chain and **spawns the `inference-bazaar-operator run` instance itself** +when a user's service request is approved — you never `ExecStart` the instance binary: ```bash -cargo run -p inference-bazaar-operator --bin inference-bazaar-operator --features blueprint -- run +cargo-tangle blueprint run --protocol tangle \ + --http-rpc-url --ws-rpc-url \ + --keystore-uri --data-dir /bpm-data \ + --chain testnet --settings-file /settings.env +# systemd SyslogIdentifier=blueprint-manager ``` -Kick a quote cycle (or let the tick-keeper do it every ~5 min): +Your operator config (the env in §6 / `.env.operator`) goes in `settings.env`, +which the manager passes to the instance it spawns. The full register → request → +approve → spawn flow is §8. `deploy/gen-resell-operator.sh` generates this unit + +settings file for a resell operator. Reference: `ai-trading-blueprint/deploy/go-live.sh`. + +Once an instance is live, kick a quote cycle (or let the tick-keeper do it every ~5 min): ```bash curl -X POST http://127.0.0.1:9100/mm-tick \ @@ -206,15 +225,18 @@ local command — it's the Tangle service lifecycle: deploy/base-sepolia.sh # cargo tangle blueprint deploy tangle … # Then, as the operator: -cargo tangle blueprint register --blueprint-id … +cargo tangle blueprint register --blueprint-id --keystore-uri +# A USER requests a service selecting registered operators: cargo tangle blueprint request-service --blueprint-id … -# An approver (governance/Safe) calls approve on-chain; once approved you appear -# in the service's operator set and the runner above serves under SERVICE_ID. +# An approver (governance/Safe) approves on-chain; once approved, the Blueprint +# Manager you started in §7 SPAWNS your instance with the assigned SERVICE_ID. +# You never run the instance binary yourself — the manager does. ``` -The production runtime unit (`deploy/hetzner/inference-bazaar-blueprint-runtime.service`) -shows the full env a registered operator runs with (`BLUEPRINT_ID`, -`SERVICE_ID`, `TANGLE_CONTRACT`, `STAKING_CONTRACT`, keystore, data dir). +The production unit is the **Blueprint Manager** (§7; `deploy/gen-resell-operator.sh` +generates one). The older `deploy/hetzner/inference-bazaar-blueprint-runtime.service` +runs the instance binary directly with `TEST_MODE=true` + a hardcoded `SERVICE_ID` +— that is a **test/dev** config, not the production pattern. ---