Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 0 additions & 5 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,5 @@
# Docs TODO

## At next release cut (v0.18 promote)

- [ ] Add `versions/latest/service/rest-api/client/list-embedding-models` to the REST API "Client" nav group in docs.json (page is staged in `next/` only; `scripts/docs.py` will also flag it during promote)
- [ ] C++ docs are removed in `next/`. Before running `promote`, delete `versions/latest/embedded/cpp/`, the C++ nav group for the latest version in docs.json, and point the `/cpp-embedded` redirect to `/versions/latest/embedded/python/introduction`. Until then, `promote` refuses and `sync-next` re-creates the pages in `next/`.

## Should have

- [ ] Langchain for Python SDK
Expand Down
2 changes: 1 addition & 1 deletion changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ CyborgDB follows a bi-monthly release cadence, with new features, enhancements,
- Install via `pip install cyborgdb-core[langchain]`
- Use via `from cyborgdb_core.integrations.langchain import CyborgVectorStore`
- Supports both `cyborgdb-core` and `cyborgdb-lite`
- Added configurable logging utility for embedded library in [C++](/versions/latest/embedded/cpp/client/logger) and [Python](/versions/latest/embedded/python/client/logger)
- Added configurable logging utility for embedded library in C++ and [Python](/versions/latest/embedded/python/client/logger)

</Update>

Expand Down
1,049 changes: 527 additions & 522 deletions docs.json

Large diffs are not rendered by default.

57 changes: 4 additions & 53 deletions versions/latest/embedded/guides/advanced/access-control.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ CyborgDB supports per-user access control (RBAC) on an encrypted index. Instead

## Minting User Keys

The root holder calls `create_user_keys` (Python) / `CreateUserKeys` (C++), supplying the new user's 16-byte identifier, their 32-byte per-user KEK, the permissions to grant, and the **root** index KEK as the admin gate.
The root holder calls `create_user_keys`, supplying the new user's 16-byte identifier, their 32-byte per-user KEK, the permissions to grant, and the **root** index KEK as the admin gate.

<CodeGroup>
```python Python icon="python"
import cyborgdb_core as cyborgdb
import secrets

client = cyborgdb.Client(storage_config=cyborgdb.StorageConfig.memory())
client = cyborgdb.Client(storage_config=cyborgdb.StorageConfig.disk("/tmp/cyborgdb"))

# Root index KEK — the admin gate for user management
index_key = secrets.token_bytes(32)
Expand All @@ -35,31 +35,6 @@ writer_kek = secrets.token_bytes(32)
index.create_user_keys(writer_id, writer_kek, permissions=["read", "write"], index_key=index_key)
```

```cpp C++ icon="brackets-curly"
#include "cyborgdb_core/client.hpp"
#include "cyborgdb_core/encrypted_index.hpp"
#include <array>
#include <openssl/rand.h>

cyborg::Client client("", cyborg::StorageConfig::Memory(), 0, cyborg::kNone);

// Root index KEK — the admin gate for user management
std::array<uint8_t, 32> index_key;
RAND_bytes(index_key.data(), index_key.size());
auto index = client.CreateIndex("shared_index", index_key);

// Mint a read-only user
std::array<uint8_t, 16> reader_id; RAND_bytes(reader_id.data(), reader_id.size());
std::array<uint8_t, 32> reader_kek; RAND_bytes(reader_kek.data(), reader_kek.size());
index->CreateUserKeys(reader_id, index_key, reader_kek,
/*grant_read=*/true, /*grant_write=*/false);

// Mint a read/write user
std::array<uint8_t, 16> writer_id; RAND_bytes(writer_id.data(), writer_id.size());
std::array<uint8_t, 32> writer_kek; RAND_bytes(writer_kek.data(), writer_kek.size());
index->CreateUserKeys(writer_id, index_key, writer_kek,
/*grant_read=*/true, /*grant_write=*/true);
```
</CodeGroup>

You must securely deliver each user's `user_id` and `user_kek` to that user out-of-band — together they are that user's credentials for the index.
Expand All @@ -68,7 +43,7 @@ You must securely deliver each user's `user_id` and `user_kek` to that user out-

## Loading the Index as a User

A user loads the index with their own KEK as the `index_key` and passes their `user_id`. From then on, every operation is gated to the permissions they hold — a read-only user's `upsert`/`delete` is rejected, raising `RuntimeError` in Python (`std::runtime_error` in C++).
A user loads the index with their own KEK as the `index_key` and passes their `user_id`. From then on, every operation is gated to the permissions they hold — a read-only user's `upsert`/`delete` is rejected, raising `RuntimeError`.

<CodeGroup>
```python Python icon="python"
Expand All @@ -87,21 +62,9 @@ results = user_index.query(
# user_index.upsert([...], index_key=reader_kek, user_id=reader_id) # raises RuntimeError — no write wrap
```

```cpp C++ icon="brackets-curly"
// The read-only user loads the index with their own key + user_id
auto user_index = client.LoadIndex("shared_index", reader_kek, nullptr, reader_id);

// Per-op, build a KeyContext from the user's KEK + user_id
cyborg::KeyContext user_ctx{reader_kek, reader_id};

cyborg::Array2D<float> q{{0.1f, 0.2f, 0.3f, 0.4f}};
cyborg::QueryResults results = user_index->Query(q, cyborg::QueryParams{5}, user_ctx);

// Writes throw std::runtime_error for a read-only user (no write wrap)
```
</CodeGroup>

<Note>In Python, you can also override the key per operation with `index_key=` and `user_id=` keyword arguments — required in stateless/service deployments that reload the index per request. In C++, every key-bearing method takes a trailing `KeyContext`; construct it as `cyborg::KeyContext{user_kek, user_id}` for an RBAC user, or pass a bare KEK for the root holder.</Note>
<Note>Every data operation requires the `index_key=` keyword argument; an RBAC user passes their KEK as `index_key=` together with `user_id=`.</Note>

---

Expand All @@ -119,15 +82,6 @@ for u in index.list_user_keys(index_key=index_key):
index.delete_user_keys(reader_id, index_key=index_key)
```

```cpp C++ icon="brackets-curly"
// List all users (root-gated)
for (const auto& u : index->ListUserKeys(index_key)) {
std::cout << "read=" << u.has_read << " write=" << u.has_write << "\n";
}

// Revoke a user (idempotent, root-gated)
index->DeleteUserKeys(reader_id, index_key);
```
</CodeGroup>

---
Expand All @@ -140,7 +94,4 @@ For full signatures and exceptions, refer to the API Reference:
<Card title="Python API Reference" href="../../python/encrypted-index/manage-users" icon="python">
API reference for `create_user_keys` / `list_user_keys` / `delete_user_keys` in Python
</Card>
<Card title="C++ API Reference" href="../../cpp/encrypted-index/manage-users" icon="brackets-curly">
API reference for `CreateUserKeys` / `ListUserKeys` / `DeleteUserKeys` in C++
</Card>
</CardGroup>
82 changes: 11 additions & 71 deletions versions/latest/embedded/guides/advanced/configure-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ sidebarTitle: "Configure Index"

<Info>Index configuration is automatically handled by default. This guide allows you to override these defaults to customize index behavior & performance characteristics.</Info>

CyborgDB uses a single index type: **DiskIVF**, a disk-backed inverted-file index. Rather than choosing between several index variants, you tune one index with a small set of knobs.
CyborgDB uses a single index type: a disk-backed inverted-file (IVF) index. Rather than choosing between several index variants, you tune one index with a small set of knobs.

DiskIVF retrieves results in two stages: a fast first stage narrows down candidates using a compact Product-Quantized (PQ) representation, then a rerank stage recomputes exact distances against the stored vectors (in `float32` or `float16`) to deliver high recall. This gives you the speed of a quantized index with the accuracy of an exact rerank, all within a single index that scales to disk.
The index retrieves results in two stages: a fast first stage narrows down candidates using a compact Product-Quantized (PQ) representation, then a rerank stage recomputes exact distances against the stored vectors to deliver high recall. This gives you the speed of a quantized index with the accuracy of an exact rerank, all within a single index that scales to disk.

There are no longer any `IVFFlat` / `IVFPQ` / `IVFSQ` variants to choose between — a single DiskIVF index covers all of these use cases. You configure it at **creation time** (dimension, storage precision, metric), at **training time** (clustering parameters), and at **query time** (`n_probes`, `rerank_mult`).
You configure the index at **creation time** (dimension, storage precision, metric), at **training time** (clustering parameters), and at **query time** (`n_probes`, `rerank_mult`).

---

## Creating a DiskIVF Index
## Creating an Index

The most common path is to let CyborgDB choose sensible defaults. You only need an index name and a 32-byte key; the `dimension` is auto-detected from your first upsert (or derived from `embedding_model` if you provide one).

Expand All @@ -22,37 +22,23 @@ The most common path is to let CyborgDB choose sensible defaults. You only need
import cyborgdb_core as cyborgdb
import secrets

client = cyborgdb.Client(storage_config=cyborgdb.StorageConfig.memory())
client = cyborgdb.Client(storage_config=cyborgdb.StorageConfig.disk("/tmp/cyborgdb"))

index_key = secrets.token_bytes(32) # 32-byte index KEK

# Create a DiskIVF index with defaults (dimension auto-detected on first upsert)
# Create an index with defaults (dimension auto-detected on first upsert)
index = client.create_index("test_index", index_key)
```

```cpp C++ icon="brackets-curly"
#include "cyborgdb_core/client.hpp"
#include "cyborgdb_core/encrypted_index.hpp"
#include <array>
#include <openssl/rand.h>

cyborg::Client client("", cyborg::StorageConfig::Memory(), 0, cyborg::kNone);

std::array<uint8_t, 32> index_key;
RAND_bytes(index_key.data(), index_key.size()); // 32-byte index KEK

// Create a DiskIVF index with defaults (dimension auto-detected on first upsert)
auto index = client.CreateIndex("test_index", index_key);
```
</CodeGroup>

### Creation Parameters

You can override the defaults at creation time:

- `dimension`: vector dimensionality. Optional — auto-detected from the first upsert, or derived from `embedding_model` if provided.
- `storage_precision`: the on-disk dtype used for the rerank vectors. `float32` (default) gives the highest recall; `float16` roughly **halves disk footprint** with a slight precision loss. Acceptable values are `numpy.float32` / `numpy.float16` (or the strings `"float32"` / `"float16"`) in Python, and `StoragePrecision::Float32` / `StoragePrecision::Float16` in C++.
- `embedding_model`: an optional [`sentence-transformers`](https://www.sbert.net/) model name (Python only) that enables automatic embedding generation and fixes the dimension.
- `storage_precision`: the on-disk dtype used for the rerank vectors. `float32` (default) gives the highest recall; `float16` roughly **halves disk footprint** with a slight precision loss. Acceptable values are `numpy.float32` / `numpy.float16` (or the strings `"float32"` / `"float16"`) in Python.
- `embedding_model`: an optional built-in model name that enables automatic text embedding (no `sentence-transformers`/`torch` install required) and overwrites the dimension. Call `cyborgdb.supported_embedding_models()` for accepted names.
- `metric`: the distance metric — `"euclidean"` (default), `"cosine"`, or `"squared_euclidean"`.

<CodeGroup>
Expand All @@ -63,7 +49,7 @@ import secrets

index_key = secrets.token_bytes(32)

# Create a DiskIVF index with explicit configuration
# Create an index with explicit configuration
index = client.create_index(
"test_index",
index_key,
Expand All @@ -73,20 +59,6 @@ index = client.create_index(
)
```

```cpp C++ icon="brackets-curly"
#include "cyborgdb_core/client.hpp"
#include "cyborgdb_core/encrypted_index.hpp"

// Build a DiskIVF config: dimension 768, float16 rerank storage
cyborg::IndexDiskIVF index_config(
/*dimension=*/768,
/*embedding_model=*/"",
cyborg::StoragePrecision::Float16); // halve disk footprint vs. Float32

// Create the index with a cosine metric
auto index = client.CreateIndex(
"test_index", index_key, index_config, cyborg::DistanceMetric::Cosine);
```
</CodeGroup>

<Tip>Use `float16` storage precision when disk footprint matters more than the last fraction of a percent of recall. For most workloads the recall difference is negligible.</Tip>
Expand Down Expand Up @@ -115,17 +87,6 @@ index.train(
)
```

```cpp C++ icon="brackets-curly"
// Train the index with custom clustering parameters
cyborg::TrainingConfig training_config(
/*n_lists=*/4096,
/*batch_size=*/0, // auto
/*max_iters=*/100,
/*tolerance=*/1e-6,
/*max_memory=*/0); // no limit

index->TrainIndex(training_config, index_key);
```
</CodeGroup>

For more on the training lifecycle, see [Training an Encrypted Index](../encrypted-indexes/train-index).
Expand All @@ -134,10 +95,10 @@ For more on the training lifecycle, see [Training an Encrypted Index](../encrypt

## Query-Time Parameters

DiskIVF exposes two knobs at query time that trade recall against latency:
The index exposes two knobs at query time that trade recall against latency:

- `n_probes`: how many clusters to search per query. Higher values increase recall at some latency cost. `0` (default) auto-selects based on `n_lists`.
- `rerank_mult`: the stage-1 retrieval multiplier. CyborgDB first retrieves `rerank_mult * top_k` candidates using the compact PQ representation, then reranks them against the stored `float32`/`float16` vectors. Higher values improve recall at some latency cost (default `10`).
- `rerank_mult`: the stage-1 retrieval multiplier. CyborgDB first retrieves `rerank_mult * top_k` candidates using the compact PQ representation, then reranks them against the stored vectors. Higher values improve recall at some latency cost (default `50`).

<CodeGroup>
```python Python icon="python"
Expand All @@ -151,19 +112,6 @@ results = index.query(
)
```

```cpp C++ icon="brackets-curly"
// Tune recall vs. latency at query time
cyborg::Array2D<float> query_vectors{{0.5, 0.9, 0.2, 0.7}};
cyborg::QueryParams query_params(
/*top_k=*/10,
/*n_probes=*/32,
/*filters=*/"",
/*include=*/{},
/*greedy=*/false,
/*rerank_mult=*/10);

cyborg::QueryResults results = index->Query(query_vectors, query_params, index_key);
```
</CodeGroup>

---
Expand All @@ -183,11 +131,6 @@ index = client.create_index(
)
```

```cpp C++ icon="brackets-curly"
// Existing setup ...

auto index = client.CreateIndex("index_name", index_key, cyborg::DistanceMetric::Cosine);
```
</CodeGroup>

The currently supported distance metrics are:
Expand All @@ -206,7 +149,4 @@ For more information on configuring an encrypted index, refer to the API Referen
<Card title="Python API Reference" href="../../python/types" icon="python">
API reference for `StoragePrecision` and index types in Python
</Card>
<Card title="C++ API Reference" href="../../cpp/types" icon="brackets-curly">
API reference for `IndexDiskIVF` and `StoragePrecision` in C++
</Card>
</CardGroup>
35 changes: 1 addition & 34 deletions versions/latest/embedded/guides/advanced/kms.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ sidebarTitle: "KMS"
mode: "wide"
---

CyborgDB can persist a per-index **KMS envelope** — a small record describing how an index's 32-byte KEK is wrapped by an external Key Management Service. A service layer can then read the envelope at request time, unwrap the KEK with the external KMS, and hand the plaintext KEK to the SDK. The envelope itself is represented by the `KMSBlob` type and stored as a FlatBuffer next to the index keystore.
CyborgDB can persist a per-index **KMS envelope** — a small record describing how an index's 32-byte KEK is wrapped by an external Key Management Service. A service layer can then read the envelope at request time, unwrap the KEK with the external KMS, and hand the plaintext KEK to the SDK. The envelope itself is represented by the `KMSBlob` type and stored alongside the index.

<Note>KMS key wrapping is **advanced and optional**, and primarily intended for the **service layer**. Embedded SDK users who supply their own KEK directly (see [Managing Encryption Keys](./managing-keys)) can ignore these functions entirely, or use `provider="none"`, in which case nothing key-derived is stored and the caller supplies the plaintext KEK per request.</Note>

Expand Down Expand Up @@ -49,26 +49,6 @@ cyborgdb.create_index_kms(config, "my_index", blob)
cyborgdb.push_index_kms(config, "my_index", blob)
```

```cpp C++ icon="brackets-curly"
#include "cyborgdb_core/index_kms.hpp"

// Where the index keystore lives
cyborg::StorageConfig config = cyborg::StorageConfig::Disk("/tmp/cyborgdb");

cyborg::KMSBlob blob;
blob.kms_name = "prod-kms";
blob.provider = "aws-kms";
blob.key_id = "arn:aws:kms:us-east-1:123456789012:key/abcd-...";
blob.region = "us-east-1";
blob.wrapped_kek = {/* KEK wrapped by the external KMS */};
blob.version = 1;

// Strict insert (throws if an envelope already exists for this index)
cyborg::CreateIndexKMS(config, "my_index", blob);

// Idempotent upsert (creates or overwrites)
cyborg::PushIndexKMS(config, "my_index", blob);
```
</CodeGroup>

---
Expand All @@ -91,16 +71,6 @@ print(blob.provider, blob.key_id, blob.version)
cyborgdb.delete_index_kms(config, "my_index")
```

```cpp C++ icon="brackets-curly"
#include "cyborgdb_core/index_kms.hpp"

// Read the envelope back
cyborg::KMSBlob blob = cyborg::GetIndexKMS(config, "my_index");
// ... unwrap blob.wrapped_kek with your KMS to recover the 32-byte KEK ...

// Delete the envelope (idempotent)
cyborg::DeleteIndexKMS(config, "my_index");
```
</CodeGroup>

<Tip>For embedded deployments where you already hold the KEK, you don't need the KMS envelope at all — pass the KEK directly to `create_index` / `load_index`. See [Managing Encryption Keys](./managing-keys) and [Access Control (RBAC)](./access-control) for per-user keys.</Tip>
Expand All @@ -115,7 +85,4 @@ For full signatures and the `KMSBlob` type, refer to the API Reference:
<Card title="Python API Reference" href="../../python/kms" icon="python">
API reference for the KMS module functions and `KMSBlob` in Python
</Card>
<Card title="C++ API Reference" href="../../cpp/kms" icon="brackets-curly">
API reference for the KMS module functions and `KMSBlob` in C++
</Card>
</CardGroup>
Loading
Loading