Skip to content

Document the valkey-large-object module - #552

Draft
kevinmcgehee wants to merge 1 commit into
valkey-io:mainfrom
kevinmcgehee:largeobject-docs
Draft

kevinmcgehee wants to merge 1 commit into
valkey-io:mainfrom
kevinmcgehee:largeobject-docs

Conversation

@kevinmcgehee

@kevinmcgehee kevinmcgehee commented Nov 14, 2026 •

Copy link
Copy Markdown

Draft: do not merge yet. This PR depends on two other changes. See Dependencies and merge order.

Summary

Add documentation for the valkey-large-object module. The module stores large binary values, for example ML KV-cache blocks, tensors and file blobs, in DRAM or in DRAM plus local NVMe disk, with optional RDMA transfers to and from client memory.

  • Command reference: one page for each of the six BLOB.* commands (BLOB.TCP_SET, BLOB.TCP_GET, BLOB.RDMA_HELLO, BLOB.RDMA_SET, BLOB.RDMA_GET, BLOB.INFO), each with a ## Reply section in the style of Move replies from resp{2,3}_replies.json into the markdown files #518, in a new "Large object" command group (groups.json).
  • Topic section, in the same pattern as the Valkey Search pages:
    • topics/large-object.md: overview, key concepts, limitations, security
    • topics/large-object-getting-started.md: requirements, build, load, first commands, troubleshooting
    • topics/large-object-use-cases.md: use cases and examples
    • topics/large-object-rdma.md: RDMA transfers, and how they differ from Valkey RDMA connections
    • topics/large-object-configuration.md: configuration, eviction, ACL, INFO fields
  • Site integration: modules.json and modules-nav/valkey-large-object.yml entries, links from data-types.md, quickstart.md and RDMA.md, wordlist additions, and the Makefile, README and man-page workflow changes to build the module's man pages when VALKEY_LARGE_OBJECT_ROOT is set (the same way as the other modules).
  • Version note: modules.json has a version_notice text for the module: "valkey-large-object is at version 0.x. Commands, arguments, replies, and configuration can change before version 1.0." The topic pages show it, and the website change shows it at the top of each BLOB.* command page.

Dependencies and merge order

  1. docs: add getting started and command reference valkey-large-object#197 adds the command metadata (src/commands/blob.*.json). The website and the man-page build read the command title, usage, arguments, ACL categories and "Since module version" from these files.
  2. Add valkey-large-object command support valkey-io.github.io#764, the valkey.io website change: adds the module's command JSON directory to the build, the module row and the version note on the command pages, and the "Large object" docs menu section.
  3. This PR.

A merge to main here starts a site deploy. If this PR merges before 1 and 2 are live, the new command pages show "ERROR. Command JSON Not loaded."

Testing

  • spellchecker with the repository configuration: clean.
  • utils/check-links.pl: no broken links (the existing orphan topics/search-monitoring.md is not part of this PR).
  • The pages were built locally with the valkey.io site and the website change, with Zola 0.23.6: the command index group, the six command pages, the version note, the docs menu section and the topic pages render, with no broken internal links or anchors.
  • The example transcripts were run against a live Valkey server with the module loaded.
  • The man-page make was not run locally (no pandoc).

Add command reference pages for the six BLOB.* commands of the
valkey-large-object module, with replies, in a new "Large object"
command group. Add a "Valkey Large Object" topic section: overview,
getting started, use cases, RDMA transfers, and configuration and
monitoring, with its menu entries in modules-nav/.

Register the module in modules.json and groups.json, link the topic
from data-types.md, quickstart.md and RDMA.md, and build its man
pages when VALKEY_LARGE_OBJECT_ROOT is set.

The command pages need the command JSON files from
valkey-io/valkey-large-object#197 and the valkey.io website change
that reads them.

Signed-off-by: Kevin McGehee <mcgehee@amazon.com>
@kevinmcgehee

Copy link
Copy Markdown
Author

@madolson, could you take an early look at this? It adds the docs for the valkey-large-object module: the six BLOB.* command pages, a "Large object" command group, and a topic section in the same pattern as Valkey Search.

It is a draft because it must not merge before its dependencies:

  1. docs: add getting started and command reference valkey-large-object#197 adds the command JSON files (src/commands/blob.*.json).
  2. Add valkey-large-object command support valkey-io.github.io#764 makes the website build read them.
  3. Then this PR.

If this PR merges first, the site deploy shows "ERROR. Command JSON Not loaded." on the new command pages. Review comments are welcome now; I will mark it ready when #197 and #764 are merged.

@madolson madolson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure mostly looks fine to me. I think there is probably some pruning that we should do to avoid having the docs repeat themselves.

I also flagged some things that seem weird to me from an API perspective. Feel free to ignore them here and I can re-post them elsewhere. One thing I didn't comment on was that is there a way to automatically create an object with TTL that is planned?

All configurations can be set as module arguments when the module loads:

```bash
valkey-server --loadmodule ./libvalkey_large_object.so operating-mode Dram segment-size 67108864 max-object-size 33554432

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Not doc related) Is there a reason you're using the legacy configuration system? The modern approach is with https://valkey.io/commands/module-loadex/ which has better integration around module option rewriting and loading. It also solves the weird phrasing you have around the prefix.

Comment on lines +22 to +30
At run time, use `CONFIG GET` for all configurations, and `CONFIG SET` for mutable configurations:

```
127.0.0.1:6379> CONFIG GET largeobj.max-object-size
1) "largeobj.max-object-size"
2) "33554432"
127.0.0.1:6379> CONFIG SET largeobj.max-object-size 16mb
OK
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to document how configs work.

OK
```

`CONFIG SET` on an immutable configuration fails.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are any of your configs immutable?


`CONFIG SET` on an immutable configuration fails.
In module arguments, give memory sizes in bytes.
Module arguments with units, such as `64mb`, cause the module load to fail with the log message `invalid digit found in string`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another reason not to do bespoke configs

Comment thread topics/large-object.md
Comment on lines +67 to +74
```plaintext
BLOB.TCP_SET
BLOB.TCP_GET
BLOB.RDMA_HELLO
BLOB.RDMA_SET
BLOB.RDMA_GET
BLOB.INFO
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```plaintext
BLOB.TCP_SET
BLOB.TCP_GET
BLOB.RDMA_HELLO
BLOB.RDMA_SET
BLOB.RDMA_GET
BLOB.INFO
```

Comment thread topics/large-object.md

These operations work while large object keys exist, but they do not keep the object data:

* `SAVE`, `BGSAVE`, `SHUTDOWN` with save points, and the automatic RDB snapshots save the key names without the object data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we were going to crash?

Comment thread commands/blob.info.md

One of the following:

* Array: when no field is given, a flat array of field names and values: `len` and the object length as an integer, `crc` and the CRC32C checksum as an integer, and `tier` and the storage tier (`dram` or `disk`).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A map would be the more reasonable response type here.

Comment thread commands/blob.info.md

Clients can compare `LEN` and `CRC` with the values that [`BLOB.RDMA_GET`](blob.rdma_get.md) returns, or with a checksum that they calculate after [`BLOB.TCP_GET`](blob.tcp_get.md).

## Reply

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Not doc related) Although I don't think it's documented anywhere. Is there a reason the return type here changes depending on requests? This makes client logic more

| Name | Type | Default | Range | Mutable | Description |
|:-----|:-----|:--------|:------|:--------|:------------|
| `largeobj.operating-mode` | enum | `Dram` | `Dram`, `Tiered` | No | `Dram`: objects are in DRAM only. `Tiered`: objects are on NVMe and DRAM is a read cache. |
| `largeobj.disk-dir` | string | `""` | - | No | Directory for object files. Necessary in `Tiered` mode. Must be on XFS, the only supported file system. Must be a dedicated directory, not a mount point, with a writable parent. Must support `O_DIRECT` and user extended attributes. At startup and shutdown, the module deletes this directory and all its contents, then creates it again, empty. |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated, my AI nuked it's own test directory because it set this wrong and it all got deleted.

description: Configuration parameters, ACL category, and INFO fields of the Valkey Large Object module
---

> **Note:** [valkey-large-object](https://github.com/valkey-io/valkey-large-object) is at version 0.x.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This notice should be on all the pages ideally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants