Repository navigation
Document the valkey-large-object module - #552
kevinmcgehee wants to merge 1 commit into
Conversation
Add command reference pages for the six BLOB.* commands of the valkey-large-object module, with replies, in a new "Large object" command group. Add a "Valkey Large Object" topic section: overview, getting started, use cases, RDMA transfers, and configuration and monitoring, with its menu entries in modules-nav/. Register the module in modules.json and groups.json, link the topic from data-types.md, quickstart.md and RDMA.md, and build its man pages when VALKEY_LARGE_OBJECT_ROOT is set. The command pages need the command JSON files from valkey-io/valkey-large-object#197 and the valkey.io website change that reads them. Signed-off-by: Kevin McGehee <mcgehee@amazon.com>
|
@madolson, could you take an early look at this? It adds the docs for the valkey-large-object module: the six It is a draft because it must not merge before its dependencies:
If this PR merges first, the site deploy shows "ERROR. Command JSON Not loaded." on the new command pages. Review comments are welcome now; I will mark it ready when #197 and #764 are merged. |
madolson
left a comment
There was a problem hiding this comment.
The structure mostly looks fine to me. I think there is probably some pruning that we should do to avoid having the docs repeat themselves.
I also flagged some things that seem weird to me from an API perspective. Feel free to ignore them here and I can re-post them elsewhere. One thing I didn't comment on was that is there a way to automatically create an object with TTL that is planned?
| All configurations can be set as module arguments when the module loads: | ||
|
|
||
| ```bash | ||
| valkey-server --loadmodule ./libvalkey_large_object.so operating-mode Dram segment-size 67108864 max-object-size 33554432 |
There was a problem hiding this comment.
(Not doc related) Is there a reason you're using the legacy configuration system? The modern approach is with https://valkey.io/commands/module-loadex/ which has better integration around module option rewriting and loading. It also solves the weird phrasing you have around the prefix.
| At run time, use `CONFIG GET` for all configurations, and `CONFIG SET` for mutable configurations: | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> CONFIG GET largeobj.max-object-size | ||
| 1) "largeobj.max-object-size" | ||
| 2) "33554432" | ||
| 127.0.0.1:6379> CONFIG SET largeobj.max-object-size 16mb | ||
| OK | ||
| ``` |
There was a problem hiding this comment.
You don't need to document how configs work.
| OK | ||
| ``` | ||
|
|
||
| `CONFIG SET` on an immutable configuration fails. |
There was a problem hiding this comment.
Are any of your configs immutable?
|
|
||
| `CONFIG SET` on an immutable configuration fails. | ||
| In module arguments, give memory sizes in bytes. | ||
| Module arguments with units, such as `64mb`, cause the module load to fail with the log message `invalid digit found in string`. |
There was a problem hiding this comment.
Another reason not to do bespoke configs
| ```plaintext | ||
| BLOB.TCP_SET | ||
| BLOB.TCP_GET | ||
| BLOB.RDMA_HELLO | ||
| BLOB.RDMA_SET | ||
| BLOB.RDMA_GET | ||
| BLOB.INFO | ||
| ``` |
There was a problem hiding this comment.
| ```plaintext | |
| BLOB.TCP_SET | |
| BLOB.TCP_GET | |
| BLOB.RDMA_HELLO | |
| BLOB.RDMA_SET | |
| BLOB.RDMA_GET | |
| BLOB.INFO | |
| ``` |
|
|
||
| These operations work while large object keys exist, but they do not keep the object data: | ||
|
|
||
| * `SAVE`, `BGSAVE`, `SHUTDOWN` with save points, and the automatic RDB snapshots save the key names without the object data. |
There was a problem hiding this comment.
I thought we were going to crash?
|
|
||
| One of the following: | ||
|
|
||
| * Array: when no field is given, a flat array of field names and values: `len` and the object length as an integer, `crc` and the CRC32C checksum as an integer, and `tier` and the storage tier (`dram` or `disk`). |
There was a problem hiding this comment.
A map would be the more reasonable response type here.
|
|
||
| Clients can compare `LEN` and `CRC` with the values that [`BLOB.RDMA_GET`](blob.rdma_get.md) returns, or with a checksum that they calculate after [`BLOB.TCP_GET`](blob.tcp_get.md). | ||
|
|
||
| ## Reply |
There was a problem hiding this comment.
(Not doc related) Although I don't think it's documented anywhere. Is there a reason the return type here changes depending on requests? This makes client logic more
| | Name | Type | Default | Range | Mutable | Description | | ||
| |:-----|:-----|:--------|:------|:--------|:------------| | ||
| | `largeobj.operating-mode` | enum | `Dram` | `Dram`, `Tiered` | No | `Dram`: objects are in DRAM only. `Tiered`: objects are on NVMe and DRAM is a read cache. | | ||
| | `largeobj.disk-dir` | string | `""` | - | No | Directory for object files. Necessary in `Tiered` mode. Must be on XFS, the only supported file system. Must be a dedicated directory, not a mount point, with a writable parent. Must support `O_DIRECT` and user extended attributes. At startup and shutdown, the module deletes this directory and all its contents, then creates it again, empty. | |
There was a problem hiding this comment.
Unrelated, my AI nuked it's own test directory because it set this wrong and it all got deleted.
| description: Configuration parameters, ACL category, and INFO fields of the Valkey Large Object module | ||
| --- | ||
|
|
||
| > **Note:** [valkey-large-object](https://github.com/valkey-io/valkey-large-object) is at version 0.x. |
There was a problem hiding this comment.
This notice should be on all the pages ideally.
Summary
Add documentation for the valkey-large-object module. The module stores large binary values, for example ML KV-cache blocks, tensors and file blobs, in DRAM or in DRAM plus local NVMe disk, with optional RDMA transfers to and from client memory.
BLOB.*commands (BLOB.TCP_SET,BLOB.TCP_GET,BLOB.RDMA_HELLO,BLOB.RDMA_SET,BLOB.RDMA_GET,BLOB.INFO), each with a## Replysection in the style of Move replies from resp{2,3}_replies.json into the markdown files #518, in a new "Large object" command group (groups.json).topics/large-object.md: overview, key concepts, limitations, securitytopics/large-object-getting-started.md: requirements, build, load, first commands, troubleshootingtopics/large-object-use-cases.md: use cases and examplestopics/large-object-rdma.md: RDMA transfers, and how they differ from Valkey RDMA connectionstopics/large-object-configuration.md: configuration, eviction, ACL, INFO fieldsmodules.jsonandmodules-nav/valkey-large-object.ymlentries, links fromdata-types.md,quickstart.mdandRDMA.md,wordlistadditions, and the Makefile, README and man-page workflow changes to build the module's man pages whenVALKEY_LARGE_OBJECT_ROOTis set (the same way as the other modules).modules.jsonhas aversion_noticetext for the module: "valkey-large-object is at version 0.x. Commands, arguments, replies, and configuration can change before version 1.0." The topic pages show it, and the website change shows it at the top of eachBLOB.*command page.Dependencies and merge order
src/commands/blob.*.json). The website and the man-page build read the command title, usage, arguments, ACL categories and "Since module version" from these files.A merge to
mainhere starts a site deploy. If this PR merges before 1 and 2 are live, the new command pages show "ERROR. Command JSON Not loaded."Testing
spellcheckerwith the repository configuration: clean.utils/check-links.pl: no broken links (the existing orphantopics/search-monitoring.mdis not part of this PR).makewas not run locally (no pandoc).