Repository navigation
Document the valkey-large-object module #552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| Returns metadata about the large object stored at `key`. | ||
| The command does not transfer the payload. | ||
|
|
||
| If you do not give a field, the command returns all the fields as a flat array of field names and values. | ||
| If you give one field, the command returns only the value of that field. | ||
| The field name is not case-sensitive. | ||
|
|
||
| The fields are: | ||
|
|
||
| * `LEN` - The length of the object, in bytes, as the client stored it. | ||
| It does not include DRAM allocation overhead (`Dram` mode) or file system overhead (`Tiered` mode). | ||
| * `CRC` - The CRC32C (Castagnoli) checksum of the payload, as an unsigned 32-bit integer. | ||
| The server calculates this value when it stores the object. | ||
| * `TIER` - The storage tier of the object: `dram` or `disk`. | ||
|
|
||
| In the `Dram` operating mode, `TIER` is always `dram`. | ||
| In the `Tiered` operating mode, `TIER` is `dram` only when a cached copy of the object is ready in DRAM. | ||
| Otherwise, `TIER` is `disk`. | ||
| For more information about the operating modes, see [Valkey Large Object - Overview](../topics/large-object.md). | ||
|
|
||
| If `key` does not exist, the command returns nil. | ||
| If `key` holds a value of a different type, the command returns a `WRONGTYPE` error. | ||
|
|
||
| Clients can compare `LEN` and `CRC` with the values that [`BLOB.RDMA_GET`](blob.rdma_get.md) returns, or with a checksum that they calculate after [`BLOB.TCP_GET`](blob.tcp_get.md). | ||
|
|
||
| ## Reply | ||
|
|
||
| One of the following: | ||
|
|
||
| * Array: when no field is given, a flat array of field names and values: `len` and the object length as an integer, `crc` and the CRC32C checksum as an integer, and `tier` and the storage tier (`dram` or `disk`). | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A map would be the more reasonable response type here.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed. Today the module returns a flat array, also in RESP3 ( |
||
| * Integer: the object length in bytes when `LEN` is given, or the CRC32C checksum when `CRC` is given. | ||
| * `dram` or `disk`: the storage tier when `TIER` is given. | ||
| * Null: if the key does not exist. | ||
|
|
||
| An error occurs if the field is not valid, if `largeobj.enabled` is `no`, or if the key holds a value of a different type. | ||
|
|
||
| ## Examples | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.TCP_SET mykey "hello world" | ||
| OK | ||
| 127.0.0.1:6379> BLOB.INFO mykey | ||
| 1) len | ||
| 2) (integer) 11 | ||
| 3) crc | ||
| 4) (integer) 3381945770 | ||
| 5) tier | ||
| 6) dram | ||
| 127.0.0.1:6379> BLOB.INFO mykey LEN | ||
| (integer) 11 | ||
| 127.0.0.1:6379> BLOB.INFO mykey crc | ||
| (integer) 3381945770 | ||
| 127.0.0.1:6379> BLOB.INFO mykey TIER | ||
| dram | ||
| 127.0.0.1:6379> BLOB.INFO mykey SIZE | ||
| (error) ERR invalid information field | ||
| 127.0.0.1:6379> BLOB.INFO nosuchkey | ||
| (nil) | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| Writes the payload of the large object stored at `key` directly into client memory with RDMA. | ||
| The payload does not go through the Valkey connection. | ||
|
|
||
| The connection must have an RDMA session. | ||
| To start a session, use [`BLOB.RDMA_HELLO`](blob.rdma_hello.md). | ||
|
|
||
| Each `region` is a triple that identifies a client memory region that the client registered with its libfabric endpoint: | ||
|
|
||
| * `rkey` - The remote key of the registered memory region. | ||
| * `addr` - The start address in the region. | ||
| This is a virtual address, or an offset into the registered memory when the fabric provider does not use virtual addresses. | ||
| `0` is a valid value. | ||
| * `len` - The number of bytes that the server can write into the region. | ||
| The value must be more than `0`. | ||
|
|
||
| All values are unsigned decimal integers. | ||
| You can give up to 256 regions. | ||
| The server writes the object bytes into the regions in the order that you give them, and fills each region before it continues to the next one. | ||
|
|
||
| The total length of all the regions must be equal to or more than the object length. | ||
| The regions can be larger than the object. | ||
| The reply gives the object length, so that the client knows where the valid data ends. | ||
|
|
||
| The server replies only after all the RDMA writes are complete. | ||
| The reply is a two-element array: the object length in bytes, and the CRC32C (Castagnoli) checksum that the server stored for the object. | ||
| The client can calculate the checksum of the received bytes and compare it with this value. | ||
|
|
||
| If `key` does not exist, the command returns nil. | ||
| The server looks up the key before it parses the regions and before it checks the session. | ||
| Thus, a missing key returns nil even when the connection has no RDMA session. | ||
| In `Tiered` mode, if too many disk operations run at the same time for the disk staging buffers, the command returns `ERR disk staging buffer pool exhausted; retry later`. | ||
| The error is temporary: a retry later can succeed, after operations in progress finish. | ||
| If `key` holds a value of a different type, the command returns a `WRONGTYPE` error. | ||
|
|
||
| This is a blocking command, so it cannot run in transactions, scripts, functions, or module calls (see [Limitations](../topics/large-object.md#limitations)). | ||
|
|
||
| To read the payload over TCP instead, use [`BLOB.TCP_GET`](blob.tcp_get.md). | ||
| To get only the length and checksum, use [`BLOB.INFO`](blob.info.md). | ||
| If the server cannot post the transfer for 1 second (for example, because the client endpoint is gone), the command fails with `ERR RDMA write failed`. | ||
| There is no other server-side timeout for the transfer. | ||
| To stop a transfer, close the connection. | ||
| For more information, see [Valkey Large Object - RDMA Transfers](../topics/large-object-rdma.md). | ||
|
|
||
| ## Reply | ||
|
|
||
| One of the following: | ||
|
|
||
| * Array: a two-element array, after all the RDMA writes are complete: the object length in bytes as an integer, and the CRC32C checksum of the object as an integer. | ||
| * Null: if the key does not exist. | ||
|
|
||
| An error occurs if the memory regions are not valid or are too small for the object, if the connection has no RDMA session, if the RDMA write or a disk read fails, or if the key holds a value of a different type. See [Errors](../topics/large-object-rdma.md#errors) for all errors. | ||
|
|
||
| ## Examples | ||
|
|
||
| You cannot do the transfer from `valkey-cli`. | ||
| The command needs a client that has a libfabric endpoint and registered memory, and that has an RDMA session on the connection. | ||
| The example shows the command shape and the reply. | ||
| In this example, the key holds a 1MB (1048576 bytes) object of zero bytes. | ||
| The first command writes the object into one region. | ||
| The second command writes the object into two regions of 512KB each. | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.RDMA_GET mykey 4242 140237318455296 1048576 | ||
| 1) (integer) 1048576 | ||
| 2) (integer) 338267154 | ||
| 127.0.0.1:6379> BLOB.RDMA_GET mykey 4242 140237318455296 524288 4243 140237320552448 524288 | ||
| 1) (integer) 1048576 | ||
| 2) (integer) 338267154 | ||
| 127.0.0.1:6379> BLOB.RDMA_GET mykey 4242 140237318455296 4096 | ||
| (error) ERR client memory address length 4096 smaller than object length 1048576 | ||
| 127.0.0.1:6379> BLOB.RDMA_GET nosuchkey 4242 140237318455296 1048576 | ||
| (nil) | ||
| ``` | ||
|
|
||
| On a connection without an RDMA session, an existing key with regions that are large enough returns the session error. | ||
| The server checks the regions before the session: | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.RDMA_GET mykey 999 0 1048576 | ||
| (error) ERR no RDMA session (call BLOB.RDMA_HELLO first) | ||
| ``` |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| Establishes an RDMA session for the current connection. | ||
| A client must send this command once on a connection before it can use [`BLOB.RDMA_SET`](blob.rdma_set.md) or [`BLOB.RDMA_GET`](blob.rdma_get.md) on that connection. | ||
|
|
||
| `client-rdma-address` is the fabric address of the client endpoint, encoded as hexadecimal text. | ||
| The client gets this address from its libfabric endpoint. | ||
| When `largeobj.fabric-provider` is `EfaDirect`, the address must be exactly 32 bytes (64 hexadecimal characters). | ||
| When `largeobj.fabric-provider` is `Emulated`, the address must be a socket address: 16 bytes for IPv4 (`sockaddr_in`) or 28 bytes for IPv6 (`sockaddr_in6`). | ||
| The command takes exactly one argument. | ||
| With no argument or more than one argument, it returns an error. | ||
|
|
||
| The server adds the client address to each of its fabric services. | ||
| The reply is an array with one hexadecimal fabric address for each server fabric service. | ||
| The client must add all the returned addresses to its own address vector before the first transfer. | ||
|
|
||
| The session belongs to the connection. | ||
| The server removes the session when the connection closes. | ||
| A connection can have only one session: a second `BLOB.RDMA_HELLO` on the same connection returns an error. | ||
| To use a new client endpoint, open a new connection. | ||
|
|
||
| If the RDMA fabric did not start on the server, the command returns an error. | ||
| In that case, only the TCP commands, [`BLOB.TCP_SET`](blob.tcp_set.md) and [`BLOB.TCP_GET`](blob.tcp_get.md), are available. | ||
|
|
||
| For the full RDMA flow and the fabric configuration, see [Valkey Large Object - RDMA Transfers](../topics/large-object-rdma.md). | ||
|
|
||
| ## Reply | ||
|
|
||
| Array of strings: one string for each server fabric service, with the hexadecimal fabric address of that service. | ||
|
|
||
| An error occurs if RDMA is not available on the server, if the client address is not valid for the fabric provider, or if the connection already has an RDMA session. See [Errors](../topics/large-object-rdma.md#errors) for all errors. | ||
|
|
||
| ## Examples | ||
|
|
||
| A useful `BLOB.RDMA_HELLO` needs a client that has a libfabric endpoint. | ||
| `valkey-cli` can send the command, but it cannot do the RDMA transfers that follow. | ||
| The example shows the command shape and the reply, with example addresses. | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 0a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff | ||
| 1) "fe80000000000000049a1ffffe2b3c4d0b000000c5a3d1e20000000000000000" | ||
| 2) "fe80000000000000049a1ffffe2b3c4e0c000000d7b4e2f30000000000000000" | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 0a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff | ||
| (error) ERR RDMA session already established | ||
| ``` | ||
|
|
||
| These errors do not need an RDMA client. | ||
| The server in this example uses the `EfaDirect` fabric provider: | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO zz | ||
| (error) ERR invalid rdma address hex | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 0a1b | ||
| (error) ERR rdma address must be 32 bytes | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 0a1b extra | ||
| (error) ERR wrong number of arguments for 'BLOB.RDMA_HELLO' command | ||
| ``` | ||
|
|
||
| With the `Emulated` fabric provider, `valkey-cli` can open a session. | ||
| The reply has one address for each server fabric service, here one: | ||
|
|
||
| ``` | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 0a1b | ||
| (error) ERR rdma address is not a valid sockaddr | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 02001f907f0000010000000000000000 | ||
| 1) "020098ff7f0000010000000000000000" | ||
| 127.0.0.1:6379> BLOB.RDMA_HELLO 02001f907f0000010000000000000000 | ||
| (error) ERR RDMA session already established | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Not doc related) Although I don't think it's documented anywhere. Is there a reason the return type here changes depending on requests? This makes client logic more
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, it makes client code harder. There is no strong reason for it. valkey-io/valkey-large-object#214 covers it: either remove the field argument or always return a map with the requested fields. The docs describe the current behavior.