Hide accelerator latency in an unmodified server, with LD_PRELOAD.
I started this project in 2017, during an internship at Microsoft Research. The idea was the one in this repository. There was a draft, but never a working runtime, and it sat unfinished for nine years.
I finished it this year with a large language model doing the building. I specify and review by voice, and Claude Code writes the runtime, runs the experiments, and drafts the paper.
The measurements are below. 17.3× on an unmodified binary, and 2.1× to 3.5× on ten stock servers.
— Bojie Li
A server that offloads work to an accelerator waits tens of microseconds to a few milliseconds for the result. Blocking costs a context switch about as expensive as the offload itself. Busy-waiting burns the core.
The standard fix is to overlap that wait with another request. Normally this means rewriting the server as async code.
The top row is the synchronous case. The bottom row runs requests B and C on the core while A waits.
LD_PRELOAD=./build/lib/libtransparent.so ./your-serverlibtransparent.so turns each connection thread into a fiber, and each
blocking offload into a submit and a yield. One core keeps working while the
device is busy. The application keeps its blocking style and is never
recompiled.
git clone https://github.com/19PINE-AI/transparent-offload
cd transparent-offload
sudo apt install build-essential libssl-dev zlib1g-dev
make && make demo offload | stock rps | toffl rps | speedup | toffl p50 | bad
-----------+------------+------------+---------+----------+------
20us | 123444 | 119607 | 0.97x | 257us | 0
50us | 110651 | 115068 | 1.04x | 280us | 0
100us | 90857 | 114585 | 1.26x | 274us | 0
200us | 70951 | 104461 | 1.47x | 295us | 0
Same binary in both columns. The runtime column stays flat while the device gets ten times slower, which is the latency being hidden. The stock column falls away, because that is a core doing nothing.
The 20 µs row is slower, and it stays in the table on purpose. Below roughly 50 µs the offload is no longer than parking and resuming a fiber, so there is nothing to win. Overlap pays from about 10 µs to a few milliseconds.
make check runs 14 correctness tests in under a minute. No GPU, no root.
The demo above uses an emulated device so that it runs anywhere. The numbers below use real AES on an NVIDIA RTX PRO 6000 Blackwell.
A thread-per-connection server under 64 connections. The binary is not recompiled, relinked, or reconfigured.
| throughput | p50 | wrong answers | |
|---|---|---|---|
| stock binary, synchronous GPU offload | 5,687 rps | 10,534 µs | 0 |
same binary + LD_PRELOAD |
98,624 rps | 548 µs | 0 |
That is 17.3× throughput and 19× lower median latency, with no source change of any kind. A second unrelated binary gives 11.8×. Details in bench/results/transparency.md §3.
Transparency stops at the event loop. nginx and Redis load the library safely but gain nothing, because they never block anywhere the runtime can see.
The alternative is to reroute the offload through machinery those servers already have, such as a thread pool or a module API. That costs 22 to 138 lines and works on every architecture we tried.
Ten off-the-shelf servers, same 1 MiB GPU AES offload in both arms. The only difference between the arms is the edit that lets the offload overlap. Data in bench/results/minimal-edit.md.
Nine of the ten integrations go in through a first-class extension point, so no existing line of the server changes. memcached has no such hook and needs exactly one. The star is the zero-edit runtime on its own binary.
Overlap pays when the offload is heavier than the per-request CPU work it displaces.
Speedup rises from 1.2× at 4 KB to 5.4× at 8 MB on a single-event-loop server. Below the crossover there is nothing to hide.
Throughput is not the only effect. Under open-loop Poisson arrivals, overlap holds the same median latency out to roughly four times the offered load.
The blocking thread pool knees at 103K req/s. The overlapped runtime holds flat to 410K, and its tail stays one to two orders of magnitude lower throughout.
The zero-edit path needs two things. One thread per connection, and every blocking point reachable at the libc layer. We profiled six stock servers and turned that into a five-minute test you can run yourself, described in docs/is-my-server-supported.md.
| Category | Examples | Outcome |
|---|---|---|
| Thread-per-connection, libc blocking | stunnel, conn_server |
overlap |
| Event loop | nginx, redis, memcached | loads safely, no overlap |
| Sub-libc blocking | MariaDB / InnoDB | out of scope |
For the minimal-edit path, the concurrency model predicts both where the integration goes and what it returns.
The edit has the same shape every time. You reroute the offload through a suspend and resume mechanism the server already implements, rather than introducing one.
Ten worked examples live in
examples/integrations/, counted line by line in
bench/results/minimal-edit.md.
Four functions. The runtime never learns what the work is.
long accel_submit(unsigned char *buf, int n); /* start, return a handle, never block */
int accel_done(long id); /* cheap non-blocking poll */
void accel_release(long id); /* free the handle */
void accel_run(unsigned char *buf, int n); /* blocking; what the app calls */Three backends ship. An emulated device with real AES-CTR and a latency knob,
CUDA with cudaEventQuery as the poll, and an RSA-2048 signer over TCP.
Writing another is about fifty lines, described in
docs/writing-a-backend.md.
Fibers interleave where OS threads did not, so this part is measured rather than asserted.
Overlapping unlocked cross-connection read-modify-writes really does corrupt state, and the detector really does stop it. A coarse lock is correct too, at 1.7× to 1.8× lower throughput.
- Per-fiber
errnoand the OpenSSL error queue are saved and restored at every yield. Without this, 100% corruption. With it, none. - Application mutexes held across an offload are respected by a fiber-aware mutex that parks instead of deadlocking the carrier.
- A page-protection conflict detector finds unlocked cross-connection read-modify-writes with no application cooperation. Zero false positives on a safe server, and 125,039 conflicts found on a hostile one.
TOFFLOAD_ENFORCE=1serializes conflicting handlers. Zero lost updates, and no application locks needed.
The suite asserts the hazard is real before asserting that the mitigation works. A change that removes the hazard cannot make these tests pass for the wrong reason.
docs/safety.md covers what the detector costs and where the runtime is unsafe.
| quickstart.md | build, run, try it on your own server |
| is-my-server-supported.md | the five-minute test |
| architecture.md | how the mechanism works |
| writing-a-backend.md | support your accelerator |
| safety.md | correctness model and the unsafe cases |
| configuration.md | every knob |
| troubleshooting.md | when something breaks |
| reproducing.md | the paper's numbers |
include/toffload/ public headers: accel.h, toffload.h, fw_fiber.h
src/ the runtime: interposers, scheduler, detector, config
backends/ emulated, cuda, remote, tls
examples/servers/ servers the tests and demo drive
examples/integrations/ ten real servers with minimal-edit offload
tests/ correctness suite (make check)
bench/ measurement scripts, data, and results
docs/ the documentation above
docs/figs/ the figures on this page
experiments/ the frozen research artifact behind the paper
paper/ LaTeX source and the compiled PDF
Every figure above comes from paper/figs/gen_figs.py. Running it with png
rewrites docs/figs/, and running it with no argument rewrites the paper's
PDFs as well.
INDEX.md maps every claim in the paper to the file that evidences it. docs/paper-path-map.md translates the paths used in the published paper into their locations here.
Reports from running this against a real server, backends for hardware we do not have, and work on the detector's 40% overhead are all welcome. See CONTRIBUTING.md and ROADMAP.md.
This project is the artifact behind "Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines" (Bojie Li, Pine AI), measured on an NVIDIA RTX PRO 6000 Blackwell GPU and a real RSA-2048 TCP signer.
@misc{transparentoffload,
author = {Li, Bojie},
title = {Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines},
year = {2026},
eprint = {2607.02630},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.02630},
note = {Code and data: https://github.com/19PINE-AI/transparent-offload},
}arXiv: https://arxiv.org/abs/2607.02630 · Project page:
https://01.me/research/transparent-offload · Paper PDF:
paper/paper.pdf
MIT, see LICENSE.