diff --git a/research/attachments.md b/research/attachments.md new file mode 100644 index 0000000000..58914779d2 --- /dev/null +++ b/research/attachments.md @@ -0,0 +1,170 @@ +--- +issue: https://github.com/vercel/eve/issues/3419 +status: proposed +last_updated: "2026-10-10" +--- + +# Attachments without a new primitive + +## Summary + +eve's attachment pipeline mostly works, but its rough edges keep breaking +sessions. One file that eve can't resolve or a provider rejects stays in +history, so every later model call fails the same way (#3419, #855, #497, and +earlier #399, #2089, #276). Channels also give `FilePart.data` different +meanings, the inline decision trusts declared media types, and `read_file` +can't reopen a PDF. + +This plan keeps today's storage: staged files in the session sandbox, refs in +history. It replaces #4223's pluggable attachment store. It fixes the pipeline +around one rule: **every attachment becomes a staged file or a note before it +enters history**. It adds three small authoring surfaces and no new +definition type. + +## Mental model + +An attachment is a file in the session. A channel gives eve bytes or a URL. +Inside the workflow step, eve turns it into a file in the sandbox or replaces +it with a note that says why it couldn't. Before each model call, eve shows the +file inline when the model can read it natively, and always names its path. + +```text +channel staging (in step) model call +bytes | URL ---> fetchFile / eve fetch --> sandbox --> label + inline bytes + | ref or label only + +--> note (text part) +``` + +"Inline" means the bytes travel in one model request, read back from the +sandbox for that call. History and workflow snapshots hold only the ref: path, +size, media type, and image dimensions or PDF page count. + +## Invariants + +- History holds only `eve-sandbox:` refs or text notes for attachments. It + never holds a raw URL, an `eve-url:` marker, or inline bytes. This covers user + messages, cancelled turns, and tool results. The one exception is an AI SDK + provider file reference, which names a file the provider already holds and + passes through unchanged. +- The provider never fetches an attachment. eve fetches it or writes a note, + so an expired or private link can't fail later calls. +- The inline decision stays a pure function of the ref, so the prompt cache + holds. It inlines only formats that eve verified from the bytes. +- Every attachment the model sees has a label with its sandbox path. +- Channels don't filter attachments by media type. `uploadPolicy` decides what + eve accepts, and the inline decision decides inline or label. + +## Authoring surface + +| Surface | Change | +| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `FilePart.data` | Bytes, base64, and `data:` URLs are bytes. Any other string or `URL` with a scheme is a link to resolve. A provider reference passes through. | +| `fetchFile(url, ctx)` | Also on `eveChannel()`. Returning `null` now means "not mine", not "let the provider fetch it". | +| `attachmentError(message)` | New in `eve/channels`. A `fetchFile` throws it to give the model a safe reason. | +| `FetchFileContext` | Exposes `session`, which staging already passes. `FetchFileFunction` and its types export from `eve/channels`. | +| `read_file` | Opens PDFs up to 20 MiB as files the model reads. | + +When `fetchFile` returns `null`, or the channel has none, eve fetches public +`https:` links itself, at most 10 per message and one at a time. The fetch +rejects private and reserved addresses, stops at the 25 MB default upload cap, +and times out. Any other scheme, or a failed +fetch, becomes a note. Built-in channel fetchers keep their own caps. + +```ts title="agent/channels/eve.ts" +import { eveChannel } from "eve/channels/eve"; +import { attachmentError } from "eve/channels"; + +export default eveChannel({ + async fetchFile(url) { + if (!url.startsWith("https://uploads.example.com/")) return null; + const response = await fetch(url, { headers: { authorization: `Bearer ${TOKEN}` } }); + if (!response.ok) throw attachmentError(`The upload returned HTTP ${response.status}.`); + return Buffer.from(await response.arrayBuffer()); + }, +}); +``` + +A web client can then upload a large file to its own storage and send the URL +in a file part. The bytes skip the request body limit and the workflow start +payload, and the step loads them. + +## Model-facing rendering + +Each attachment renders as a label, followed by its bytes when it inlines: + +```text +Attached file /workspace/.eve/attachments//chart.png (image/png) +``` + +| Kind | Inlines when | +| -------------------------- | ---------------------------------------------------------------------- | +| PNG, JPEG, GIF, WebP image | Up to 3 MiB and at most 8000 px per side, by the dimensions on the ref | +| PDF | Up to 20 MiB, and the bytes start with a PDF header | +| Anything else | Never; the label is enough for `read_file` or `bash` | + +Staging corrects the declared media type from the bytes for these formats. A +declared image or PDF whose bytes don't match is staged as +`application/octet-stream`. HEIC, SVG, and TIFF render as labels. Tool-result +files follow the same rule; today they always inline. + +The new labels change how existing messages render, so each live session pays +one prompt-cache rewrite on upgrade. + +## Channel fixes + +- When a mention carries no files, Slack collects the files of every message + since the previous mention of the app, capped at 10 messages and skipping + the app's own messages (#705). That earlier mention's turn collected the + files before it. A message that doesn't mention the app gets no lookback. It skips `mode: "external"` files such + as Google Drive links, whose link stays in the message text (#855). It stops + dropping audio and video, and its file fetch gets a timeout. +- Bytes in a `send()` payload cross the queue as `data:` URLs, so no world + serializer sees a raw `Uint8Array` (#497). `message.received` reports an + inline file's size instead of echoing its `data:` URL, because the session + stream stores that event. +- A cancelled turn stages its attachments inside the framework providers, so a + photo sent just before a steering message survives (#3419). + +## Deferred + +- **A pluggable store** (#4223). No open issue reports lost files since #325 + turned missing bytes into a `FileNotFound` note. The sandbox boot per + attachment costs time, not correctness. +- **Inline audio and video for Gemini** (#543). Gemini reads them natively, + but inline media is re-sent on every model call and counts against a + per-request size limit. A few clips could exceed it and fail every later + call, which is the failure this plan removes. Provider file references, such + as the Gemini Files API, fit this better and need a store. Until then, audio + and video render as labels, and the agent can process them with tools. +- **Per-request image limits.** Anthropic lowers its per-side limit to 2000 px + when a request carries more than 20 images. Gating on that would need the + inline decision to count images across history, which breaks its purity. +- Outbound files, a `toModel` conversion hook, model capability metadata, and + trace content. + +## Rollout + +Each step is one PR in a stack, in this order: + +1. This plan. +2. Never persist an unresolved attachment: cancelled turns, no-sandbox + fallbacks, unknown `data` shapes, bytes across the queue, and removal of the + unused `eve-attachment:` refs. +3. eve fetches unresolved links instead of the provider. +4. Labels on every attachment, media types verified from bytes, and one + inline rule for inbound and tool-result files. +5. `read_file` opens PDFs. +6. `fetchFile` on `eveChannel()` (#4194). +7. `attachmentError`, `FetchFileContext.session`, and the exported `fetchFile` + types (#4304). +8. Slack thread lookback, external files, audio and video, and fetch timeout + (#705, #855). + +## Verification + +- Unit: `data` normalization, the media type check, the inline rule, labels, + `read_file` on a PDF, and Slack lookback. +- Integration: staging across a cancelled turn, a link with no resolver, a + failing fetch, and hydration of old refs. +- Channel conformance: the web chat driver sends files, so the HTTP channel + joins the attachment rows.