Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/attachments-labels-verified-media.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"eve": patch
---

Every attachment now reaches the model with a label that names its sandbox path, including images and PDFs that also arrive as bytes, so the agent can pass a visible file to `bash` or open it again. eve checks image and PDF media types against the file's bytes. Only PNG, JPEG, GIF, and WebP images up to 3 MiB and 8000 pixels per side, and PDFs up to 20 MiB, go to the model as bytes, and tool results follow the same rule. A HEIC photo, an oversized screenshot, or a mislabeled file renders as its label instead of failing the model call. Live sessions pay one prompt-cache rewrite after upgrading.
8 changes: 5 additions & 3 deletions docs/sandbox/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,12 @@ Docker and microsandbox mount the compiled trees read-only under `/eve/resources

eve writes byte-backed files that reach the agent into the sandbox under `/workspace/.eve/attachments/` and keeps only a reference in session history. This covers files attached to user messages and files that tools return to the model with `toolOutputPart.file` or as MCP image, audio, and blob content.

Before every model call, eve reads the referenced files back from the sandbox:
Before every model call, eve reads the referenced files back from the sandbox. Every file, whether attached to a message or returned by a tool, reaches the model with a label that names its sandbox path, such as `Attached file /workspace/.eve/attachments/<hash>/report.csv (text/csv)`. The agent can open that path with `read_file` or `bash`. The file's bytes follow its label only when every major provider reads the format natively:

- An image up to 3 MiB or a PDF up to 20 MiB attached to a message reaches the model as a file. Larger files and other types reach the model as a text reference, such as `Attached file /workspace/.eve/attachments/<hash>/report.csv (text/csv)`, that the agent can open with `read_file` or `bash`.
- A file returned by a tool always reaches the model as a file.
- A PNG, JPEG, GIF, or WebP image up to 3 MiB and at most 8000 pixels per side.
- A PDF up to 20 MiB.

Other files, such as CSV, HEIC, audio, and video, reach the model as the label alone. eve checks image and PDF media types against the file's bytes when it stages the file. A file whose bytes don't match its declared image or PDF type is staged as `application/octet-stream`, so a provider never receives bytes it would reject.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't match what the code does. When the bytes are a different format that eve recognizes, the file gets retyped to that format rather than to application/octet-stream. The new integration test stages PNG bytes declared image/jpeg as image/png and inlines them. application/octet-stream only happens when the declared type is PNG/JPEG/GIF/WebP/PDF and the bytes match none of those formats. Also, a declared application/octet-stream file can get upgraded to an image or PDF. Suggest something like: "eve sets image and PDF media types from the file's bytes when it stages the file. A file declared as one of these types whose bytes match none of them is staged as application/octet-stream." The changeset has the same issue: "a mislabeled file renders as its label" is only true when the bytes are unrecognized.


If a referenced file is missing, or its bytes no longer match what eve staged because something in the sandbox overwrote it, the model receives a `FileNotFound` note in its place.

Expand Down
2 changes: 1 addition & 1 deletion e2e/fixtures/agent-compaction-regressions/agent/agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ const taskModel = mockModel({
toolText.includes(CONTENT_OUTPUT_LEAD_MARKER) ? "LEAD_PRESERVED" : "LEAD_LOST",
// The stub names the file's sandbox path, which ends in its filename.
toolText.includes("Attached file ") &&
toolText.includes(`${CONTENT_OUTPUT_FILENAME} (application/octet-stream)`)
toolText.includes(`${CONTENT_OUTPUT_FILENAME} (image/png)`)
? "FILE_STUB_RENDERED"
: "FILE_STUB_LOST",
request.userMessages.some((text) => text.includes("[case: content-output-file-stub]"))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,20 @@ import {
CONTENT_OUTPUT_TAIL_MARKER,
} from "../../constants";

// The canary sits within the first 2000 serialized characters, where the old
// prefix-clipping rendering would expose it to the compaction model. The rest
// of the payload pushes the trailing text far past that clip budget.
const INLINE_FILE_BASE64 = `${"A".repeat(400)}${CONTENT_OUTPUT_PAYLOAD_CANARY}${"A".repeat(11_600)}`;
// A PNG signature and 2000x2000 IHDR chunk (33 bytes, so 44 base64 characters
// with no padding) make eve verify the file as an image it inlines, so the
// payload reaches the model and weighs on history like any screenshot. The
// canary sits within the first 2000 serialized characters, where the old
// prefix-clipping rendering would expose it to the compaction model, and on a
// base64 group boundary so it survives eve's decode and re-encode unchanged.
// The rest of the payload pushes the trailing text far past that clip budget.
const PNG_HEADER = Buffer.from(
"89504e470d0a1a0a" + "0000000d49484452" + "000007d0000007d00806000000" + "00000000",
"hex",
);
const INLINE_FILE_BASE64 =
`${PNG_HEADER.toString("base64")}${"A".repeat(400)}` +
`${CONTENT_OUTPUT_PAYLOAD_CANARY}${"A".repeat(11_599)}`;

export default defineTool({
description:
Expand All @@ -28,7 +38,7 @@ export default defineTool({
toolOutputPart.text(`Alice's completed review note: ${CONTENT_OUTPUT_LEAD_MARKER}`),
toolOutputPart.file(INLINE_FILE_BASE64, {
filename: CONTENT_OUTPUT_FILENAME,
mediaType: "application/octet-stream",
mediaType: "image/png",
}),
// "The attachment's filename" is deliberately not spelled out: the
// checkpoint can only carry it by reading the rendered stub.
Expand Down
143 changes: 124 additions & 19 deletions packages/eve/src/harness/attachment-staging.integration.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { FilePart, ModelMessage, UserContent } from "ai";
import type { FilePart, ModelMessage, ToolResultPart, UserContent } from "ai";
import { describe, expect, it, vi } from "vitest";

import type { ChannelAdapter, FetchFileResult } from "#channel/adapter.js";
Expand All @@ -11,8 +11,10 @@ import {
ATTACHMENTS_ROOT,
hydrateSandboxAttachments,
stageAttachmentsToSandbox,
stageToolResultMedia,
} from "#harness/attachment-staging.js";
import { captureLogRecords } from "#internal/testing/log-records.js";
import { pngBytes } from "#internal/testing/media-fixtures.js";
import { requestPublicUrl } from "#execution/web-fetch/request.js";

vi.mock("#execution/web-fetch/request.js", () => ({ requestPublicUrl: vi.fn() }));
Expand Down Expand Up @@ -248,7 +250,7 @@ describe("stageAttachmentsToSandbox (integration)", () => {
});

it("refines FilePart.mediaType when fetchFile returns a FetchFileResult", async () => {
const resolvedBytes = Buffer.from("PNGDATA", "utf8");
const resolvedBytes = pngBytes(1, 1);
const adapter: ChannelAdapter<any> = {
async fetchFile() {
const result: FetchFileResult = {
Expand Down Expand Up @@ -310,7 +312,7 @@ describe("stageAttachmentsToSandbox (integration)", () => {

it("downloads a link no channel resolver claims, so the provider never fetches it", async () => {
vi.mocked(requestPublicUrl).mockResolvedValueOnce(
new Response("PNG", { headers: { "content-type": "image/png" } }),
new Response(pngBytes(1, 1), { headers: { "content-type": "image/png" } }),
);
const adapter: ChannelAdapter<any> = { kind: "custom-channel", state: {} };
const sandbox = mockSandbox({ id: "sbx_public_link" });
Expand Down Expand Up @@ -597,13 +599,10 @@ describe("stageAttachmentsToSandbox (integration)", () => {
});

describe("hydrateSandboxAttachments (integration)", () => {
// 1 KiB of PNG-like bytes. Small enough to fall under the image
// inline-cap, arbitrary enough that the byte-equality assertion
// catches any corruption on the sandbox round trip.
const smallImageBytes = Buffer.alloc(1024, 0x89);
// 1 KiB of PDF-like bytes. Small enough to fall under the PDF
// inline-cap.
const smallPdfBytes = Buffer.alloc(1024, 0x25);
// A small PNG and PDF under the inline caps; the byte-equality assertions
// catch any corruption on the sandbox round trip.
const smallImageBytes = pngBytes(32, 32, 1024);
const smallPdfBytes = pdfBytes(1024);

it("hydrates small images inline as bytes — provider consumes them multimodally", async () => {
const sandbox = mockSandbox({ id: "sbx_hydrate_image" });
Expand Down Expand Up @@ -641,6 +640,12 @@ describe("hydrateSandboxAttachments (integration)", () => {
expect(hydratedFilePart.filename).toMatch(
/^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\/logo\.png$/,
);
// A label naming the sandbox path precedes the bytes.
expect(hydratedContent).toEqual([
{ type: "text", text: "describe the image" },
{ text: `Attached file ${hydratedFilePart.filename} (image/png)`, type: "text" },
hydratedFilePart,
]);
});

it("hydrates small PDFs inline as bytes — provider handles them natively", async () => {
Expand Down Expand Up @@ -709,7 +714,7 @@ describe("hydrateSandboxAttachments (integration)", () => {
const runtime = await createTestRuntime();
// One byte over the 3 MiB cap so the ref size fails the inline
// check without allocating two massive buffers in the test.
const oversizedImage = Buffer.alloc(3 * 1024 * 1024 + 1, 0x89);
const oversizedImage = pngBytes(32, 32, 3 * 1024 * 1024 + 1);

const stagedContent = (await runtime.runAsSession({ sandbox }, async () =>
stageAttachmentsToSandbox([
Expand All @@ -735,7 +740,7 @@ describe("hydrateSandboxAttachments (integration)", () => {
it("treats oversized PDFs (>20 MiB) as non-inlinable — renders a text reference instead of bytes", async () => {
const sandbox = mockSandbox({ id: "sbx_hydrate_big_pdf" });
const runtime = await createTestRuntime();
const oversizedPdf = Buffer.alloc(20 * 1024 * 1024 + 1, 0x25);
const oversizedPdf = pdfBytes(20 * 1024 * 1024 + 1);

const stagedContent = (await runtime.runAsSession({ sandbox }, async () =>
stageAttachmentsToSandbox([
Expand Down Expand Up @@ -794,6 +799,104 @@ describe("hydrateSandboxAttachments (integration)", () => {
});
});

it("inlines only images whose bytes prove a format every provider reads", async () => {
const sandbox = mockSandbox({ id: "sbx_verified_media" });
const runtime = await createTestRuntime();

const staged = (await runtime.runAsSession({ sandbox }, async () =>
stageAttachmentsToSandbox([
{
data: Buffer.from("not a png"),
filename: "fake.png",
mediaType: "image/png",
type: "file",
},
{
data: Buffer.from("ftypheic"),
filename: "photo.heic",
mediaType: "image/heic",
type: "file",
},
{ data: pngBytes(8001, 10), filename: "wide.png", mediaType: "image/png", type: "file" },
{ data: pngBytes(4, 4), filename: "real.jpg", mediaType: "image/jpeg", type: "file" },
] as UserContent),
)) as FilePart[];
const hydrated = await runtime.runAsSession({ sandbox }, async () =>
hydrateSandboxAttachments([{ content: staged, role: "user" }]),
);

expect(staged.map((part) => part.mediaType)).toEqual([
"application/octet-stream",
"image/heic",
"image/png",
"image/png",
]);
expect(hydrated[0]?.content).toEqual([
{ text: `Attached file ${staged[0]!.filename} (application/octet-stream)`, type: "text" },
{ text: `Attached file ${staged[1]!.filename} (image/heic)`, type: "text" },
{ text: `Attached file ${staged[2]!.filename} (image/png)`, type: "text" },
{ text: `Attached file ${staged[3]!.filename} (image/png)`, type: "text" },
expect.objectContaining({ mediaType: "image/png", type: "file" }),
]);
});

it("applies the same rule to files a tool returns", async () => {
const sandbox = mockSandbox({ id: "sbx_tool_gate" });
const runtime = await createTestRuntime();
const file = (data: Buffer, filename: string, mediaType = "image/png") => ({
data: { data: data.toString("base64"), type: "data" as const },
filename,
mediaType,
type: "file" as const,
});
const messages: ModelMessage[] = [
{
content: [
{
output: {
type: "content",
value: [
file(pngBytes(9000, 9000), "huge.png"),
file(pngBytes(8, 8), "shot.png"),
file(pngBytes(8, 8), "mislabeled.jpg", "image/jpeg"),
],
},
toolCallId: "shot-1",
toolName: "screenshot",
type: "tool-result",
},
],
role: "tool",
},
];

const hydrated = await runtime.runAsSession({ sandbox }, async () =>
hydrateSandboxAttachments(await stageToolResultMedia(messages)),
);

const [result] = hydrated[0]!.content as ToolResultPart[];
expect(result!.output).toEqual({
type: "content",
value: [
{
text: expect.stringMatching(/^Attached file .*\/huge\.png \(image\/png\)$/),
type: "text",
},
{
text: expect.stringMatching(/^Attached file .*\/shot\.png \(image\/png\)$/),
type: "text",
},
expect.objectContaining({ filename: "shot.png", type: "file" }),
{
text: expect.stringMatching(/^Attached file .*\/mislabeled\.jpg \(image\/png\)$/),
type: "text",
},
// The bytes go to the provider under the type they prove.
expect.objectContaining({ filename: "mislabeled.jpg", mediaType: "image/png" }),
],
});
});

it("renders a link or eve-url marker left in history by an older release as a note", async () => {
const runtime = await createTestRuntime();
const messages: ModelMessage[] = [
Expand Down Expand Up @@ -959,18 +1062,16 @@ describe("hydrateSandboxAttachments (integration)", () => {

const stagedContent = (await runtime.runAsSession({ sandbox }, async () =>
stageAttachmentsToSandbox([
{
data: Buffer.from("original"),
filename: "logo.png",
mediaType: "image/png",
type: "file",
},
{ data: smallImageBytes, filename: "logo.png", mediaType: "image/png", type: "file" },
] as UserContent),
)) as UserContent;
const stagedPath = (stagedContent[0] as FilePart).filename as string;

// Same size, different bytes: only the content address can tell them apart.
await sandbox.session.writeBinaryFile({ content: Buffer.from("replaced"), path: stagedPath });
await sandbox.session.writeBinaryFile({
content: Buffer.alloc(smallImageBytes.byteLength, 0x41),
path: stagedPath,
});

const messages = [{ content: stagedContent, role: "user" as const }];
const hydrated = await runtime.runAsSession({ sandbox }, async () =>
Expand Down Expand Up @@ -1012,3 +1113,7 @@ describe("hydrateSandboxAttachments (integration)", () => {
}
});
});

function pdfBytes(paddingBytes: number): Buffer {
return Buffer.concat([Buffer.from("%PDF-1.4\n"), Buffer.alloc(paddingBytes, 0x25)]);
}
Loading
Loading