feat(e2e): add basic e2e test suite for runtime templates by Hweinstock · Pull Request #2339 · aws/agentcore-cli

Hweinstock · 2026-09-18T19:12:54Z

Problem

We are missing integration tests.

Solution

add tests that trigger on push events only (we can add scoped down version to PRs soon).
tests leverage vitest (not bun build in) because it gives some nice features like tagging. (which we can leverage for canary).
port over stack clean up logic.

Some patterns this PR attempts to establish:

tests should read as a sequence of CLI commands.
configuration should be separated from the the execution, such that making configuration changes is as easy as possible.
parametrize tests when possible to make it easy to re-use logic for different variants.
tests should be concurrent when possible, otherwise explicitly serial.

Verification

Running e2e locally using instructions (note I had to patch my CDK due to breaking changes).
there are two other bugs/gaps for follow-up: dev race condition and local invoke support for other protocols.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Nice PR. The test structure follows the "spawn the real CLI as a subprocess" pattern the guidelines call out, avoids mocking entirely, and reads as a sequence of CLI commands as advertised. A few minor observations, none blocking:

test/setup.ts runs cleanupStaleStacks at module top-level, so it will execute once per vitest worker. That's probably fine (stale-stack deletion is idempotent and stack age already excludes anything <2h old), but worth being aware of if workers ever run in parallel — you could end up issuing duplicate DeleteStack calls for the same stacks. Not a correctness issue.
CliRunner uses shell: true with concatenated arguments and a platform-specific quoteShellArg. Since input comes only from tests and AGENTCORE_CLI_PATH, this is fine, but it's worth remembering that the AGENTCORE_CLI_PATH value itself is not quoted — a workspace path with spaces on Windows would break the Windows job. Given CodeBuild's default checkout path this shouldn't happen today.
The last test is named "deploys the empty project" but actually does both a remove all and a deploy. Cosmetic.
"deploys all runtimes" asserts deployment.message contains "Deployed project", which couples the test to the exact wording of a user-facing string. Consider asserting on a stable structural field if one exists, otherwise fine.

Telemetry isn't applicable here — this PR only adds test infrastructure.

agentcore-devx-automation · 2026-09-18T19:16:43Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-09-18T21:42:02Z

Claude Security Review: no high-confidence findings. (run)

codecov-commenter · 2026-09-18T21:45:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.26%. Comparing base (a23e472) to head (62c709a).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #2339   +/-   ##
=========================================
  Coverage     97.26%   97.26%           
=========================================
  Files           610      610           
  Lines         40590    40590           
=========================================
  Hits          39480    39480           
  Misses         1110     1110

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-09-18T21:55:49Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock · 2026-09-18T21:41:01Z

      CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
+  e2e:
+    if: github.event_name == 'push'


we skip for now for simplicity, but eventually I think we should support a separate tag to run a reduced set.

that is a good idea! when should that suite run and what would it contain?

are you thinking of the idea of a tag that can "mark" certain suites as canary?

are you thinking of the idea of a tag that can "mark" certain suites as canary?

yes exactly!

that is a good idea! when should that suite run and what would it contain?

I have some ideas here.

One is that we should run the basic tests across functionality (effectively smoke test each feature)

another is that we could maybe check the filepaths touched to guess some tags that should run.

another is that a reviewer or harness reviewer could manually/automatically ask for certain tags to be run.

I'll likely start with the first one, and we can explore the others as we get more time. Overall, I want to keep the PR e2e tests short for faster feedback, and reserve all of them for the push event.

2 and 3 would be killer!

Hweinstock · 2026-09-21T13:34:36Z

@@ -0,0 +1,121 @@
+name: e2e-test


separating this out into its own workflow with workflow_call means the canary will be a thin wrapper.

jariy17

LGTM, just small nits.

jariy17 · 2026-09-21T18:05:39Z

@@ -0,0 +1,386 @@
+import { afterAll, beforeAll, describe, expect, test } from "vitest";


should we name the test folder to e2e-tests? It would be clear for agents to only add e2e tests in here.

sure, I can rename to e2eTest to match the camel case convention. I'm used to the test/ convention from other languages, but not sure if this is a pattern in TS.

jariy17 · 2026-09-21T18:09:37Z

+  }
+
+  if (runtime.protocol === "A2A") {
+    const response = parseResult(A2aResponseSchema, result);


Can you verify we got a 2xx response for A2A and http?

I don't think we get a direct status code on non-http protocol runtimes, but I can add a check that there is not an error in the response.

agentcore-devx-automation · 2026-09-21T19:01:16Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-09-21T19:09:52Z

Claude Security Review: no high-confidence findings. (run)

github-actions Bot added the size/l PR size: L label Sep 18, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 18, 2026

agentcore-devx-automation Bot reviewed Sep 18, 2026

View reviewed changes

agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 18, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 18, 2026

Hweinstock added 18 commits September 18, 2026 21:39

test(e2e): add CLI runner, workspace, and cleanup helpers

ea63e93

test(e2e): add runtime, harness, and memory suites

6ea061b

ci: add reusable e2e-test workflow

10d9dac

test(e2e): run project journeys through compiled CLI

c1765ce

test(e2e): accept completed empty runtime streams

8e847eb

test(e2e): reduce suite to runtime lifecycle

acaecfb

feat(e2e): add e2e test framework

2378641

test(e2e): simplify runtime lifecycle suite

466f554

ci(e2e): run suite across supported platforms

8f66692

fix(e2e): include all test files

c6d2d45

test(e2e): invoke deployed runtimes concurrently

c70d70e

refactor(e2e): clean up test names

a3f1eed

refactor(e2e): swap to vitest for native tagging support

cf72595

refactor(logs): add a simple logger

34b6e3d

docs(e2e): update readme

784d489

refactor(e2e): keep dev port allocation out of PR

f0d6e41

ci(e2e): support main pushes and reusable calls

e60b1ba

ci(e2e): run reusable suite from CI pushes

9be3605

Hweinstock force-pushed the fix/e2e-integration-suite branch from baec51c to 9be3605 Compare September 18, 2026 21:39

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 18, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 18, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 18, 2026

fix(test): remove e2e test from unit tests

685bd56

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 18, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 18, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 18, 2026

Hweinstock marked this pull request as ready for review September 18, 2026 22:05

Hweinstock commented Sep 21, 2026

View reviewed changes

jariy17 reviewed Sep 21, 2026

View reviewed changes

Hweinstock added 3 commits September 21, 2026 18:26

refactor(e2e): rename top level dir to e2eTest

427189f

feat(e2e): add check for no erro on non-http protocols

e02b7be

fix(e2e): shrink some names to avoid hitting the cap

8880b26

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 21, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 21, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 21, 2026

fix(test): ignore e2e tests in unit test suite

62c709a

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 21, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 21, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 21, 2026

jariy17 approved these changes Sep 21, 2026

View reviewed changes

jariy17 merged commit e72a6a4 into aws:refactor Sep 21, 2026
16 of 18 checks passed

Hweinstock deleted the fix/e2e-integration-suite branch September 21, 2026 21:50

		@@ -0,0 +1,386 @@
		import { afterAll, beforeAll, describe, expect, test } from "vitest";

Conversation

Hweinstock commented Sep 18, 2026

Problem

Solution

Verification

Uh oh!

agentcore-devx-automation Bot left a comment

Choose a reason for hiding this comment

AgentCore Harness Review

Uh oh!

agentcore-devx-automation Bot commented Sep 18, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 18, 2026

Uh oh!

codecov-commenter commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot commented Sep 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Sep 21, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Sep 18, 2026 •

edited

Loading

Hweinstock Sep 21, 2026 •

edited

Loading