Skip to content

chore: add Lambda e2e integration test suite#619

Closed
ava-silver wants to merge 15 commits into
mainfrom
ava.silver/chore/lambda-e2e-suite
Closed

chore: add Lambda e2e integration test suite#619
ava-silver wants to merge 15 commits into
mainfrom
ava.silver/chore/lambda-e2e-suite

Conversation

@ava-silver

@ava-silver ava-silver commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a full-lifecycle end-to-end test suite for the DatadogLambda construct that runs a real cdk deploy against AWS and verifies telemetry actually reaches Datadog. It conforms to the serverless instrumentation e2e contract (serverless-ci/e2e/spec.md) and is distinct from the existing synth-only snapshot integration tests.

The construct is the instrumentation mechanism, so APPLY and REMOVE are both cdk deploy of the same stack (e2e/app/app.ts); only E2E_INSTRUMENT differs:

  1. Provision an uninstrumented Node.js Lambda (E2E_INSTRUMENT=false), uniquely named one-e2e-cdk-lambda-<runid> and freshness-tagged at creation.
  2. APPLY (E2E_INSTRUMENT=true) -- verify config: pinned Datadog Node + Extension layers, handler redirected to the Datadog wrapper, and the required DD_* env vars asserted by identity (DD_SERVICE, DD_ENV, DD_VERSION, DD_SITE, DD_API_KEY, DD_TRACE_ENABLED, DD_LOGS_INJECTION, run id in DD_TAGS).
  3. Trigger (aws lambda invoke) -- poll Datadog spans + logs filtered by run id and assert each carries the expected service/env/version/run id (identity, not mere existence).
  4. Re-APPLY -- cdk diff --fail asserts no diff (idempotent).
  5. REMOVE -- redeploy uninstrumented; assert layers, all DD_* env vars, and the dd_cdk_construct tag are gone. The hygiene freshness tag intentionally survives so the cross-repo sweeper can still reap the function.

Teardown (cdk destroy) always runs, even on failure.

Motivation

The repo had no test that exercised the construct against real AWS and confirmed telemetry lands in Datadog -- only synth-time snapshot checks. This suite closes that gap end-to-end and plugs the construct into the shared serverless e2e contract.

Testing Guidelines

The lambda-e2e job runs the full lifecycle in CI (deploy + config verify + spans/logs telemetry identity + idempotent re-apply + remove/clean verify + teardown). It is path-filtered to construct or suite changes and self-skips (no-op) via SKIP_LAMBDA_TESTS when nothing relevant changed; forks no-op because OIDC only exists on the canonical repo. When it does run, the AWS OIDC / dd-sts steps must succeed -- an auth or telemetry failure fails the job loudly.

Local run (prerequisites and commands in e2e/README.md):

aws-vault exec sso-serverless-sandbox-account-admin -- DD_API_KEY=... DATADOG_API_KEY=... DATADOG_APP_KEY=... yarn test:e2e

Additional Notes

  • Auth, no static Datadog keys. AWS access is via GitHub OIDC into the dedicated gha-datadog-cdk-e2e role in the serverless sandbox account, scoped to deploy one-e2e-cdk-lambda-* through the CDK bootstrap roles. Datadog API + App keys are minted at runtime via DataDog/dd-sts-action under the datadog-cdk-constructs-e2e policy (GitHub OIDC -> short-lived keys) -- no static Datadog keys live in this repo. IAM resources are cataloged in serverless-ci/e2e/iam-infra.md.
  • Config comes from repo variables: AWS_ROLE_ARN_E2E, AWS_REGION_E2E, DD_SITE_E2E, SKIP_LAMBDA_TESTS.
  • The CDK app is bundled from src with esbuild (the repo TS 6 toolchain is incompatible with ts-node), so the suite exercises the construct code in this repo, not a published package.
  • Pinned artifact versions live in e2e/helpers/versions.ts (Node 22 runtime, Node layer + Extension layer); bump deliberately so a failure blames this construct's wiring, not an upstream layer/tracer change.
  • TS helpers are runner-agnostic (node:assert, no jest/vitest imports) per the spec, so other suites can reuse them.
  • Wired through .projenrc.js (not direct edits to generated files): vitest + @datadog/datadog-api-client devDeps and a test:e2e task.
  • Hygiene: every resource is named one-e2e-cdk-lambda-<runid> and tagged one_e2e_created:<unix-ts>; in-test teardown is best-effort and the cross-repo sweeper is the real guarantee.

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@ava-silver ava-silver changed the title add Lambda e2e integration test suite chore: add Lambda e2e integration test suite Jun 16, 2026
@datadog-datadog-prod-us1-2

This comment has been minimized.

@ava-silver
ava-silver force-pushed the ava.silver/chore/lambda-e2e-suite branch from ca902f6 to a186edd Compare June 26, 2026 20:56
@ava-silver

Copy link
Copy Markdown
Contributor Author

Superseded by stacked PRs #625#626#627.

@ava-silver ava-silver closed this Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant