Skip to content

fix(testing): the shipped E2E test cannot pass without a backend, so every fork inherits a red runΒ #33

Description

@koniz-dev

Context

With the native harness in place (#30), Patrol now genuinely runs - and the
shipped test genuinely fails:

πŸ“ Total: 1   βœ… Successful: 0   ❌ Failed: 1

Expected: exactly one matching candidate
[<'e2e_home_content'>]: []
Which: means none were found but one was expected
app_e2e_test.dart:20

Evidence: run 32966672465.

integration_test/app_e2e_test.dart enters credentials, taps
e2e_login_submit, then asserts e2e_home_content. The sample auth flow posts
to BASE_URL. With no server the login call fails, the app stays on the login
screen, and the assertion cannot pass - in CI, and on any fresh clone.

This is a starter. Every fork inherits it. Shipping an E2E suite that is red by
construction trains people to ignore it, which is how the four-month-red main
in #11 happened.

Proposed shape

Split into two tests:

  1. Smoke, always runs, no backend. App boots, routes to login, renders
    e2e_login_submit, and the form accepts input. This genuinely exercises the
    whole native harness, app bootstrap, routing, and localization - it is not a
    token test.
  2. Authenticated flow, opt-in. The existing auth to home assertions, with
    skip: !bool.fromEnvironment('E2E_BACKEND'). Teams with a real API run
    patrol test --dart-define=E2E_BACKEND=true. patrolTest supports skip:
    (patrol 3.20.0 lib/src/common.dart:87).

Also in scope

integration_test/auth_flow_test.dart asserts e2e_home_content the same way
and has the same problem.

tool/golden/{stripped,no_tasks,no_feature_flags}/integration_test/ carry
counterparts of both files. Per CLAUDE.md, changing a file with a
tool/golden/* counterpart means updating that counterpart too, or
strip-smoke.yml breaks.

Acceptance criteria

  1. A workflow_dispatch run of E2E Android on main reports Total: greater
    than 0, Failed: 0, and the job concludes success with no backend
    configured. Record the run id and paste the summary.
  2. The skipped authenticated test is visible as skipped, not silently absent:
    the summary shows a non-zero Skipped: count.
  3. The smoke test asserts something real - at minimum that
    e2e_login_submit renders. A test that only calls app.main() does not
    count.
  4. integration_test/auth_flow_test.dart is consistent with the same approach.
  5. All six tool/golden/*/integration_test/*.dart counterparts are updated, and
    the Strip + analyze + test check passes.
  6. integration_test/README.md documents how to run the authenticated flow
    (--dart-define=E2E_BACKEND=true) and states plainly that it is skipped by
    default and why.
  7. ./scripts/dev/audit_template.sh exits 0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

epic:testingtest/, integration_test/, coverage gates, Patrol E2E, golden acceptancepriority:P1Next up: blocks adopters or a documented workflow is wrongstatus:in-progressClaimed by an assignee and being worked right nowtype:bugSomething that is broken relative to documented behavior

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions