automate-manual-test-cases: gate "done" on actual test execution#149
Open
DavertMik wants to merge 1 commit into
Open
automate-manual-test-cases: gate "done" on actual test execution#149DavertMik wants to merge 1 commit into
DavertMik wants to merge 1 commit into
Conversation
The skill treated running the generated test as a step but never made a passing run a precondition for reporting completion. In a real session the agent wrote a CodeceptJS suite, `tsc` passed, `npx codeceptjs run` stopped at bootstrap on a missing env var (`BETA_MAIN_USER_GENERAL_TOKEN`) — yet it still presented a "Implemented / coverage" table that read as done. The user could not tell the tests had never executed. Changes: - Add a hard completion gate: never report done / present coverage as complete unless every generated test actually executed and passed. Compiling or type-checking is not done. - Add step 4.0 "Check Run Prerequisites" — read .env(.example)/config for required env vars/credentials early so blockers surface before writing a full suite, not after. - Distinguish a framework bootstrap/environment blocker (missing env var, failed include, unreachable service) from a test failure: it is ⛔ BLOCKED, not healed and not done. - Error Handling: report BLOCKED at the top with the exact missing prerequisite; never claim unrun tests pass or are implemented. - FINAL_SUMMARY_TEMPLATE: add Status (✅ Done |⚠️ Needs review | ⛔ Blocked) and an Executed count; Done requires Executed == generated and all passed. Add a Blocked example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
automate-manual-test-casesskill listed "Execute test" as a step but never made a passing run a precondition for reporting completion. In a real Testeiya session the agent:prettier+tsc --noEmit(both passed),npx codeceptjs run …— which stopped at framework bootstrap because env varBETA_MAIN_USER_GENERAL_TOKENwas missing (thetestomatApiinclude needs it),The tests never executed, but the summary looked complete. The user could not tell, and (understandably) exploded: "did you execute tests?" → "WHAT THE FUCK??? HOW DID YOU WRITE THEM THEN??"
Fix
Make actual execution a hard gate on "done":
.env(.example)/ config for required env vars, tokens, and credentials early (right after project analysis) so a run blocker surfaces before a full suite is written, not after.Statusfield (✅ Done | ⚠️ Needs review | ⛔ Blocked) and anExecutedcount —✅ Doneonly whenExecuted == Tests generatedand all passed. Added a Blocked example modeled on the real failure.Scope
Docs-only (skill markdown). No behavior change beyond the model's own reporting discipline.
🤖 Generated with Claude Code