Skip to content

Externalize auto-test harness prompts into package prompt/#376

Open
tninja wants to merge 4 commits into
mainfrom
kang_feat
Open

Externalize auto-test harness prompts into package prompt/#376
tninja wants to merge 4 commits into
mainfrom
kang_feat

Conversation

@tninja
Copy link
Copy Markdown
Owner

@tninja tninja commented May 31, 2026

This moves the auto-test harness prompts out of per-project .ai.code.files/harness storage and into the package-installed prompt/ directory.

The harness code now resolves the ai-code package installation directory, creates prompt/ there on demand, and writes generated harness files into that package path instead of the project tree. Existing bundled prompt files are preserved if they are already present, and the prompt suffix now references the local file with the expected @ form.

If a packaged prompt file is missing or cannot be created, the code falls back to the inline harness suffix instead of failing the send path. I also updated the harness tests to cover the new package prompt directory behavior, bundled-file preservation, on-demand generation, and the fallback path. Verification: emacs -Q -batch -L test/stubs -L . --eval '(setq load-prefer-newer t)' -l ert -l test/test_ai-code-harness.el -f ert-run-tests-batch-and-exit and emacs -Q -batch -L test/stubs -L . --eval '(setq load-prefer-newer t)' -f batch-byte-compile test/test_ai-code-harness.el.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 647ce6de33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ai-code-harness.el
(insert content)
(unless (bolp)
(insert "\n")))
(unless (file-exists-p file-path)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve user-customized test prompts

When the package now ships prompt/test-after-change*.v1.md, this guard prevents regenerating the harness file even though content was just computed from the current ai-code-test-after-code-change-suffix. In any installation with the bundled prompt present, a user who customizes that suffix (or changes it in the session) still gets a reference to the stale bundled file, so the AI ignores the user's configured test-after-change instruction. Consider updating generated files when their dynamic/customized content differs, or only preserving truly static bundled prompts.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR relocates auto-test harness prompt files from per-project .ai.code.files/harness/ to the package's installation prompt/ directory. Bundled prompt markdown files are now shipped with the package and used directly; missing files are generated on demand at the package location, and a file-error falls back to the inline suffix. Tests are updated to mock locate-library and verify the new storage layout, preservation of bundled files, on-demand creation, and the fallback path.

Changes:

  • Removed ai-code-auto-test-harness-cache-directory defcustom and its autoload; introduced ai-code--package-directory and reworked ai-code--auto-test-harness-directory to point to the package prompt/ folder.
  • Bundled six harness markdown files under prompt/ (test-after-change, tdd, tdd-with-refactoring, plus their diagnostics variants); ensure-auto-test-harness-file no longer overwrites existing files, and the suffix string now prepends @ to the path returned by ai-code--auto-test-harness-prompt-path (which itself no longer prepends @).
  • Updated harness tests to set up a fake package directory + project directory, mock locate-library, and assert package-prompt behavior; added a fallback test for file-error.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
ai-code-harness.el Drops cache-directory custom; resolves harness dir via locate-library "ai-code"; preserves bundled files; emits @-prefixed prompt path in suffix.
ai-code-autoloads.el Removes the dropped defcustom autoload.
prompt/test-after-change.v1.md Bundled inline test-after-change instruction.
prompt/test-after-change-diagnostics.v1.md Bundled diagnostics variant of test-after-change.
prompt/tdd.v1.md Bundled TDD red/green stage instruction.
prompt/tdd-with-refactoring.v1.md Bundled TDD red/green/blue stage instruction.
prompt/tdd-diagnostics.v1.md Bundled TDD red/green with diagnostics MCP guidance.
prompt/tdd-with-refactoring-diagnostics.v1.md Bundled TDD red/green/blue with diagnostics MCP guidance.
test/test_ai-code-harness.el Rewrites tests for the package-prompt model: directory default, on-demand creation, bundled-file preservation, file-error fallback, and updated end-to-end suffix tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants