From 412bb0b86cb09f92b599862fe1ea1a91bc48d2bd Mon Sep 17 00:00:00 2001 From: Jynn Nelson Date: Fri, 17 Apr 2026 11:07:32 +0200 Subject: [PATCH 01/20] [draft] Link to proposed LLM policy --- src/about-this-guide.md | 3 ++- src/contributing.md | 6 ++++++ src/getting-started.md | 2 ++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 4dc453a204..57f6e67686 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -72,6 +72,7 @@ You might also find the following sites useful: - [rustc API docs] -- rustdoc documentation for the compiler, devtools, and internal tools - [Forge] -- contains documentation about Rust infrastructure, team procedures, and more +- `rust-lang/rust`'s [LLM policy] - [compiler-team] -- the home-base for the Rust compiler team, with description of the team procedures, active working groups, and the team calendar. - [std-dev-guide] -- a similar guide for developing the standard library. @@ -93,7 +94,7 @@ You might also find the following sites useful: For example, searching for `* -> vec` should find all functions that return a `Vec`. _Hint:_ Find more tips and keyboard shortcuts by typing `?` on any Rustdoc page! - +[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html [rustc dev guide]: about-this-guide.md [gsearchdocs]: https://www.google.com/search?q=site:doc.rust-lang.org+your+query+here [stddocs]: https://doc.rust-lang.org/std diff --git a/src/contributing.md b/src/contributing.md index 7f4779515b..05685d9f2f 100644 --- a/src/contributing.md +++ b/src/contributing.md @@ -517,6 +517,12 @@ This is used for [RFCs], issues, and pull requests. [rfcbot]: https://github.com/anp/rfcbot-rs/ [RFCs]: https://github.com/rust-lang/rfcs +## LLM policy + +See [Forge][LLM policy]. + +[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html + ## Helpful links and information This section has moved to the ["About this guide"] chapter. diff --git a/src/getting-started.md b/src/getting-started.md index a5da912aa4..12167347d4 100644 --- a/src/getting-started.md +++ b/src/getting-started.md @@ -48,6 +48,8 @@ But avoid using LLM tools that generate long, complex words. In daily teamwork, **simple and clear words** are best for easy understanding. Even small typos or grammar mistakes can make you seem more human, and people connect better with humans. +See also [our LLM policy](https://forge.rust-lang.org/policies/llm-usage.html). + ### Experts Not all `t-compiler` members are experts on all parts of `rustc`; From 093585280f462bf192c274eb2a532a89b2158a4b Mon Sep 17 00:00:00 2001 From: jyn Date: Sat, 23 May 2026 06:41:40 +0200 Subject: [PATCH 02/20] add guidance for working with LLMs --- src/SUMMARY.md | 1 + src/llm-guidance.md | 15 +++++++++++++++ 2 files changed, 16 insertions(+) create mode 100644 src/llm-guidance.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9a98611860..29a259130d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -50,6 +50,7 @@ - [About the compiler team](./compiler-team.md) - [Using Git](./git.md) - [Mastering @rustbot](./rustbot.md) +- [Running LLMs](./llm-guidance.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [Implementing new language features](./implementing-new-features.md) - [Stability guarantees](./stability-guarantees.md) diff --git a/src/llm-guidance.md b/src/llm-guidance.md new file mode 100644 index 0000000000..1e668d1624 --- /dev/null +++ b/src/llm-guidance.md @@ -0,0 +1,15 @@ +# Running LLMs + +This is a non-binding list of suggestions for working with LLMs. +This is not our moderation policy; see [Forge][LLM policy]. + +## Review bots + +- If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. +- Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. + +## LLM-authored code + +- We recommend, but do not require, using a second LLM for adversarial local review before publishing your changes. + +[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html From 356fb2cbd22a77ec5d065ffbd2ea9ecd7770e1bd Mon Sep 17 00:00:00 2001 From: jyn Date: Sat, 6 Jun 2026 13:16:48 +0200 Subject: [PATCH 03/20] Suggest using an LLM to generate tools, rather than making the LLM the tool --- src/llm-guidance.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index 1e668d1624..b93b3c6d52 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -7,9 +7,13 @@ This is not our moderation policy; see [Forge][LLM policy]. - If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. - Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. +- Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. + This both saves on token costs, and allows people who are not using an LLM to run the analysis. ## LLM-authored code - We recommend, but do not require, using a second LLM for adversarial local review before publishing your changes. +- Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. + You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. [LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html From 9d85466cd3370a0b515b7b498d049c5a327a4aac Mon Sep 17 00:00:00 2001 From: jyn Date: Thu, 11 Jun 2026 17:58:39 +0200 Subject: [PATCH 04/20] add note that LLMs prefer their own output --- src/llm-guidance.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index b93b3c6d52..d76f819fb9 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -9,6 +9,7 @@ This is not our moderation policy; see [Forge][LLM policy]. - Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. - Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. This both saves on token costs, and allows people who are not using an LLM to run the analysis. +- Note that LLMs sometimes demonstrate a preference for LLM output, particularly output from the same model. ## LLM-authored code From 2ee3725d2944124eca2363682f12d4e2371dae95 Mon Sep 17 00:00:00 2001 From: jyn Date: Thu, 18 Jun 2026 02:01:10 +0200 Subject: [PATCH 05/20] extend llm guidance --- src/llm-guidance.md | 52 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 6 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index d76f819fb9..bbb12bd0e3 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -1,20 +1,60 @@ -# Running LLMs +# Working with LLMs This is a non-binding list of suggestions for working with LLMs. This is not our moderation policy; see [Forge][LLM policy]. -## Review bots +[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html + +## Automated checks and LLM review - If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. - Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. - Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. This both saves on token costs, and allows people who are not using an LLM to run the analysis. -- Note that LLMs sometimes demonstrate a preference for LLM output, particularly output from the same model. +- LLMs sometimes prefer LLM-generated output, particularly output from the same + model. Treat LLM review as advisory, and do not rely on the model that + produced a change as its only reviewer. -## LLM-authored code +## Writing LLM-created code -- We recommend, but do not require, using a second LLM for adversarial local review before publishing your changes. +- We recommend, but do not require, using a different model for adversarial + local review before publishing your changes. This does not replace human + self-review. - Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. -[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html +[`ast-grep`]: https://astgrep.com/ + +## Reviewing LLM-created code + +Point people to [#llm-mentoring](https://rust-lang.zulipchat.com/#narrow/channel/606558-llm-mentoring/) liberally. +Deal with low-quality PRs by closing the PR and asking the author to follow the "solicited" rule in the Forge policy. +Deal with borderline PRs by asking the author to do work that shows they're paying attention; it's ok to ask for that work before you've put much time into review yourself. +For example, ask them to reproduce the bug, explain the change in their own +words, identify relevant edge cases, or add or justify tests. + +If you find yourself suggesting the same fixes on multiple PRs, +consider adding them to the dev-guide. + +## Disclosure guidelines + +Disclose the *extent* and *purpose* of your LLM use. +We don't care which model you used, but we do care whether you used the LLM to +implement the idea or to come up with it. +Write the disclosure yourself. You may use an LLM to privately review a +disclosure you have written, but not to draft or rewrite it. + +**Good** examples: + +> LLM disclosure: I wrote the three commits by hand after viewing profiling data. I used an LLM to review the commits before submitting. The LLM identified that ImplString::is_negative was no longer used, so I removed that field by hand. + +> Created with the help of Claude Code, which: +> - traced the missing cache hits to the unconditional return(pass) by inspecting Fastly vs CloudFront headers, +> - reviewed the git history to understand why the snippet was added, and +> - made the VCL change. + +**Bad** examples: + +> 🤖 Generated with Claude Code + +> Co-Authored-By: Claude Opus 4.6 (1M context) From 403887ed3ca65f464cc7afed161d98293659610f Mon Sep 17 00:00:00 2001 From: jyn Date: Tue, 28 Jul 2026 13:54:01 +0200 Subject: [PATCH 06/20] extend LLM guidance with a summary of the policy --- src/llm-guidance.md | 135 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 109 insertions(+), 26 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index bbb12bd0e3..5426f6f894 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -1,10 +1,70 @@ # Working with LLMs -This is a non-binding list of suggestions for working with LLMs. -This is not our moderation policy; see [Forge][LLM policy]. +This is a list of guidelines for working with LLMs, as well as a summary of the moderation policy. +This is not the policy itself; see [Forge][LLM policy]. +If the two conflict, Forge is canonical. [LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html +## Writing LLM-created code + +### Rules + +Before anything else, you **must** find a reviewer who volunteers to review your PR. +LLM PRs without an assigned reviewer will be closed. + +Disclose your use of LLMs, following the disclosure guidelines below. +Write the disclosure yourself. +You may use an LLM to privately review a disclosure you have written, but not to draft or rewrite it. + +You may *not* use an LLM to write doc-comments, `// SAFETY` comments, diagnostics, or soundness-critical code. +If you don't know what counts as soundness-critical, discuss it with your reviewer. + +**Write your own PR description and comments**. +LLM-created PR descriptions are banned. +LLM-created Github comments are banned. +We want to hear from you, not from your agent. + +### Guidelines + +Start with one PR at a time. +We understand it's easy to be excited about contributing to Rust, but your PRs are not only a gift but a responsibility for the reviewers. +Go slow. + +Avoid E-easy and E-mentor issues. +Those are intended for people to get familiar with the project, not for LLMs. + +Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. +You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. + +Read the documentation. Strive for quality. Review your own PR before opening it: +Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? + +We recommend using a different model for adversarial local review before publishing your changes. +This does not replace human self-review. + +[`ast-grep`]: https://astgrep.com/ + +#### Disclosure guidelines + +Disclose the *extent* and *purpose* of your LLM use. +We don't care which model you used, but we do care whether you used the LLM to implement the idea or to come up with it. + +**Good** examples: + +> LLM disclosure: I wrote the three commits by hand after viewing profiling data. I used an LLM to review the commits before submitting. The LLM identified that `ImplString::is_negative` was no longer used, so I removed that field by hand. + +> Created with the help of Claude Code, which: +> - traced the missing cache hits to the unconditional `return(pass)` by inspecting Fastly vs CloudFront headers, +> - reviewed the git history to understand why the snippet was added, and +> - made the VCL change. + +**Bad** examples: + +> 🤖 Generated with Claude Code + +> Co-Authored-By: Claude Opus 4.6 (1M context) + ## Automated checks and LLM review - If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. @@ -15,20 +75,25 @@ This is not our moderation policy; see [Forge][LLM policy]. model. Treat LLM review as advisory, and do not rely on the model that produced a change as its only reviewer. -## Writing LLM-created code +## Reviewing LLM-created code -- We recommend, but do not require, using a different model for adversarial - local review before publishing your changes. This does not replace human - self-review. -- Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. - You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. +First, add the new `ai-assisted` label to the PR. -[`ast-grep`]: https://astgrep.com/ +### Rules -## Reviewing LLM-created code +We expect everyone to follow the new policy, not just authors. +That means it is **your responsibility** to check whether an `ai-assisted` PR touches an area that's disallowed by the policy. +You may request that the author redo it without LLM-generated code, in which case this section doesn't apply. + +The following areas are currently banned: +- Code that affects soundness. If the author is not obviously experienced in the domain, you are required to close the PR. +- Diagnostics. All user-facing diagnostics must be human-written. +- Docs. All public doc-comments, and all `SAFETY` comments, must be human-written. + +### Guidelines Point people to [#llm-mentoring](https://rust-lang.zulipchat.com/#narrow/channel/606558-llm-mentoring/) liberally. -Deal with low-quality PRs by closing the PR and asking the author to follow the "solicited" rule in the Forge policy. +Deal with low-quality PRs by closing the PR and asking the author to follow the policy. Deal with borderline PRs by asking the author to do work that shows they're paying attention; it's ok to ask for that work before you've put much time into review yourself. For example, ask them to reproduce the bug, explain the change in their own words, identify relevant edge cases, or add or justify tests. @@ -36,25 +101,43 @@ words, identify relevant edge cases, or add or justify tests. If you find yourself suggesting the same fixes on multiple PRs, consider adding them to the dev-guide. -## Disclosure guidelines +#### Missing disclosure -Disclose the *extent* and *purpose* of your LLM use. -We don't care which model you used, but we do care whether you used the LLM to -implement the idea or to come up with it. -Write the disclosure yourself. You may use an LLM to privately review a -disclosure you have written, but not to draft or rewrite it. +If you see a PR that is "obviously" LLM-created without disclosing that use, you have the option—but not the responsibility—to close it unilaterally. -**Good** examples: +We suggest using the following wording: -> LLM disclosure: I wrote the three commits by hand after viewing profiling data. I used an LLM to review the commits before submitting. The LLM identified that ImplString::is_negative was no longer used, so I removed that field by hand. +> This PR appears to be LLM-generated without disclosing use of an LLM, +> so I am going to close this PR. +> You are welcome to open additional PRs as long as they follow our [policy][forge-page]. +> For more information, see [#llm-mentoring] on Zulip. -> Created with the help of Claude Code, which: -> - traced the missing cache hits to the unconditional return(pass) by inspecting Fastly vs CloudFront headers, -> - reviewed the git history to understand why the snippet was added, and -> - made the VCL change. +Examples of "obvious" LLM tells are: +- PR descriptions that are completely wrong/don't match the code. +- PR descriptions that state the exact tests that were run (e.g. `./x test --stage 1 tests/ui/.rs`) or useless tests such as `git diff --check`. +- Responses to reviewer questions that fall into one of the above categories. -**Bad** examples: +You do not have an obligation to detect LLM-created PRs; +you don't need to play detective. -> 🤖 Generated with Claude Code +PR templates will have a "Did you use an LLM?" question so that this rarely comes up. +If the author deleted the question without answering it, you can close the PR, no questions asked. -> Co-Authored-By: Claude Opus 4.6 (1M context) +#### Missing solicited reviewer + +If the PR discloses use, but does not assign a reviewer following the [experiment guidelines], you can close it similarly: + +> You've opened an LLM-generated PR, but it's in the normal review queue, which breaks our [policy][experiment guidelines]. +> I am going to close this PR. +> Please do not re-open it until you find a project member who has volunteered to review it. +> For more information, see [#llm-mentoring] on Zulip. + +#### Missing tests, low-quality, or not self-reviewed + +If a PR is clearly not ready for review, you do not have to review it. +It's ok to simply skim the PR and tell the author "you need to add tests before I can review this". +If you notice on your skim that the PR is clearly the wrong approach, it's ok to close the PR and tell the author to talk with you in the [#llm-mentoring] channel before opening a new PR. + +[experiment guidelines]: https://forge.rust-lang.org/policies/llm-usage.html#experiment-llm-created-code-changes-intended-for-review +[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ +[forge-page]: https://forge.rust-lang.org/policies/llm-usage.html From 0440a748091c7b6dc843408fcc14484996a0a029 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 09:04:02 +0200 Subject: [PATCH 07/20] flesh out author guidance --- src/llm-guidance.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index 5426f6f894..aff7197463 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -11,7 +11,8 @@ If the two conflict, Forge is canonical. ### Rules Before anything else, you **must** find a reviewer who volunteers to review your PR. -LLM PRs without an assigned reviewer will be closed. +LLM-created PRs without an assigned reviewer will be closed. +If you do not know where to find a reviewer, see [#llm-mentoring] on Zulip. Disclose your use of LLMs, following the disclosure guidelines below. Write the disclosure yourself. @@ -27,6 +28,8 @@ We want to hear from you, not from your agent. ### Guidelines +#### Before you write code + Start with one PR at a time. We understand it's easy to be excited about contributing to Rust, but your PRs are not only a gift but a responsibility for the reviewers. Go slow. @@ -34,15 +37,33 @@ Go slow. Avoid E-easy and E-mentor issues. Those are intended for people to get familiar with the project, not for LLMs. +#### While working + +When fixing a bug, reproduce the bug *before* fixing it. +Write a test that fails, fix the bug, then verify the test now succeeds. +Otherwise, you don't know that you were testing the right thing. + Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. -Read the documentation. Strive for quality. Review your own PR before opening it: +#### Before opening a PR + +Review your own PR before opening it: Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? +Run tests to verify your change works. +Do NOT report which tests you ran in the PR description; +that's useless to us, since CI will run them anyway. + +You must review diagnostic snapshots; don't simply `--bless` them away. + We recommend using a different model for adversarial local review before publishing your changes. This does not replace human self-review. +You must understand and be able to explain your own change and its edge cases. +Asking the LLM can be a starting point but is not sufficient. +You are responsible for your own code; you cannot disclaim responsibility to your agent. + [`ast-grep`]: https://astgrep.com/ #### Disclosure guidelines @@ -92,7 +113,7 @@ The following areas are currently banned: ### Guidelines -Point people to [#llm-mentoring](https://rust-lang.zulipchat.com/#narrow/channel/606558-llm-mentoring/) liberally. +Point people to [#llm-mentoring] liberally. Deal with low-quality PRs by closing the PR and asking the author to follow the policy. Deal with borderline PRs by asking the author to do work that shows they're paying attention; it's ok to ask for that work before you've put much time into review yourself. For example, ask them to reproduce the bug, explain the change in their own From 7216c57e5a695cee3a12e8045e22b10cf97ce02a Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 09:04:10 +0200 Subject: [PATCH 08/20] more author guidance; cross-references --- src/llm-guidance.md | 37 ++++++++++++++++++++++++++++++------- src/tests/best-practices.md | 2 +- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index aff7197463..183e81c10f 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -18,7 +18,8 @@ Disclose your use of LLMs, following the disclosure guidelines below. Write the disclosure yourself. You may use an LLM to privately review a disclosure you have written, but not to draft or rewrite it. -You may *not* use an LLM to write doc-comments, `// SAFETY` comments, diagnostics, or soundness-critical code. +You may *not* use an LLM to write doc-comments, `// SAFETY` comments, diagnostic wording, or soundness-critical code. +Write them yourself instead. If you don't know what counts as soundness-critical, discuss it with your reviewer. **Write your own PR description and comments**. @@ -37,11 +38,21 @@ Go slow. Avoid E-easy and E-mentor issues. Those are intended for people to get familiar with the project, not for LLMs. +Determine whether this is a *useful* and *well-scoped* change. +For example: + +- Search for related issues and PRs. +- Find relevant code, tests, git history, and Zulip discussion. +- If this is a cross-cutting change, consult the "cross-cutting" section of [the contributing docs](./contributing.md#pull-requests). +- Make the smallest change that fixes the problem. + Do not combine it with unrelated refactors or cleanups. + #### While working -When fixing a bug, reproduce the bug *before* fixing it. -Write a test that fails, fix the bug, then verify the test now succeeds. -Otherwise, you don't know that you were testing the right thing. +When fixing a bug, verify that your test fails before and succeeds after your change. +Consult [adding new tests](./tests/adding.md) and [best practices](./tests/best-practices.md) for test procedures. +Tests are absolutely required; either existing tests or new tests you write. +Untested LLM PRs will not be merged. Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. @@ -51,19 +62,31 @@ You may use an LLM for generating the instructions for that tool, but you should Review your own PR before opening it: Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? -Run tests to verify your change works. +[Run tests](tests/running.md) to verify your change works. Do NOT report which tests you ran in the PR description; that's useless to us, since CI will run them anyway. -You must review diagnostic snapshots; don't simply `--bless` them away. +You must [review diagnostic snapshots](tests/adding.md#step-4-review-the-output); +don't simply `--bless` them away. We recommend using a different model for adversarial local review before publishing your changes. This does not replace human self-review. +#### Understand your own change + You must understand and be able to explain your own change and its edge cases. Asking the LLM can be a starting point but is not sufficient. You are responsible for your own code; you cannot disclaim responsibility to your agent. +Some questions that you should be able to answer: + +- What is the original bug? When does it happen? How severe is it? What causes it? +- Why is this the right fix? Are there other fixes possible? What are their advantages or disadvantages? +- Are there any edge cases? Does your code handle them? +- What behavior is *unchanged*? What test establishes that? +- Why does your test trigger the bug? +- What are you still not certain about? + [`ast-grep`]: https://astgrep.com/ #### Disclosure guidelines @@ -107,7 +130,7 @@ That means it is **your responsibility** to check whether an `ai-assisted` PR to You may request that the author redo it without LLM-generated code, in which case this section doesn't apply. The following areas are currently banned: -- Code that affects soundness. If the author is not obviously experienced in the domain, you are required to close the PR. +- Code that affects soundness. If the author is not an org member who is experienced in the domain, you are required to close the PR. - Diagnostics. All user-facing diagnostics must be human-written. - Docs. All public doc-comments, and all `SAFETY` comments, must be human-written. diff --git a/src/tests/best-practices.md b/src/tests/best-practices.md index b6daffa668..4e6f40c4b7 100644 --- a/src/tests/best-practices.md +++ b/src/tests/best-practices.md @@ -140,7 +140,7 @@ A good rule of thumb is non-trivial things exercised in the test deserves some explanation to help other contributors to understand. This may include remarks on: -- How an ICE can get triggered if it's quite elaborate. +- How an ICE or bug can get triggered if it's quite elaborate. - Related issues and tests (e.g. this test is like another test but is kept separate because...). - Platform-specific behaviors. From 32dca76297ce06c9507f9fc061fa98ae4b37f00d Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 10:30:18 +0200 Subject: [PATCH 09/20] change tone from policy to mentorship --- src/llm-guidance.md | 56 +++++++++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index 183e81c10f..175b3d1e38 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -8,18 +8,35 @@ If the two conflict, Forge is canonical. ## Writing LLM-created code +LLMs are a valuable tool, but one that is easy to misuse. +The main risks are **overwhelming volume** and **lack of understanding**. +When writing your PR, keep in mind that there is a person on the other end who needs to review and understand your change. +Help us out by making your change small, targeted, and easy to review. + +Keep in mind this quote: + +> Programs must be written for people to read, and only incidentally for machines to execute. +> — Harold Abelson and Gerald Jay Sussman, [*Structure and Interpretation of Computer Programs*][sicp] + +[sicp]: https://web.mit.edu/6.001/6.037/sicp.pdf + ### Rules -Before anything else, you **must** find a reviewer who volunteers to review your PR. -LLM-created PRs without an assigned reviewer will be closed. -If you do not know where to find a reviewer, see [#llm-mentoring] on Zulip. +Before anything else, find a reviewer who volunteers to review your PR. +If you do not know where to find a reviewer, ask in [#llm-mentoring] on Zulip. +Your first message should say: + +- your relevant experience, so we can find an issue that's suitable for you +- which problem (or kind of problem) you want to work on +- (optional) ideas you have so far for a solution +- (optional) how you expect to test your solution Disclose your use of LLMs, following the disclosure guidelines below. Write the disclosure yourself. You may use an LLM to privately review a disclosure you have written, but not to draft or rewrite it. -You may *not* use an LLM to write doc-comments, `// SAFETY` comments, diagnostic wording, or soundness-critical code. -Write them yourself instead. +Write your own doc-comments, `// SAFETY` comments, diagnostic wording, and soundness-critical code. +As before, you can use an LLM to review your work, but not to write it from scratch. If you don't know what counts as soundness-critical, discuss it with your reviewer. **Write your own PR description and comments**. @@ -32,11 +49,11 @@ We want to hear from you, not from your agent. #### Before you write code Start with one PR at a time. -We understand it's easy to be excited about contributing to Rust, but your PRs are not only a gift but a responsibility for the reviewers. +Your PRs are not only a gift but a responsibility for reviewers. Go slow. -Avoid E-easy and E-mentor issues. -Those are intended for people to get familiar with the project, not for LLMs. +Do not use an LLM for `E-easy` issues; those are meant for you to write the code yourself. +Ask first before working on an `E-mentor` issue; mentors may not want to work with LLM-generated code. Determine whether this is a *useful* and *well-scoped* change. For example: @@ -63,22 +80,24 @@ Review your own PR before opening it: Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? [Run tests](tests/running.md) to verify your change works. -Do NOT report which tests you ran in the PR description; -that's useless to us, since CI will run them anyway. +Do NOT report which UI tests you ran in the PR description; +that's noise, since CI will run them anyway. +If you did manual testing or benchmarking, do report that, +but note that all LLM PRs must have automated tests. -You must [review diagnostic snapshots](tests/adding.md#step-4-review-the-output); +[Review diagnostic snapshots](tests/adding.md#step-4-review-the-output); don't simply `--bless` them away. We recommend using a different model for adversarial local review before publishing your changes. -This does not replace human self-review. +You're still responsible for reviewing your own changes yourself. #### Understand your own change -You must understand and be able to explain your own change and its edge cases. -Asking the LLM can be a starting point but is not sufficient. -You are responsible for your own code; you cannot disclaim responsibility to your agent. +We want you to understand and be able to explain your own change and its edge cases. +Asking the LLM can be a starting point but it's not the same as explaining it yourself. -Some questions that you should be able to answer: +Try explaining your change to yourself before opening the PR. +For example, ask yourself: - What is the original bug? When does it happen? How severe is it? What causes it? - Why is this the right fix? Are there other fixes possible? What are their advantages or disadvantages? @@ -87,6 +106,9 @@ Some questions that you should be able to answer: - Why does your test trigger the bug? - What are you still not certain about? +It's ok to be uncertain and to ask for help. +We would much rather help you because you're not sure than have you guess wrong and then have to reverse-engineer where you went wrong. + [`ast-grep`]: https://astgrep.com/ #### Disclosure guidelines @@ -138,7 +160,7 @@ The following areas are currently banned: Point people to [#llm-mentoring] liberally. Deal with low-quality PRs by closing the PR and asking the author to follow the policy. -Deal with borderline PRs by asking the author to do work that shows they're paying attention; it's ok to ask for that work before you've put much time into review yourself. +Deal with borderline PRs by asking the author to put in the work themselves rather than offloading it to you. For example, ask them to reproduce the bug, explain the change in their own words, identify relevant edge cases, or add or justify tests. From 9498d10f707a9fd0312dd346ba727dcfa5ff5e2d Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 11:44:43 +0200 Subject: [PATCH 10/20] add "maintainable code" section --- src/llm-guidance.md | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/src/llm-guidance.md b/src/llm-guidance.md index 175b3d1e38..d186df8779 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance.md @@ -10,8 +10,9 @@ If the two conflict, Forge is canonical. LLMs are a valuable tool, but one that is easy to misuse. The main risks are **overwhelming volume** and **lack of understanding**. -When writing your PR, keep in mind that there is a person on the other end who needs to review and understand your change. -Help us out by making your change small, targeted, and easy to review. +When writing your PR, keep in mind that there is a person on the other end who needs to review and understand your change, +and that other people in the project will need to read your code for years to come. +Help us out by making your change small, targeted, and high-quality. Keep in mind this quote: @@ -49,7 +50,7 @@ We want to hear from you, not from your agent. #### Before you write code Start with one PR at a time. -Your PRs are not only a gift but a responsibility for reviewers. +Keep your changes small enough that you and your reviewer can understand every part of them. Go slow. Do not use an LLM for `E-easy` issues; those are meant for you to write the code yourself. @@ -74,11 +75,45 @@ Untested LLM PRs will not be merged. Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. +Work in small steps. +Run tests after every meaningful change, so you know where you first went wrong. +Do not write all your code at once; that will make it very hard to know what broke. +Ideally, commit your changes in [small atomic commits] as you go. + +Verify your understanding against the existing code, documentation, and tests. +You can get better advice from your LLM by telling *it* to read the relevant materials. +Do not rely on the LLM as a source of truth. + +[small atomic commits](https://github.blog/developer-skills/github/write-better-commits-build-better-projects/#%e2%9a%9b%ef%b8%8f-resize-and-stabilize-the-commits) + +#### Write maintainable code + +Treat generated code as a *draft*, not a final product. + +Follow the style of surrounding code. +Use existing helpers and avoid duplicating logic or validation. + +Represent data in [one normalized place](https://react.dev/learn/choosing-the-state-structure); +trying to keep data in sync between two different places is a code smell. + +When practical, [make invalid states unrepresentable](https://kentcdodds.com/blog/make-impossible-states-impossible), [not just checked at construction time](https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-type-safety/). + +Write comments that say *why* you have done a thing, not *what* you have done. +It's ok to go into detail about non-obvious bugs. + +Preserve existing behavior. +Consider platform differences and error cases. + #### Before opening a PR Review your own PR before opening it: Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? +Remove outdated or prototyping code and debugging. + +Re-read the whole diff, *not* just your conversation with the agent. +Your reviewer is going to see your code, not your conversation. + [Run tests](tests/running.md) to verify your change works. Do NOT report which UI tests you ran in the PR description; that's noise, since CI will run them anyway. @@ -102,6 +137,8 @@ For example, ask yourself: - What is the original bug? When does it happen? How severe is it? What causes it? - Why is this the right fix? Are there other fixes possible? What are their advantages or disadvantages? - Are there any edge cases? Does your code handle them? +- Does the code have existing [invariants](https://brooker.co.za/blog/2023/07/28/ds-testing.html)? + Did you preserve those invariants? - What behavior is *unchanged*? What test establishes that? - Why does your test trigger the bug? - What are you still not certain about? From 50e2615ef9ae62d87edc16954e08d6b1c88738c0 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 11:45:16 +0200 Subject: [PATCH 11/20] split authoring and reviewing sections --- src/SUMMARY.md | 4 +- src/llm-guidance/index.md | 10 ++ src/llm-guidance/reviewing.md | 76 +++++++++++++ .../writing.md} | 106 ++---------------- 4 files changed, 100 insertions(+), 96 deletions(-) create mode 100644 src/llm-guidance/index.md create mode 100644 src/llm-guidance/reviewing.md rename src/{llm-guidance.md => llm-guidance/writing.md} (59%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 29a259130d..d082f6e253 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -50,7 +50,9 @@ - [About the compiler team](./compiler-team.md) - [Using Git](./git.md) - [Mastering @rustbot](./rustbot.md) -- [Running LLMs](./llm-guidance.md) +- [Running LLMs](./llm-guidance/index.md) + - [Writing code with LLMs](./llm-guidance/writing.md) + - [Reviewing code with LLMs](./llm-guidance/reviewing.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [Implementing new language features](./implementing-new-features.md) - [Stability guarantees](./stability-guarantees.md) diff --git a/src/llm-guidance/index.md b/src/llm-guidance/index.md new file mode 100644 index 0000000000..3709e7ead8 --- /dev/null +++ b/src/llm-guidance/index.md @@ -0,0 +1,10 @@ +# Working with LLMs + +This section is a list of guidelines for working with LLMs, as well as a summary of the moderation policy. +This is not the policy itself; see [Forge][LLM policy]. +If the two conflict, Forge is canonical. + +If you are writing code using an LLM, see [Writing LLM-created code](./writing.md). +If you are reviewing LLM-created code, or reviewing using an LLM, see [Reviewing with LLMs](./reviewing.md). + +[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html diff --git a/src/llm-guidance/reviewing.md b/src/llm-guidance/reviewing.md new file mode 100644 index 0000000000..4feab70ec3 --- /dev/null +++ b/src/llm-guidance/reviewing.md @@ -0,0 +1,76 @@ +## Using an LLM to review code + +- If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. +- Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. +- Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. + This both saves on token costs, and allows people who are not using an LLM to run the analysis. +- LLMs sometimes prefer LLM-generated output, particularly output from the same + model. Treat LLM review as advisory, and do not rely on the model that + produced a change as its only reviewer. + +## Reviewing LLM-created code + +First, add the new `ai-assisted` label to the PR. + +### Rules + +We expect everyone to follow the new policy, not just authors. +That means it is **your responsibility** to check whether an `ai-assisted` PR touches an area that's disallowed by the policy. +You may request that the author redo it without LLM-generated code, in which case this section doesn't apply. + +The following areas are currently banned: +- Code that affects soundness. If the author is not an org member who is experienced in the domain, you are required to close the PR. +- Diagnostics. All user-facing diagnostics must be human-written. +- Docs. All public doc-comments, and all `SAFETY` comments, must be human-written. + +### Guidelines + +Point people to [#llm-mentoring] liberally. +Deal with low-quality PRs by closing the PR and asking the author to follow the policy. +Deal with borderline PRs by asking the author to put in the work themselves rather than offloading it to you. +For example, ask them to reproduce the bug, explain the change in their own +words, identify relevant edge cases, or add or justify tests. + +If you find yourself suggesting the same fixes on multiple PRs, +consider adding them to the dev-guide. + +#### Missing disclosure + +If you see a PR that is "obviously" LLM-created without disclosing that use, you have the option—but not the responsibility—to close it unilaterally. + +We suggest using the following wording: + +> This PR appears to be LLM-generated without disclosing use of an LLM, +> so I am going to close this PR. +> You are welcome to open additional PRs as long as they follow our [policy][forge-page]. +> For more information, see [#llm-mentoring] on Zulip. + +Examples of "obvious" LLM tells are: +- PR descriptions that are completely wrong/don't match the code. +- PR descriptions that state the exact tests that were run (e.g. `./x test --stage 1 tests/ui/.rs`) or useless tests such as `git diff --check`. +- Responses to reviewer questions that fall into one of the above categories. + +You do not have an obligation to detect LLM-created PRs; +you don't need to play detective. + +PR templates will have a "Did you use an LLM?" question so that this rarely comes up. +If the author deleted the question without answering it, you can close the PR, no questions asked. + +#### Missing solicited reviewer + +If the PR discloses use, but does not assign a reviewer following the [experiment guidelines], you can close it similarly: + +> You've opened an LLM-generated PR, but it's in the normal review queue, which breaks our [policy][experiment guidelines]. +> I am going to close this PR. +> Please do not re-open it until you find a project member who has volunteered to review it. +> For more information, see [#llm-mentoring] on Zulip. + +#### Missing tests, low-quality, or not self-reviewed + +If a PR is clearly not ready for review, you do not have to review it. +It's ok to simply skim the PR and tell the author "you need to add tests before I can review this". +If you notice on your skim that the PR is clearly the wrong approach, it's ok to close the PR and tell the author to talk with you in the [#llm-mentoring] channel before opening a new PR. + +[experiment guidelines]: https://forge.rust-lang.org/policies/llm-usage.html#experiment-llm-created-code-changes-intended-for-review +[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ +[forge-page]: https://forge.rust-lang.org/policies/llm-usage.html diff --git a/src/llm-guidance.md b/src/llm-guidance/writing.md similarity index 59% rename from src/llm-guidance.md rename to src/llm-guidance/writing.md index d186df8779..c4ed09bc65 100644 --- a/src/llm-guidance.md +++ b/src/llm-guidance/writing.md @@ -1,12 +1,4 @@ -# Working with LLMs - -This is a list of guidelines for working with LLMs, as well as a summary of the moderation policy. -This is not the policy itself; see [Forge][LLM policy]. -If the two conflict, Forge is canonical. - -[LLM policy]: https://forge.rust-lang.org/policies/llm-usage.html - -## Writing LLM-created code +# Writing LLM-created code LLMs are a valuable tool, but one that is easy to misuse. The main risks are **overwhelming volume** and **lack of understanding**. @@ -20,8 +12,9 @@ Keep in mind this quote: > — Harold Abelson and Gerald Jay Sussman, [*Structure and Interpretation of Computer Programs*][sicp] [sicp]: https://web.mit.edu/6.001/6.037/sicp.pdf +[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ -### Rules +## Rules Before anything else, find a reviewer who volunteers to review your PR. If you do not know where to find a reviewer, ask in [#llm-mentoring] on Zulip. @@ -45,9 +38,9 @@ LLM-created PR descriptions are banned. LLM-created Github comments are banned. We want to hear from you, not from your agent. -### Guidelines +## Guidelines -#### Before you write code +### Before you write code Start with one PR at a time. Keep your changes small enough that you and your reviewer can understand every part of them. @@ -65,7 +58,7 @@ For example: - Make the smallest change that fixes the problem. Do not combine it with unrelated refactors or cleanups. -#### While working +### While working When fixing a bug, verify that your test fails before and succeeds after your change. Consult [adding new tests](./tests/adding.md) and [best practices](./tests/best-practices.md) for test procedures. @@ -84,9 +77,9 @@ Verify your understanding against the existing code, documentation, and tests. You can get better advice from your LLM by telling *it* to read the relevant materials. Do not rely on the LLM as a source of truth. -[small atomic commits](https://github.blog/developer-skills/github/write-better-commits-build-better-projects/#%e2%9a%9b%ef%b8%8f-resize-and-stabilize-the-commits) +[small atomic commits]: https://github.blog/developer-skills/github/write-better-commits-build-better-projects/#%e2%9a%9b%ef%b8%8f-resize-and-stabilize-the-commits -#### Write maintainable code +### Write maintainable code Treat generated code as a *draft*, not a final product. @@ -104,7 +97,7 @@ It's ok to go into detail about non-obvious bugs. Preserve existing behavior. Consider platform differences and error cases. -#### Before opening a PR +### Before opening a PR Review your own PR before opening it: Does it make sense? Can you tell what the goal of the PR is? Does it achieve that goal? @@ -126,7 +119,7 @@ don't simply `--bless` them away. We recommend using a different model for adversarial local review before publishing your changes. You're still responsible for reviewing your own changes yourself. -#### Understand your own change +### Understand your own change We want you to understand and be able to explain your own change and its edge cases. Asking the LLM can be a starting point but it's not the same as explaining it yourself. @@ -148,7 +141,7 @@ We would much rather help you because you're not sure than have you guess wrong [`ast-grep`]: https://astgrep.com/ -#### Disclosure guidelines +### Disclosure guidelines Disclose the *extent* and *purpose* of your LLM use. We don't care which model you used, but we do care whether you used the LLM to implement the idea or to come up with it. @@ -167,80 +160,3 @@ We don't care which model you used, but we do care whether you used the LLM to i > 🤖 Generated with Claude Code > Co-Authored-By: Claude Opus 4.6 (1M context) - -## Automated checks and LLM review - -- If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. -- Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. -- Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. - This both saves on token costs, and allows people who are not using an LLM to run the analysis. -- LLMs sometimes prefer LLM-generated output, particularly output from the same - model. Treat LLM review as advisory, and do not rely on the model that - produced a change as its only reviewer. - -## Reviewing LLM-created code - -First, add the new `ai-assisted` label to the PR. - -### Rules - -We expect everyone to follow the new policy, not just authors. -That means it is **your responsibility** to check whether an `ai-assisted` PR touches an area that's disallowed by the policy. -You may request that the author redo it without LLM-generated code, in which case this section doesn't apply. - -The following areas are currently banned: -- Code that affects soundness. If the author is not an org member who is experienced in the domain, you are required to close the PR. -- Diagnostics. All user-facing diagnostics must be human-written. -- Docs. All public doc-comments, and all `SAFETY` comments, must be human-written. - -### Guidelines - -Point people to [#llm-mentoring] liberally. -Deal with low-quality PRs by closing the PR and asking the author to follow the policy. -Deal with borderline PRs by asking the author to put in the work themselves rather than offloading it to you. -For example, ask them to reproduce the bug, explain the change in their own -words, identify relevant edge cases, or add or justify tests. - -If you find yourself suggesting the same fixes on multiple PRs, -consider adding them to the dev-guide. - -#### Missing disclosure - -If you see a PR that is "obviously" LLM-created without disclosing that use, you have the option—but not the responsibility—to close it unilaterally. - -We suggest using the following wording: - -> This PR appears to be LLM-generated without disclosing use of an LLM, -> so I am going to close this PR. -> You are welcome to open additional PRs as long as they follow our [policy][forge-page]. -> For more information, see [#llm-mentoring] on Zulip. - -Examples of "obvious" LLM tells are: -- PR descriptions that are completely wrong/don't match the code. -- PR descriptions that state the exact tests that were run (e.g. `./x test --stage 1 tests/ui/.rs`) or useless tests such as `git diff --check`. -- Responses to reviewer questions that fall into one of the above categories. - -You do not have an obligation to detect LLM-created PRs; -you don't need to play detective. - -PR templates will have a "Did you use an LLM?" question so that this rarely comes up. -If the author deleted the question without answering it, you can close the PR, no questions asked. - -#### Missing solicited reviewer - -If the PR discloses use, but does not assign a reviewer following the [experiment guidelines], you can close it similarly: - -> You've opened an LLM-generated PR, but it's in the normal review queue, which breaks our [policy][experiment guidelines]. -> I am going to close this PR. -> Please do not re-open it until you find a project member who has volunteered to review it. -> For more information, see [#llm-mentoring] on Zulip. - -#### Missing tests, low-quality, or not self-reviewed - -If a PR is clearly not ready for review, you do not have to review it. -It's ok to simply skim the PR and tell the author "you need to add tests before I can review this". -If you notice on your skim that the PR is clearly the wrong approach, it's ok to close the PR and tell the author to talk with you in the [#llm-mentoring] channel before opening a new PR. - -[experiment guidelines]: https://forge.rust-lang.org/policies/llm-usage.html#experiment-llm-created-code-changes-intended-for-review -[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ -[forge-page]: https://forge.rust-lang.org/policies/llm-usage.html From c382799422700681c2d65623668bf3c556be9e56 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 11:57:35 +0200 Subject: [PATCH 12/20] be more clear about what i mean around linters --- src/llm-guidance/reviewing.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/llm-guidance/reviewing.md b/src/llm-guidance/reviewing.md index 4feab70ec3..7bb722195f 100644 --- a/src/llm-guidance/reviewing.md +++ b/src/llm-guidance/reviewing.md @@ -1,13 +1,20 @@ +# Reviewing with LLMs + ## Using an LLM to review code - If a more reliable tool, such as a linter or formatter, already exists for the language you're writing, we strongly suggest using that tool instead of or in addition to the LLM. - Configure LLM review tools to reduce false positives and excessive focus on trivialities, as these are common, exhausting failure modes. -- Wherever possible, ask an LLM to *generate a linter*, which you then tell it to run. +- Wherever possible, ask an LLM to *generate or configure a linter*, which you then tell it to run. This both saves on token costs, and allows people who are not using an LLM to run the analysis. + For example, if you have a codebase-specific wrapper around command spawning, + rather than getting an LLM to look for places where you should use the wrapper, + [configure clippy to disallow `Command::new`][disallowed-methods]. - LLMs sometimes prefer LLM-generated output, particularly output from the same model. Treat LLM review as advisory, and do not rely on the model that produced a change as its only reviewer. +[disallowed-methods]: https://doc.rust-lang.org/clippy/lint_configuration.html#disallowed-methods + ## Reviewing LLM-created code First, add the new `ai-assisted` label to the PR. From be072c8db95db441dea2414076d0b9353e19c52b Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:07:00 +0200 Subject: [PATCH 13/20] add more links --- src/llm-guidance/writing.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index c4ed09bc65..8366b5c79d 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -55,9 +55,11 @@ For example: - Search for related issues and PRs. - Find relevant code, tests, git history, and Zulip discussion. - If this is a cross-cutting change, consult the "cross-cutting" section of [the contributing docs](./contributing.md#pull-requests). -- Make the smallest change that fixes the problem. +- [Make the smallest change that fixes the problem][small-cls]. Do not combine it with unrelated refactors or cleanups. +[small-cls]: https://google.github.io/eng-practices/review/developer/small-cls.html + ### While working When fixing a bug, verify that your test fails before and succeeds after your change. @@ -91,9 +93,11 @@ trying to keep data in sync between two different places is a code smell. When practical, [make invalid states unrepresentable](https://kentcdodds.com/blog/make-impossible-states-impossible), [not just checked at construction time](https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-type-safety/). -Write comments that say *why* you have done a thing, not *what* you have done. +[Write comments that say *why*][mit-comment-style] you have done a thing, not *what* you have done. It's ok to go into detail about non-obvious bugs. +[mit-comment-style]: https://mitcommlab.mit.edu/broad/commkit/coding-and-comment-style/ + Preserve existing behavior. Consider platform differences and error cases. From b2deb36902d3441af713041988d72f3248551a4a Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:16:40 +0200 Subject: [PATCH 14/20] move correctness suggestions to a better chapter --- src/conventions.md | 24 ++++++++++++++++++++++++ src/llm-guidance/writing.md | 17 +---------------- 2 files changed, 25 insertions(+), 16 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index 0440fb0c5e..f30921f35b 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -141,6 +141,30 @@ if foo { If you want to leave a note in the codebase, use `// FIXME` instead. +### Follow the style of surrounding code. + +Use existing helpers and avoid duplicating logic or validation. + +### Represent data in [one normalized place](https://react.dev/learn/choosing-the-state-structure) + +Trying to keep data in sync between two different places is a code smell. + +### Make invalid states unrepresentable + +When practical, [make invalid states unrepresentable](https://kentcdodds.com/blog/make-impossible-states-impossible), [not just checked at construction time](https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-type-safety/). + +### Write useful comments + +[Write comments that say *why*][mit-comment-style] you have done a thing, not *what* you have done. +It's ok to go into detail about non-obvious bugs. + +[mit-comment-style]: https://mitcommlab.mit.edu/broad/commkit/coding-and-comment-style/ + +### Preserve existing behavior. + +Consider platform differences and error cases. +Look for relevant tests that exercise the edge cases. + ## Using crates from crates.io diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index 8366b5c79d..3a35896712 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -84,22 +84,7 @@ Do not rely on the LLM as a source of truth. ### Write maintainable code Treat generated code as a *draft*, not a final product. - -Follow the style of surrounding code. -Use existing helpers and avoid duplicating logic or validation. - -Represent data in [one normalized place](https://react.dev/learn/choosing-the-state-structure); -trying to keep data in sync between two different places is a code smell. - -When practical, [make invalid states unrepresentable](https://kentcdodds.com/blog/make-impossible-states-impossible), [not just checked at construction time](https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-type-safety/). - -[Write comments that say *why*][mit-comment-style] you have done a thing, not *what* you have done. -It's ok to go into detail about non-obvious bugs. - -[mit-comment-style]: https://mitcommlab.mit.edu/broad/commkit/coding-and-comment-style/ - -Preserve existing behavior. -Consider platform differences and error cases. +Follow the [correctness conventions](../conventions.md#cc). ### Before opening a PR From 3e87cf6e0df7e31088c05c04035e2186fc20a018 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:23:16 +0200 Subject: [PATCH 15/20] address Sasha's review comments --- src/about-this-guide.md | 3 ++- src/llm-guidance/reviewing.md | 42 +++++++++++++++++++++++++++-------- 2 files changed, 35 insertions(+), 10 deletions(-) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 57f6e67686..cc98ad7b2b 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -72,7 +72,7 @@ You might also find the following sites useful: - [rustc API docs] -- rustdoc documentation for the compiler, devtools, and internal tools - [Forge] -- contains documentation about Rust infrastructure, team procedures, and more -- `rust-lang/rust`'s [LLM policy] +- [`rust-lang/rust`]'s [LLM policy] - [compiler-team] -- the home-base for the Rust compiler team, with description of the team procedures, active working groups, and the team calendar. - [std-dev-guide] -- a similar guide for developing the standard library. @@ -114,3 +114,4 @@ You might also find the following sites useful: [std-dev-guide]: https://std-dev-guide.rust-lang.org/ [rust-analyzer book]: https://rust-analyzer.github.io/book/ [z]: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler +[`rust-lang/rust`]: https://github.com/rust-lang/rust/ diff --git a/src/llm-guidance/reviewing.md b/src/llm-guidance/reviewing.md index 7bb722195f..44ae4b9e6c 100644 --- a/src/llm-guidance/reviewing.md +++ b/src/llm-guidance/reviewing.md @@ -28,7 +28,22 @@ You may request that the author redo it without LLM-generated code, in which cas The following areas are currently banned: - Code that affects soundness. If the author is not an org member who is experienced in the domain, you are required to close the PR. - Diagnostics. All user-facing diagnostics must be human-written. -- Docs. All public doc-comments, and all `SAFETY` comments, must be human-written. +- Docs. All public doc-comments, and all `// SAFETY` comments, must be human-written. + +"Code that affects soundness" is both broader and narrower than it sounds. +It's broader because almost all of the compiler is relevant to soundness; +it's narrower because there's quite a lot of rust-lang/rust that isn't the compiler +(library, bootstrap, compiletest, rustdoc, CI, ...). + +If in doubt, we suggest this criteria: +Do not allow LLM-generated code for parts of the compiler where [wrong code does not look wrong][joel-wrong]. +Ultimately, this is up to your judgement as a reviewer. + +[joel-wrong]: https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ + +You are still expected to respect your [r+ rights](../compiler-team.html#r-rights). +Please do not merge PRs unless you are confident in that part of that code, +even if the maintainer does not wish to review LLM PRs. ### Guidelines @@ -47,10 +62,14 @@ If you see a PR that is "obviously" LLM-created without disclosing that use, you We suggest using the following wording: -> This PR appears to be LLM-generated without disclosing use of an LLM, -> so I am going to close this PR. -> You are welcome to open additional PRs as long as they follow our [policy][forge-page]. -> For more information, see [#llm-mentoring] on Zulip. +```markdown +This PR appears to be LLM-generated without disclosing use of an LLM, so I am going to close this PR. +You are welcome to open additional PRs as long as they follow our [policy][forge-page]. +For more information, see [#llm-mentoring] on Zulip. + +[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ +[forge-page]: https://forge.rust-lang.org/policies/llm-usage.html +``` Examples of "obvious" LLM tells are: - PR descriptions that are completely wrong/don't match the code. @@ -67,10 +86,15 @@ If the author deleted the question without answering it, you can close the PR, n If the PR discloses use, but does not assign a reviewer following the [experiment guidelines], you can close it similarly: -> You've opened an LLM-generated PR, but it's in the normal review queue, which breaks our [policy][experiment guidelines]. -> I am going to close this PR. -> Please do not re-open it until you find a project member who has volunteered to review it. -> For more information, see [#llm-mentoring] on Zulip. +```markdown +You've opened an LLM-generated PR, but it's in the normal review queue, which breaks our [policy][experiment guidelines]. +I am going to close this PR. +Please do not re-open it until you find a project member who has volunteered to review it. +For more information, see [#llm-mentoring] on Zulip. + +[#llm-mentoring]: https://rust-lang.zulipchat.com/join/rlfvpemsaacs3pfi6kwqnqjb/ +[forge-page]: https://forge.rust-lang.org/policies/llm-usage.html +``` #### Missing tests, low-quality, or not self-reviewed From abf7311b2f5f6d6566f0ceb117edb089d068ffd9 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:38:23 +0200 Subject: [PATCH 16/20] fix links --- src/llm-guidance/writing.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index 3a35896712..4ea43ae008 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -34,9 +34,9 @@ As before, you can use an LLM to review your work, but not to write it from scra If you don't know what counts as soundness-critical, discuss it with your reviewer. **Write your own PR description and comments**. +We want to hear from you, not from your agent. LLM-created PR descriptions are banned. LLM-created Github comments are banned. -We want to hear from you, not from your agent. ## Guidelines @@ -54,7 +54,7 @@ For example: - Search for related issues and PRs. - Find relevant code, tests, git history, and Zulip discussion. -- If this is a cross-cutting change, consult the "cross-cutting" section of [the contributing docs](./contributing.md#pull-requests). +- If this is a cross-cutting change, consult the "cross-cutting" section of [the contributing docs](../contributing.md#pull-requests). - [Make the smallest change that fixes the problem][small-cls]. Do not combine it with unrelated refactors or cleanups. @@ -63,7 +63,7 @@ For example: ### While working When fixing a bug, verify that your test fails before and succeeds after your change. -Consult [adding new tests](./tests/adding.md) and [best practices](./tests/best-practices.md) for test procedures. +Consult [adding new tests](../tests/adding.md) and [best practices](../tests/best-practices.md) for test procedures. Tests are absolutely required; either existing tests or new tests you write. Untested LLM PRs will not be merged. @@ -96,13 +96,13 @@ Remove outdated or prototyping code and debugging. Re-read the whole diff, *not* just your conversation with the agent. Your reviewer is going to see your code, not your conversation. -[Run tests](tests/running.md) to verify your change works. +[Run tests](../tests/running.md) to verify your change works. Do NOT report which UI tests you ran in the PR description; that's noise, since CI will run them anyway. If you did manual testing or benchmarking, do report that, but note that all LLM PRs must have automated tests. -[Review diagnostic snapshots](tests/adding.md#step-4-review-the-output); +[Review diagnostic snapshots](../tests/adding.md#step-4-review-the-output); don't simply `--bless` them away. We recommend using a different model for adversarial local review before publishing your changes. @@ -148,4 +148,4 @@ We don't care which model you used, but we do care whether you used the LLM to i > 🤖 Generated with Claude Code -> Co-Authored-By: Claude Opus 4.6 (1M context) +> `Co-Authored-By: Claude Opus 4.6 (1M context) ` From 129fe3ddc832251e32ce588558b29ef129770c20 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:47:39 +0200 Subject: [PATCH 17/20] tweaks --- src/conventions.md | 17 ++++++++++++----- src/llm-guidance/writing.md | 19 ++++++++++++------- 2 files changed, 24 insertions(+), 12 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index f30921f35b..023db80481 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -141,15 +141,15 @@ if foo { If you want to leave a note in the codebase, use `// FIXME` instead. -### Follow the style of surrounding code. +### Follow the style of surrounding code Use existing helpers and avoid duplicating logic or validation. -### Represent data in [one normalized place](https://react.dev/learn/choosing-the-state-structure) +### [Avoid duplicated sources of truth](https://react.dev/learn/choosing-the-state-structure) Trying to keep data in sync between two different places is a code smell. -### Make invalid states unrepresentable +### Use types to enforce invariants When practical, [make invalid states unrepresentable](https://kentcdodds.com/blog/make-impossible-states-impossible), [not just checked at construction time](https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-type-safety/). @@ -160,11 +160,16 @@ It's ok to go into detail about non-obvious bugs. [mit-comment-style]: https://mitcommlab.mit.edu/broad/commkit/coding-and-comment-style/ -### Preserve existing behavior. +### Preserve existing behavior Consider platform differences and error cases. Look for relevant tests that exercise the edge cases. +### Work in small steps + +Work in small, independently testable steps. +Run the relevant tests after every meaningful change, so you know where you first went wrong. + ## Using crates from crates.io @@ -183,11 +188,13 @@ you rename a method, then put that rename into its own commit, along with the renames of all the uses. **More commits is usually better.** If you are doing a large change, -it's almost always better to break it up into smaller steps that can be independently understood. +it's almost always better to break it up into smaller steps that can be [independently understood][atomic commits]. The one thing to be aware of is that if you introduce some code following one strategy, then change it dramatically (versus adding to it) in a later commit, that 'back-and-forth' can be confusing. +[atomic commits]: https://github.blog/developer-skills/github/write-better-commits-build-better-projects/#%e2%9a%9b%ef%b8%8f-resize-and-stabilize-the-commits + **Format liberally.** While only the final commit of a PR must be correctly formatted, it is both easier to review and less noisy to format each commit individually using `./x fmt`. diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index 4ea43ae008..c3115e602b 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -70,21 +70,26 @@ Untested LLM PRs will not be merged. Mass renames or rewrites should *strongly* prefer using a proper syntax rewrite tool, such as [`ast-grep`]. You may use an LLM for generating the instructions for that tool, but you should be very cautious about performing the rewrite directly with an LLM. -Work in small steps. -Run tests after every meaningful change, so you know where you first went wrong. -Do not write all your code at once; that will make it very hard to know what broke. -Ideally, commit your changes in [small atomic commits] as you go. +Consider [performance] as you write. + +[performance]: ../contributing.md#performance + +Think before adding dependencies; +consult our [guidance for new dependencies][crates-io]. + +[crates-io]: ../crates-io.md Verify your understanding against the existing code, documentation, and tests. You can get better advice from your LLM by telling *it* to read the relevant materials. Do not rely on the LLM as a source of truth. -[small atomic commits]: https://github.blog/developer-skills/github/write-better-commits-build-better-projects/#%e2%9a%9b%ef%b8%8f-resize-and-stabilize-the-commits - ### Write maintainable code Treat generated code as a *draft*, not a final product. -Follow the [correctness conventions](../conventions.md#cc). +Follow our [correctness and maintainability conventions](../conventions.md#cc). + +Avoid unnecessary abstractions and compatibility layers. +Rustc does not have a stable API; you do not need to preserve backwards compatibility for internal compiler APIs. ### Before opening a PR From 7b223d00e30918928e92e794e0fcc4952dc8f2d4 Mon Sep 17 00:00:00 2001 From: jyn Date: Wed, 29 Jul 2026 12:54:54 +0200 Subject: [PATCH 18/20] headings --- src/llm-guidance/writing.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index c3115e602b..aaf37b030c 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -16,6 +16,8 @@ Keep in mind this quote: ## Rules +### Before you write code + Before anything else, find a reviewer who volunteers to review your PR. If you do not know where to find a reviewer, ask in [#llm-mentoring] on Zulip. Your first message should say: @@ -25,14 +27,21 @@ Your first message should say: - (optional) ideas you have so far for a solution - (optional) how you expect to test your solution -Disclose your use of LLMs, following the disclosure guidelines below. -Write the disclosure yourself. -You may use an LLM to privately review a disclosure you have written, but not to draft or rewrite it. +Mentors are here to help. +Talking to them early helps you avoid wasted work. + +### While working Write your own doc-comments, `// SAFETY` comments, diagnostic wording, and soundness-critical code. As before, you can use an LLM to review your work, but not to write it from scratch. If you don't know what counts as soundness-critical, discuss it with your reviewer. +### When opening a PR + +Disclose your use of LLMs, following the disclosure guidelines below. +Write the disclosure yourself. +You may use an LLM to privately review a disclosure you have written, but not to draft or rewrite it. + **Write your own PR description and comments**. We want to hear from you, not from your agent. LLM-created PR descriptions are banned. From 937ffd981c62885addcd293e0ba9daf90bd8f707 Mon Sep 17 00:00:00 2001 From: jyn Date: Thu, 30 Jul 2026 23:43:40 +0200 Subject: [PATCH 19/20] don't treat model names as a good example of disclosure --- src/llm-guidance/writing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index aaf37b030c..6719b5e786 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -153,7 +153,7 @@ We don't care which model you used, but we do care whether you used the LLM to i > LLM disclosure: I wrote the three commits by hand after viewing profiling data. I used an LLM to review the commits before submitting. The LLM identified that `ImplString::is_negative` was no longer used, so I removed that field by hand. -> Created with the help of Claude Code, which: +> Created with the help of an LLM, which: > - traced the missing cache hits to the unconditional `return(pass)` by inspecting Fastly vs CloudFront headers, > - reviewed the git history to understand why the snippet was added, and > - made the VCL change. From c7c1a5aa7a03ed12414056bf9faaadd0491a54a4 Mon Sep 17 00:00:00 2001 From: jyn Date: Sun, 2 Aug 2026 12:43:20 +0200 Subject: [PATCH 20/20] document that commit messages must be human-authored --- src/llm-guidance/writing.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/llm-guidance/writing.md b/src/llm-guidance/writing.md index 6719b5e786..6af83fd424 100644 --- a/src/llm-guidance/writing.md +++ b/src/llm-guidance/writing.md @@ -100,6 +100,11 @@ Follow our [correctness and maintainability conventions](../conventions.md#cc). Avoid unnecessary abstractions and compatibility layers. Rustc does not have a stable API; you do not need to preserve backwards compatibility for internal compiler APIs. +### Commit structure + +See ["How to structure your PR"](../contributing.md#er). +Commit messages must be authored by you, not your LLM. + ### Before opening a PR Review your own PR before opening it: