fix(pbl): explain the tool-calling requirement when generation is rejected before the first step#722
Open
ly-wang19 wants to merge 1 commit into
Open
Conversation
PBL is the only generation path that drives the model through native tool/function calls, and providers whose chat template lacks tools support reject such requests outright — surfacing as a raw provider error (e.g. an instant 500 from LM Studio with Gemma 4 in THU-MAIC#664) with no hint that the model, not the course, is the problem. Track agentic-loop progress and, when the request fails before the first step completes — the signature of missing tool support — wrap the error with an actionable message naming the requirement and the way out (pick a tool-capable model or change the scene type). Failures after steps have completed rethrow unchanged so mid-loop network or provider errors are not misdiagnosed. Regression tests cover both branches. Refs THU-MAIC#664.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #664 (first, diagnostic half — see "Scope" below for why it doesn't close it).
Problem
PBL is the only generation path that requires native tool calling (
generateText+ Zodtools+stopWhen, the agentic MCP loop inlib/pbl/generate-pbl.ts). Models whose chat template lacks tools support reject such a request outright — #664 reports LM Studio + Gemma 4 returning an instant 500 (while Gemma 3, whose LM Studio template has tools support, works). The raw provider error propagates to the failed-scene UI with no hint that the model capability, not the course or the app, is the problem — and retrying can never succeed.Fix
Track how many agentic steps completed. When the call fails before the first step — the signature of a tools-incapable model rejecting the request — wrap the error with an actionable message:
Failures after steps have completed rethrow unchanged, so mid-loop network/provider errors are not misdiagnosed.
Same shape as the merged #644 (surface TTS provider rate-limits as 429s): convert a cryptic provider failure into an actionable one at the layer that knows the cause.
Scope (why
Refsand notCloses)This makes the failure clear and actionable; it does not make tools-incapable models able to generate PBL scenes. That needs a design decision on #664 — e.g. a single-shot JSON text-mode fallback (reusing the existing json-repair machinery) or a
toolscapability flag that gatespbloutlines per model. Happy to implement either if maintainers pick a direction; the agentic MCP loop is clearly a deliberate design so I didn't want to parallel it unilaterally.Verification
tests/pbl/generate-pbl-tool-rejection.test.ts, 2 cases): zero-step rejection gets the wrapped message incl. the original error; post-step failure keeps its original error.prettierclean,eslintclean,tsc --noEmiterror count unchanged, adjacent PBL/generation suites green (11/11).