Skip to content

fix(prompts): disambiguate skills from tools to prevent direct-call failures#330

Open
sridhar-3009 wants to merge 1 commit into
HKUDS:mainfrom
sridhar-3009:fix/292-skills-prompt-disambiguation
Open

fix(prompts): disambiguate skills from tools to prevent direct-call failures#330
sridhar-3009 wants to merge 1 commit into
HKUDS:mainfrom
sridhar-3009:fix/292-skills-prompt-disambiguation

Conversation

@sridhar-3009

Copy link
Copy Markdown

Summary

Closes #292

Weaker models (Qwen72B and similar) read the skills listing in the system prompt and call the skill name directly as a tool — e.g. tavily-search(query="...") — instead of the correct skill(name="tavily-search"). The call fails with "unknown tool: tavily_search" because skills are not registered as tools; they are accessed exclusively through the skill built-in tool.

Root cause (confirmed in issue comment): The previous instruction said:

"invoke it with skill(name="<skill_name>")"

The abstract placeholder <skill_name> is easy for a model to treat as a formatting hint rather than a literal pattern. Combined with the listing format - **tavily-search**: Search the web..., the model mistakes the skill name for a registered tool name.

Fix: Three changes to _build_skills_section():

  1. Lead with an explicit prohibitionIMPORTANT: Skills are NOT tools. Do NOT call them as direct function calls.
  2. Show a concrete invocation example using the first real skill name instead of <skill_name>, so the model sees skill(name="tavily-search") verbatim.
  3. Explain the failure mode — "Calling a skill name directly as a tool will fail with 'unknown tool'" — giving the model a clear signal to remember.

Before:

# Available Skills

The following skills are available via the `skill` tool. When a user's request matches a skill, invoke it with skill(name="<skill_name>") to load detailed instructions…

- **tavily-search**: Search the web with Tavily

After:

# Available Skills

IMPORTANT: Skills are NOT tools. Do NOT call them as direct function calls.
Skills must be accessed exclusively through the `skill` tool:
  skill(name="tavily-search")

When a user's request matches a skill below, call `skill(name="<skill_name>")` to load the full instructions…
Calling a skill name directly as a tool will fail with 'unknown tool'.

Available skills (access each via `skill(name="<name>")`):

- **tavily-search**: Search the web with Tavily

Test plan

  • With a weaker model (Qwen72B or similar), install a tavily-search skill and ask a question that matches it — verify the model calls skill(name="tavily-search") instead of tavily-search(query="...").
  • With a stronger model (claude-sonnet, gpt-4o), verify existing skill invocation still works correctly.
  • Verify that when no skills are installed, the section is still omitted from the system prompt.

…ailures

Weaker models (Qwen72B and others) misread the skills listing in the
system prompt and call e.g. tavily-search(query="...") directly instead
of skill(name="tavily-search"). The call fails with "unknown tool"
because skills are not registered as tools — they are accessed
exclusively through the skill tool.

The previous wording was too subtle:
  "invoke it with skill(name="<skill_name>")"

Changes:
- Lead with "IMPORTANT: Skills are NOT tools. Do NOT call them as direct
  function calls."
- Show a concrete invocation example using the first real skill name
  (e.g. skill(name="tavily-search")) instead of the abstract placeholder
  "<skill_name>" that models tend to ignore.
- Explicitly state that calling a skill name directly as a tool will
  fail with "unknown tool", giving the model a clear failure signal to
  avoid.
- Relabel the skill list with the reminder "(access each via
  skill(name="<name>"))" so the correct pattern stays visible next to
  every entry.

Fixes HKUDS#292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: skills没法执行

1 participant