Skip to content

[Research]: Benchmark which :free OpenRouter models reliably support tools #68

Description

@shauryagangrade

Problem

README warns some free models return "No endpoints found that support tool
use", and free-tier rate limits vary wildly — but there's no data on which
models actually work with GCode's tool schema and stream reliably.

Proposed approach

  • Run a scripted benchmark: for each :free model on OpenRouter, send a
    fixed prompt that requires one tool call (e.g. read_file), record
    success/failure, tool-support, latency, and 429 rate-limit behavior.
  • Publish results as a table in the README/models docs and propose a curated
    default list plus a warning badge in /models for models known to lack tools.

Where to look

  • gcode/models.py (supports_tools derivation) — cross-check vs reality.
  • README.md:78-85 (Models section).

Acceptance criteria

  • A reproducible script + published results table.
  • README recommends specific models; /models can flag untested/failing ones.

Difficulty

Medium — great non-Python-heavy contribution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationhelp wantedExtra attention is neededquestionFurther information is requested

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions