Skip to content

feat(benchmark): add lme_one_question job to jinli_lme.yaml#333

Open
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:feat/lme-job-example
Open

feat(benchmark): add lme_one_question job to jinli_lme.yaml#333
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:feat/lme-job-example

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

Summary

  • Appends a single new job lme_one_question to reme/config/jinli_lme.yaml.
  • The job wires the two LME steps added in feat(benchmark): add lme benchmark steps #326 (context_answer_step, answer_judge_step) so the LME harness can POST one HTTP request per question and get both the direct-context answer and the LLM-judge verdict back in one round trip.
  • All six LME-relevant fields (query, session_context, current_date, agent_answer, golden_answer, question_type) are declared as required parameters with per-field descriptions, matching the input contract of the two steps.
  • No runtime code, prompts, or default.yaml are touched.

Why a separate, additive PR

The shipped jinli_lme.yaml only registered update_index / auto_memory / version / search, leaving benchmark authors to either re-derive the wiring or write a duplicate config. Adding a dedicated job matches #326's intent, is one self-contained stanza, and is documented in the new docs/en|lme_benchmark.md/zh pages (PR #332) as the canonical harness entry point.

The job is opt-in: existing jobs in jinli_lme.yaml keep their behavior, and reme start config=jinli_lme.yaml does not auto-register any new dispatch. Only an explicit reme app lme_one_question ... invocation reaches the new path.

Verification

  • python -c 'import yaml; yaml.safe_load(open("reme/config/jinli_lme.yaml"))' → loads with jobs.lme_one_question present, steps [context_answer_step, answer_judge_step], all six parameters required.
  • pre-commit run --files reme/config/jinli_lme.yaml → all hooks green (yaml check + pyroma only; Python-side hooks skipped as expected for a YAML-only diff).

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

Pushed a small follow-up commit aa0538a:

  • tests/unit/test_config_parser.py — adds test_jinli_lme_config_registers_lme_one_question_job that loads jinli_lme.yaml via the production _load_config and asserts:
    • backend == "base"
    • step order is exactly [context_answer_step, answer_judge_step]
    • all six LME fields appear in parameters.required
    • the property bag matches required (catches typos where someone adds a parameter but forgets to mark it required).
  • pytest tests/unit/test_config_parser.py → 10 passed locally.
  • pre-commit run --files tests/unit/test_config_parser.py reme/config/jinli_lme.yaml → all hooks green.

So if the future of context_answer_step / answer_judge_step moves its inputs elsewhere, this test catches it before review.

@RerankerGuo RerankerGuo force-pushed the feat/lme-job-example branch from aa0538a to ea7c18d Compare July 9, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant