ContextPrecision does not meet the specification

[fizcogar]

Hi!

From SCORERS.md:

"ContextPrecision measures the precision of retrieved context - whether relevant context appears before irrelevant context ...
Score Range: 0-1,
1.0 = All relevant context appears first
0.0 = Relevant context is buried under irrelevant context"

from autoevals.ragas import ContextPrecision

scorer = ContextPrecision(model = "Qwen3-30B-A3B-Instruct")
scorer.eval(
    input = "Where is the Eiffel Tower located?",
    expected = "The Eiffel Tower is located in Paris.",
    output = "",
    context =[
        "The Brandenburg Gate is located in Berlin.",
        "The Eiffel Tower is located in Paris."
    ]
)

Score(name='ContextPrecision', score=1, metadata={'precision': {'reason': 'The context explicitly states "The Eiffel Tower is located in Paris," which directly supports the answer to the question about its location.', 'verdict': 1}}, error=None) 

The score cannot be 1, the relevant context is the second one on the list.

The same example, implemented with Ragas:

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision

llm = llm_factory("Qwen3-30B-A3B-Instruct", client=async_client)
scorer = ContextPrecision(llm=llm)
await scorer.ascore(
    user_input="Where is the Eiffel Tower located?",
    reference="The Eiffel Tower is located in Paris.",
    retrieved_contexts=[        
        "The Brandenburg Gate is located in Berlin.",
        "The Eiffel Tower is located in Paris."
    ]
)

MetricResult(value=0.49999999995)

Thanks!

[vaibhavhariram]

Confirmed reproduction. Root cause: context list gets joined into a single string before the LLM sees it, destroying position information; single verdict returned directly with no aggregation. Before fixing, we need to decide between two valid aggregation formulas—MAP (RAGAS spec) scores [irrelevant, relevant] as 0.5, pairwise ordering as 0.0. Which do you want, exact RAGAS parity or position-aware semantics? I'll implement whichever you prefer.

[Ruthwik-Data]

Thanks for the detailed root cause analysis vaibhavhariram!

On the aggregation formula decision — I'd lean toward Mean Average Precision (MAP) for the following reasons:

1. Aligns with the spec semantics: The SCORERS.md description says "relevant context appears before irrelevant context" — this is inherently a ranking/ordering concept, and MAP is the standard metric for evaluating ranked retrieval quality. It rewards systems that surface relevant chunks earlier, not just whether they exist.

2. Pairwise ordering (score = 0.0) is too harsh: In the example from the issue ([irrelevant, relevant]), giving a score of 0.0 implies the retrieval is completely wrong — but the relevant chunk is present, just not first. A MAP score of 0.5 better reflects partial credit for a nearly-correct ordering.

3. Consistency with Ragas: Since autoevals already references RAGAS-style scorers, matching Ragas' output (0.5 for this case) makes it easier for users migrating or comparing between the two frameworks.

Suggested fix direction: Pass context chunks individually to the LLM (one verdict per chunk), then compute MAP over the binary relevance judgments ordered by position — which is exactly what vaibhavhariram's PR #178 appears to be doing.

Happy to help test this once the fix lands!