Skip to content

Record per-rule match evidence (matched facts and slot values) in EvaluationResult and AuditRecord #144

Description

@se-jo-ma

Context

Fathom's marketing and audit story promise the ability to "prove exactly which rule fired, on which facts" (elevator-pitch.md:17). The recorded data does not deliver this. The audit record knows which rules fired and what facts the caller said it asserted, but nothing binds a fired rule to the specific fact instances and slot values that satisfied its conditions.

Confirmed in code:

  • EvaluationResult.rule_trace is a list[str] of module::rule names only (src/fathom/models.py:346).
  • AuditRecord stores rules_fired (names) and input_facts (src/fathom/models.py:353-365). Per the audit doc, input_facts is caller-supplied and not auto-snapshotted from working memory (docs/concepts/audit-attestation.md:79-81), and the attestation only embeds a hash of inputs, not the facts (audit-attestation.md:223-224).
  • The trace is reconstructed by reading __fathom_decision facts after env.run() reaches quiescence (src/fathom/evaluator.py:89-113) — at which point CLIPS activations are already consumed.

An auditor faced with N asserted data_request facts and one deny cannot tell which fact (or which slot values) triggered it.

Design caveat (read before implementing)

Verified against clipspy 1.0.6 in this repo's environment: the Activation object exposes only delete, name, salience; Rule.matches() returns only counts (needed, ...), not the matched fact instances. There is no clipspy accessor for an activation's basis facts, so the issue's suggestion of "reading the activation's basis facts before firing" is not directly available. The realistic path is to have the compiler echo the matched bound variables / fact references into the RHS so they land in a diagnostic structure — which means src/fathom/compiler.py is in scope even though it was not in the original file list. The design owner must decide how to capture slot-level evidence deterministically without breaking the <100µs single-rule target (README.md:178).

Task

  • Add an optional structured field to EvaluationResult (e.g. match_evidence: list[{rule, matched_facts: [{template, slots}], bound_vars}]).
  • Capture evidence during evaluation. Since CLIPS activation basis facts are not exposed by clipspy, capture matched fact data via compiler-emitted bindings on the RHS (alongside the existing __fathom_decision assertion in compiler.py).
  • Thread the field into AuditRecord so the chained log preserves it.
  • Make it opt-in (off by default) to avoid latency regression.
  • Document the new field in docs/concepts/audit-attestation.md.

Where

  • src/fathom/models.py:341 (EvaluationResult) and :353 (AuditRecord).
  • src/fathom/evaluator.py:89-113 (_capture_trace) and src/fathom/engine.py:1148 (evaluate).
  • src/fathom/compiler.py:275-330 (RHS / __fathom_decision emission — needed for fact/slot capture).
  • docs/concepts/audit-attestation.md (document the evidence field; update the audit-log shape section).
  • tests/test_engine.py.

Acceptance criteria

  • With evidence enabled, a deny result records the specific fact(s) and slot values that matched the firing rule.
  • Default (disabled) path shows no measurable latency regression in fathom bench against the <100µs single-rule target.
  • A test covers a multi-fact scenario where only one of several asserted facts triggers the rule, and asserts the evidence names exactly that fact + its slot values.
  • match_evidence round-trips through AuditRecord JSON serialization.
  • docs/concepts/audit-attestation.md documents the new field and its opt-in behaviour.

Size: L

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority: roadmap gap or DX/toolingenhancementNew feature or requestneeds-designDesign not yet settled — please don't open a PR yetsize/L<1 week: cross-cutting feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions