ClaimTrace is a schema-safe multimodal claim verification pipeline that combines deterministic rule checks, structured VLM reasoning, selective audit, checkpoint-safe execution, and an Evidence Passport for every claim.
Originally built for the HackerRank Orchestrate (June 2026) hackathon.
git clone https://github.com/devdiv07/ClaimTrace && cd ClaimTrace
python -m venv .venv
.\.venv\Scripts\activate # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
python code/main.py --input dataset/sample/claims.csv --output dataset/sample/output.csv --mockThis runs the full deterministic pipeline — image loading, evidence-standard checks, user-history risk flags, schema validation — against a small synthetic dataset checked into this repo. No API key and no proprietary data required. --mock skips the vision-model call itself; see "About the Dataset" below for what that does and doesn't cover.
This public repo ships the full pipeline, prompts, evaluation code, and result artifacts (output.csv, checkpoints, reports), but not the underlying dataset/claims.csv, dataset/sample_claims.csv, dataset/images/, or problem_statement.md — those are the hackathon organizer's proprietary contest material and aren't mine to redistribute.
To run the pipeline against the real contest data, supply your own files with the same layout referenced below (claims.csv, sample_claims.csv, user_history.csv, evidence_requirements.csv, images/) and matching column schema.
dataset/sample/ is included — a 6-row synthetic set (fake users, fake claims, generated placeholder images) covering all three claim objects (car, laptop, package), one deliberately-missing image, and one intact-object case, so the quickstart above has real data to run against without touching proprietary content or an API key. A hand-labeled version of this same set lives at evals/golden_set.sample.csv (see evals/README.md).
Damage claims arrive with a mix of evidence: a chat transcript, one or more submitted images, the user's claim history, and a product-specific minimum-image-evidence requirement. The task is to assess each claim and produce a strict output.csv with exactly 14 columns in a fixed order.
The challenge is twofold: multimodal reasoning over images that may be blurry, mismatched, or insufficient, and exact schema compliance so a downstream evaluator can parse results reliably.
ClaimTrace adds a structured layer around the vision model:
- Exact 14-column schema validation —
validate_output_csvis called before and after every write; invalid enum values or extra columns cause a hard exit - Deterministic preprocessing — image resolution, history flags, and evidence-standard checks run identically on every execution
- User history risk flags —
user_history_riskandmanual_review_requiredare derived fromuser_history.csv; they signal risk but never override the visual verdict - Evidence requirement handling —
evidence_requirements.csvis matched per claim object and issue family to set the pre-VLM evidence bar - Structured VLM output — the VLM returns a Pydantic model; freeform text answers are not accepted
- Checkpoint / resume — every row is saved atomically; a crashed run resumes with zero duplicate API calls
- Selective audit — risky or uncertain rows get a skeptical second-pass VLM review; a balanced resolution policy merges both passes
- Evidence Passport — a judge-readable per-claim report generated from existing artifacts with no additional API calls
- Sample evaluation report — accuracy, per-class F1, and flag recall are computed against ground-truth sample labels
dataset/claims.csv + images
dataset/user_history.csv
dataset/evidence_requirements.csv
|
v
deterministic preprocessing
|
v
structured pass-1 VLM verification
|
v
selective audit for risky rows
|
v
balanced resolution policy
|
v
output.csv + evidence passport + evaluation report
.
+-- LICENSE
+-- README.md
+-- requirements.txt
+-- output.csv final 44-row, 14-column output
+-- dataset/
| +-- sample/ INCLUDED -- 5-row synthetic set (see "About the Dataset")
| | +-- claims.csv
| | +-- user_history.csv
| | +-- evidence_requirements.csv
| | +-- images/
| +-- claims.csv NOT INCLUDED -- 44-row proprietary test input
| +-- sample_claims.csv NOT INCLUDED -- 20-row proprietary sample with ground truth
| +-- user_history.csv NOT INCLUDED -- proprietary
| +-- evidence_requirements.csv NOT INCLUDED -- proprietary
| +-- images/ NOT INCLUDED -- proprietary
| +-- sample/
| +-- test/
+-- code/
+-- main.py CLI entry point (all modes)
+-- schema.py enum constants and column list
+-- utils.py loaders, image preprocessing, rule engine, validation
+-- vlm.py VLM integration (pass-1 and audit prompts)
+-- passport.py Evidence Passport generator (no API calls)
+-- final_validate.py pre-submission validation script
+-- evaluation/
| +-- main.py evaluation metrics
| +-- evaluation_report.md full evaluation report
| +-- level2c_sample_metrics.md
| +-- level3_audit_metrics.md
+-- reports/
+-- checkpoint_claims.json pass-1 checkpoint (44 entries)
+-- checkpoint_audit.json audit checkpoint (27 entries)
+-- audit_log.json per-row audit decisions
+-- evidence_passport.md per-claim glass-box report
+-- evidence_passport.json
python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txtOPENAI_API_KEY is required only for real VLM runs (--real-all, --audit-all, --real-sample).
Create a .env file in the repo root:
OPENAI_API_KEY=sk-...
Never commit .env to git.
Mock run (no API, schema check only):
.\.venv\Scripts\python.exe code/main.py --input dataset/claims.csv --output output.csv --mockFull real run (44 rows, checkpoint-safe):
.\.venv\Scripts\python.exe code/main.py --input dataset/claims.csv --output output.csv --real-allAudit run (selective second pass on risky rows):
.\.venv\Scripts\python.exe code/main.py --input dataset/claims.csv --output output.csv --audit-allEvidence Passport (no API calls):
.\.venv\Scripts\python.exe code/main.py --input dataset/claims.csv --output output.csv --passportSample evaluation (compare predictions to ground truth):
.\.venv\Scripts\python.exe code/evaluation/main.py --pred sample_vlm_output.csv --gt dataset/sample_claims.csvFinal validation (pre-submission check):
.\.venv\Scripts\python.exe code/final_validate.pyExact 14 columns in this order:
| # | Column | Type |
|---|---|---|
| 1 | user_id |
string (echo) |
| 2 | image_paths |
string (echo) |
| 3 | user_claim |
string (echo) |
| 4 | claim_object |
string (echo) |
| 5 | evidence_standard_met |
bool |
| 6 | evidence_standard_met_reason |
string |
| 7 | risk_flags |
semicolon-separated enum |
| 8 | issue_type |
enum |
| 9 | object_part |
enum per claim_object |
| 10 | claim_status |
supported / contradicted / not_enough_information |
| 11 | claim_status_justification |
string |
| 12 | supporting_image_ids |
semicolon-separated or "none" |
| 13 | valid_image |
bool |
| 14 | severity |
none / low / medium / high / unknown |
No additional columns are permitted. Internal fields (confidence, visual_summary, visual_risk_flags) exist only in checkpoints and are never written to output.csv.
code/reports/checkpoint_claims.json— each pass-1 row is saved atomically (via.tmprename) after its VLM call. Re-running--real-allloads all cached rows and makes 0 new API calls.code/reports/checkpoint_audit.json— the same pattern for audit rows. Re-running--audit-allwith a full cache makes 0 new audit calls.- Both resume modes were tested: re-run loaded all 44 / 27 entries from cache with identical output and validation passed.
code/reports/evidence_passport.md is a judge-readable glass-box report generated by --passport. It contains one section per claim with decision, audit trail, and a rule-based review note. It is completely separate from output.csv and does not alter schema, row count, or any column value.
Evaluated on 20-row sample_claims.csv with ground truth:
| Field | Accuracy |
|---|---|
| claim_status | 60.0% |
| issue_type | 50.0% |
| object_part | 85.0% |
| severity | 45.0% |
| evidence_standard_met | 60.0% |
Per-class claim_status F1: supported=0.783, contradicted=0.333, not_enough_information=0.364
Flag recall: manual_review_required=75%, claim_mismatch=50%, wrong_object=50%, blurry_image=100%
Exact risk_flags match rate: 45.0%
Production run (44 rows):
- Pass-1: 44 calls, 134,420 input / 6,160 output tokens, $0.3977, 4.98s avg latency
- Audit: 27 calls, 66,243 input / 3,969 output tokens, $0.2053
- Grand total: 200,663 input / 10,129 output tokens, $0.6029
A self-consistency ensemble was tested on the sample set but rejected because it did not improve claim_status accuracy or contradicted recall (see code/evaluation/evaluation_report.md, "Rejected Ensemble Experiment").
- User history raises risk flags but never decides
claim_status. The verdict is driven by visual evidence. - Uncertain cases use
not_enough_informationrather than defaulting to denial. manual_review_requiredis added for ambiguity, mismatches, and elevated risk — all such rows are flagged for human review.- No hidden chain-of-thought appears in
output.csv. Reasoning is captured in checkpoints and the Evidence Passport only.
- Sample claim_status accuracy is moderate (60%). The pipeline performs better on clear supported/contradicted cases and weaker on borderline not_enough_information rows.
- Severity classification is subjective; the VLM's interpretation of "medium" vs "high" damage may not match human labelers consistently.
- Contradiction detection is constrained by the VLM's tendency to prefer not_enough_information on ambiguous evidence.
- VLM latency and cost vary by provider and model. The 4.98s per-row average assumes gpt-4o; a different model or provider will change this.
- Real-world deployment would need more labeled calibration data and a structured human feedback loop.
- Stronger contradiction-specific calibration with more labeled contradiction examples
- Local or on-premises VLM for privacy-sensitive claim data
- Better severity rubric with explicit damage-area and depth criteria
- Image hashing and manipulation checks before VLM submission
- Human feedback loop to re-calibrate the prompt after each batch