Duty rosters

benchmarks/duty_rosters/README.md

@@ -0,0 +1,89 @@
+# Nursing Staff Duty Rosters
+
+## Overview
+
+Monthly nursing staff duty rosters from a Swiss care facility. Each image shows one row from a roster, representing a single staff member's schedule for the month. Models must identify shift icons, half-day splits, alternate unit assignments, and person metadata.
+
+Images are paired with two shared context images sent once per benchmark run:
+- **roster_header.jpg** — the roster header with unit name, month/year, and date columns
+- **icons.jpg** — the icon legend mapping visual shift indicators to their codes
+
+## Data
+
+- PEP-sheet `04-25_PEP_02-12-01.pdf` manually segmented into rows yielding 21 images, ~1200x100 pixels each
+- Language: German
+- Source: Swiss nursing care facility, April 2025
+
+## Ground Truth Format
+
+```json
+{
+  "year": 2025,
+  "month": "April",
+  "persons": [
+    {
+      "id": "103",
+      "profession": "WBL",
+      "employment_percent": 80,
+      "is_jumper": false,
+      "days": [
+        {
+          "date": "2025-04-01",
+          "shifts": [
+            {
+              "icon": "F-Dienst 1",
+              "length": "full",
+              "planned_on_current_unit": true,
+              "alternate_unit": "LE"
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
+
+Key fields:
+- **icon**: Descriptive name from the icon legend (e.g. "F-Dienst 1", "Ferien", "Freier Tag")
+- **length**: `full`, `half_left`, or `half_right` (cells can be split into two half-day shifts)
+- **planned_on_current_unit**: `true` if the icon has no red shading, `false` if the icon has red shading
+- **alternate_unit**: Unit abbreviation shown below the cell (e.g. "LE", "TS") if present, null otherwise. Multiple abbreviations for the same date are joined with a comma (e.g. "LE, TS").
+
+## Editor
+
+`editor/editor.html` is a browser-based correction tool for ground truth JSON files. Open it directly in Chrome/Edge/Opera (File System Access API required for in-place saves; other browsers fall back to downloads).
+
+Workflow:
+1. Pick the `images/` folder and the `ground_truths/` folder. Files are paired by basename.
+2. Optionally pick an output folder (defaults to the browser's download directory).
+3. Click a row in the sidebar to load the image + JSON. Edit person metadata, days, and shifts.
+4. Click "JSON Speichern" to write the file.
+
+Behavior worth knowing:
+- Same-date day entries are auto-merged into a single day with a combined `shifts` array on load and before save.
+- Edits are cached in memory per file, so switching files and back preserves unsaved changes.
+- "+ Tag anfügen" appends a day to the end; the calendar-plus button on each day inserts a day directly below.
+- Shifts can be split into `half_left` / `half_right` halves on the same date.
+
+## Scoring
+
+**F1 Micro** with field-level fuzzy matching (threshold: 0.92, rapidfuzz).
+
+All leaf fields are scored (year, month, person metadata, and per-day shift fields). TP/FP/FN are calculated per field across the full nested structure. F1 Macro (per-image F1 averaged) is also reported.
+
+## Ideas for Improvement
+
+1. **Composite header + row into a single image.** The header and roster row are currently sent as separate images, forcing models to align columns across images. Stitching the header directly above each row would make date-to-cell mapping visual and trivial — this is the single largest error source (~50% of dates are wrong).
+
+2. **Use day-of-month integers instead of ISO dates.** Converting cell position to a day number and then to an ISO date string is two steps of error. Since year and month are already captured at the Schedule level, `date: 17` would be simpler than `"2025-04-17"`.
+
+3. **Separate icon recognition from structure extraction in scoring.** Report a structure score (dates, length, planned_on_current_unit, alternate_unit, person metadata) and an icon score separately. This gives more diagnostic value than a single F1 number.
+
+4. **Crop individual cells for icon classification.** As a benchmark variant: pre-crop each day cell and pair it with the legend, asking "which icon is this?" This isolates icon recognition from layout understanding and reveals where the real bottleneck is.
+
+5. **Add a visual example of an empty cell to the legend.** Models confuse null (no icon) with "Freier Tag" or hallucinate icons like "HomeOffice" for empty cells. Showing what "no icon" looks like in the legend could help.
+
+6. **Score fields with different weights.** Currently all leaf fields count equally, so ~128 shift-level entries dominate 4 person-level fields. Consider per-day accuracy or separate person-metadata vs shift-level reporting.
+
+7. **Reduce the active icon vocabulary.** The legend has 33 icons but the ground truth only uses ~8. Noting the active subset in the prompt or evaluating icon accuracy within the active set would be informative.

benchmarks/duty_rosters/benchmark.py

@@ -0,0 +1,137 @@
+"""Benchmark implementation for Nursing staff duty rosters.
+
+Parse visual nursing staff duty rosters
+"""
+
+import logging
+import os
+from typing import Dict, List
+
+from scripts.benchmark_base import Benchmark
+from scripts.scoring_helper import get_all_keys, get_nested_value, calculate_fuzzy_score
+
+
+class DutyRosters(Benchmark):
+    """Benchmark for Nursing staff duty rosters."""
+
+    # Send header + icons with every request (works across all providers)
+    cache_context_per_request = True
+
+    def get_shared_context_images(self) -> List[str]:
+        """Return header and icon legend images to prepend to every request."""
+        return [
+            os.path.join(self.benchmark_dir, 'context', 'roster_header.jpg'),
+            os.path.join(self.benchmark_dir, 'context', 'icons.jpg'),
+        ]
+
+    def score_request_answer(self, image_name: str, response: dict, ground_truth: dict) -> dict:
+        data = self.prepare_scoring_data(response)
+
+        logging.info(image_name)
+        logging.debug(f"response: {data}")
+        logging.debug(f"ground_truth: {ground_truth}")
+
+        response_keys = get_all_keys(data)
+        gt_keys = get_all_keys(ground_truth)
+
+        all_keys = set(response_keys + gt_keys)
+
+        # Filter out parent keys when child keys exist
+        filtered_keys = []
+        for key in all_keys:
+            has_children = any(
+                other_key.startswith(key + '.') or other_key.startswith(key + '[')
+                for other_key in all_keys if other_key != key
+            )
+            if not has_children:
+                filtered_keys.append(key)
+
+        tp = 0
+        fp = 0
+        fn = 0
+        field_scores = {}
+        match_threshold = 0.92
+
+        for key in filtered_keys:
+            response_value = get_nested_value(data, key)
+            gt_value = get_nested_value(ground_truth, key)
+
+            if response_value == "":
+                response_value = None
+            if gt_value == "":
+                gt_value = None
+
+            field_score = calculate_fuzzy_score(response_value, gt_value)
+            field_scores[key] = {
+                'response': response_value,
+                'ground_truth': gt_value,
+                'score': field_score
+            }
+
+            if response_value is not None and gt_value is not None:
+                if field_score >= match_threshold:
+                    tp += 1
+                else:
+                    fp += 1
+                    fn += 1
+            elif response_value is not None and gt_value is None:
+                fp += 1
+            elif response_value is None and gt_value is not None:
+                fn += 1
+
+        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
+        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
+        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
+
+        score = {
+            'f1_score': round(f1, 2),
+            'precision': precision,
+            'recall': recall,
+            'true_positives': tp,
+            'false_positives': fp,
+            'false_negatives': fn,
+            'field_scores': field_scores,
+            'total_fields': len(filtered_keys)
+        }
+
+        logging.info(score['f1_score'])
+        for line in score['field_scores'].values():
+            logging.debug(line)
+
+        return score
+
+    def score_benchmark(self, all_scores: list) -> dict:
+        if not all_scores:
+            return {"f1_micro": 0.0, "f1_macro": 0.0}
+
+        total_tp = 0
+        total_fp = 0
+        total_fn = 0
+        f1_scores = []
+
+        for score_data in all_scores:
+            if isinstance(score_data, dict) and 'f1_score' in score_data:
+                total_tp += score_data.get('true_positives', 0)
+                total_fp += score_data.get('false_positives', 0)
+                total_fn += score_data.get('false_negatives', 0)
+                f1_scores.append(score_data['f1_score'])
+
+        micro_precision = total_tp / (total_tp + total_fp) if (total_tp + total_fp) > 0 else 0.0
+        micro_recall = total_tp / (total_tp + total_fn) if (total_tp + total_fn) > 0 else 0.0
+        f1_micro = 2 * micro_precision * micro_recall / (micro_precision + micro_recall) if (micro_precision + micro_recall) > 0 else 0.0
+        f1_macro = sum(f1_scores) / len(f1_scores) if f1_scores else 0.0
+
+        score = {
+            "f1_micro": f1_micro,
+            "f1_macro": f1_macro,
+            "micro_precision": micro_precision,
+            "micro_recall": micro_recall,
+            "total_instances": len(f1_scores),
+            "total_tp": total_tp,
+            "total_fp": total_fp,
+            "total_fn": total_fn
+        }
+
+        logging.info(f"f1_micro: {f1_micro}, f1_macro: {f1_macro}")
+
+        return score

benchmarks/duty_rosters/context/shared_context_prompt.txt

@@ -0,0 +1,35 @@
+This is the icon legend for a monthly nursing staff duty roster. Each icon represents a shift type or absence category. Use the descriptive name (not the number) when identifying icons in the roster rows that follow.
+
+Valid icon names:
+- F-Dienst 1 (early shift, code 300)
+- S-Dienst 1 (late shift, code 340)
+- M-Dienst 3 (code 322)
+- M-Dienst 4 (code 323)
+- N 1 (night shift, code 817)
+- SD (code 919)
+- L (code 932)
+- Bürotag (office day, code 232)
+- HomeOffice (code 200)
+- BESPRECHUNG (meeting, code 219)
+- Schülerbetreuung (code 922)
+- TV-FAL (code 905)
+- Azu-Gespräche dives (code 8897)
+- Ferien (vacation, code 202)
+- Freier Tag (free day, code 224)
+- Wunschfrei (requested free day, code 222)
+- Kranktag (sick day, code 203)
+- kIND kRANK (child sick, code 204)
+- Unfall (accident, code 206)
+- Mutterschutz (maternity protection, code 214)
+- Schwangerschaftsurlaub (maternity leave, Su)
+- Zügeltag (moving day, code 211)
+- Schultag (school day, code 216)
+- Schule (school, S)
+- Obl. Fort-Bildung (mandatory training, code 238)
+- Kurse bezahlte Weiterb. (paid courses, code 207)
+- RAI/RUG (code 228)
+- Persönliche Absenz (personal absence, PA)
+- Unbezahlter Urlaub (unpaid leave, uU)
+- Betriebsausflug (company outing)
+- Interne Einblicke (internal insights)
+- Auszubildende Lerntandem (trainee tandem)

benchmarks/duty_rosters/dataclass.py

@@ -0,0 +1,88 @@
+"""Pydantic models for Nursing staff duty rosters benchmark output validation.
+
+This module defines the expected structure of model outputs.
+"""
+
+from enum import Enum
+from typing import Literal, Optional
+
+from pydantic import BaseModel
+
+ICON_MAP = {
+    "?": "Unbekannt",
+    "200": "HomeOffice",
+    "202": "Ferien",
+    "203": "Kranktag",
+    "204": "kIND kRANK",
+    "206": "Unfall",
+    "207": "Kurse bezahlte Weiterb.",
+    "211": "Zügeltag",
+    "214": "Mutterschutz",
+    "216": "Schultag",
+    "219": "BESPRECHUNG",
+    "222": "Wunschfrei",
+    "224": "Freier Tag",
+    "228": "RAI/RUG",
+    "232": "Bürotag",
+    "238": "Obl. Fort-Bildung",
+    "300": "F-Dienst 1",
+    "322": "M-Dienst 3",
+    "323": "M-Dienst 4",
+    "340": "S-Dienst 1",
+    "817": "N 1",
+    "8897": "Azu-Gespräche dives",
+    "905": "TV-FAL",
+    "919": "SD",
+    "922": "Schülerbetreuung",
+    "932": "L",
+    "Auszubildende Lerntandem": "Auszubildende Lerntandem",
+    "Betriebsausflug": "Betriebsausflug",
+    "Interne Einblicke": "Interne Einblicke",
+    "PA": "Persönliche Absenz",
+    "S": "Schule",
+    "Su": "Schwangerschaftsurlaub",
+    "uU": "Unbezahlter Urlaub",
+}
+
+IconCode = Literal[
+    "Auszubildende Lerntandem", "Azu-Gespräche dives", "BESPRECHUNG",
+    "Betriebsausflug", "Bürotag", "F-Dienst 1", "Ferien", "Freier Tag",
+    "HomeOffice", "Interne Einblicke", "kIND kRANK", "Kranktag",
+    "Kurse bezahlte Weiterb.", "L", "M-Dienst 3", "M-Dienst 4",
+    "Mutterschutz", "N 1", "Obl. Fort-Bildung", "Persönliche Absenz",
+    "RAI/RUG", "S-Dienst 1", "Schule", "Schultag", "Schülerbetreuung",
+    "Schwangerschaftsurlaub", "SD", "TV-FAL", "Unbezahlter Urlaub",
+    "Unbekannt", "Unfall", "Wunschfrei", "Zügeltag",
+]
+
+
+class ShiftLength(str, Enum):
+    FULL = "full"
+    HALF_LEFT = "half_left"
+    HALF_RIGHT = "half_right"
+
+
+class ShiftEntry(BaseModel):
+    icon: Optional[IconCode] = None
+    length: ShiftLength
+    planned_on_current_unit: bool
+    alternate_unit: Optional[str] = None
+
+
+class DayEntry(BaseModel):
+    date: str
+    shifts: list[ShiftEntry]
+
+
+class Person(BaseModel):
+    id: str
+    profession: str
+    employment_percent: int
+    is_jumper: bool
+    days: list[DayEntry]
+
+
+class Schedule(BaseModel):
+    year: int
+    month: str
+    persons: list[Person]