180k pages already identified as foldouts need to be classified along two independent axes: content type (~10 classes, e.g. map, anatomical illustration, table, diagram) and language (~5 classes), each single-label/multi-class.
Plan: stratified sampling instead of plain random sampling, since F1 is computed per class and random sampling would leave rare classes with too few examples for a stable score. Pull ~80–100 foldouts per content class from the existing (imperfect) pre-classification, distributed across language classes where possible (~5–10 per language per content class). Total sample: ~800–1000 foldouts.
This sample gets manually annotated to form the gold standard, then run through candidate models to compare per-class F1, macro-F1 (primary metric), micro-F1/accuracy, and cost per 1000 classifications — to find the Pareto-optimal model(s) on F1 vs. cost.
Risk: if the pre-classification is too noisy to locate rare classes, may need a larger unguided sample first (~1500–2000) to surface them before stratifying.
180k pages already identified as foldouts need to be classified along two independent axes: content type (~10 classes, e.g. map, anatomical illustration, table, diagram) and language (~5 classes), each single-label/multi-class.
Plan: stratified sampling instead of plain random sampling, since F1 is computed per class and random sampling would leave rare classes with too few examples for a stable score. Pull ~80–100 foldouts per content class from the existing (imperfect) pre-classification, distributed across language classes where possible (~5–10 per language per content class). Total sample: ~800–1000 foldouts.
This sample gets manually annotated to form the gold standard, then run through candidate models to compare per-class F1, macro-F1 (primary metric), micro-F1/accuracy, and cost per 1000 classifications — to find the Pareto-optimal model(s) on F1 vs. cost.
Risk: if the pre-classification is too noisy to locate rare classes, may need a larger unguided sample first (~1500–2000) to surface them before stratifying.