LLM Deductive Coding Pipeline

A small, reusable Python pipeline for applying deductive coding schemes to text with large language models. It supports direct classification, two-stage evidence gathering and scrutiny, multiple LLM backends, batching, and resumable JSONL checkpoints.

This repository also contains code accompanying two studies of vulnerability classification in police narratives:

• “Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives”, published in the Journal of Quantitative Criminology; and
• Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs, an arXiv preprint under review.

Installation

Clone the repository and install the package:

git clone https://github.com/samrelins/vulnerability_classifier_pipeline.git
cd vulnerability_classifier_pipeline
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e .

To run the Boston paper notebooks and preprocessing scripts, install the paper dependencies as well:

python -m pip install -e '.[paper]'

OpenAI and Together AI endpoints read OPENAI_API_KEY and TOGETHER_API_KEY respectively. The llama.cpp adapter expects an OpenAI-compatible local server and does not require an API key.

Quick start

The public API uses plain dictionaries and DataFrames. create_segments_from_df converts narratives into pipeline inputs; run_pipeline_step processes them in batches and appends complete results to a JSONL checkpoint.

import pandas as pd

from vulnerability_classifier_pipeline import (
    create_segments_from_df,
    get_classifications,
    llama_cpp_endpoint,
    run_pipeline_step,
)

data = pd.DataFrame(
    {
        "report_id": ["example-1"],
        "narrative": ["Example report text."],
    }
)

segments = create_segments_from_df(
    data,
    text_col="narrative",
    id_col="report_id",
    max_words=250,
)

run_pipeline_step(
    input_df=segments,
    output_file_path="outputs/classifications.jsonl",
    processor_fn=get_classifications,
    id_col="id",
    batch_size=10,
    results_per_input=3,
    system_prompt="Classify the report as POSITIVE, INCONCLUSIVE, or NEGATIVE.",
    endpoint=llama_cpp_endpoint,
    base_url="http://localhost:8080",
    n=3,
)

Successful and complete inputs are skipped when the same checkpoint is resumed. Failed or partial inputs remain retryable. A batch is written only after every input has produced the expected number of valid results.

Two-stage classification

The two-stage workflow uses:

1. get_evidence to extract potentially relevant text.
2. scrutinise_evidence to assess the strength of each extracted item.

Because evidence gathering can create several outputs for each input, the evidence JSONL is loaded into a DataFrame and each result_id becomes the unique ID for the scrutiny stage. A complete working example is provided in experiments/two_stage_classifier.ipynb.

LLM endpoints

The package exposes three endpoint adapters with the same batched message interface:

• llama_cpp_endpoint for a local OpenAI-compatible llama.cpp server.
• openai_endpoint for OpenAI Chat Completions.
• together_endpoint for Together AI Chat Completions.

Endpoints receive a list of conversations and return one list of choices for each conversation. Classifiers validate both the batch size and requested number of responses before results are checkpointed.

Repository structure

src/vulnerability_classifier_pipeline/
├── classifiers/
│   ├── simple/          # Direct POSITIVE/INCONCLUSIVE/NEGATIVE coding
│   └── two_step/        # Evidence gathering and scrutiny
├── core/pipeline/       # Batch execution and JSONL checkpointing
├── data/                # Narrative segmentation and DataFrame loading
└── llm_endpoints/       # llama.cpp, OpenAI, and Together AI adapters

boston_fio_paper/        # Paper data, prompts, notebooks, and R analysis
uk_storm_paper/           # UK secure-data study code (restricted data not included)
experiments/             # Current simple and two-stage examples
tests/                   # Offline regression tests and opt-in live smoke test

Paper replication

The Boston Field Interrogation and Observation study classified indicators of mental ill health, drug abuse, alcohol dependence, and homelessness. It compared human and LLM labels and used demographic counterfactuals to examine classification differences.

See boston_fio_paper/README.md for data availability, environment setup, and the replication sequence. External/API execution in the notebooks is disabled by default so that “Run All” cannot accidentally submit paid work.

The UK study used restricted operational incident logs that cannot be shared. Its released materials document the LoRA fine-tuning, secure-environment classification, and statistical analysis workflows, but do not include UK narratives or data derived from them. See uk_storm_paper/README.md for the code map, expected input structures, and security boundary.

Citation

@article{relins2025llm,
  title={Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives},
  author={Relins, Sam and Birks, Daniel and Lloyd, Charlie},
  journal={Journal of Quantitative Criminology},
  year={2025},
  doi={10.1007/s10940-025-09611-z},
  url={https://doi.org/10.1007/s10940-025-09611-z}
}

@misc{relins2026finetuned,
  title={Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs},
  author={Relins, Sam and Birks, Daniel},
  year={2026},
  eprint={2607.18446},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  doi={10.48550/arXiv.2607.18446},
  url={https://arxiv.org/abs/2607.18446}
}

License

Code in this repository is released under the MIT License. Dataset licenses and provenance are described in the Boston replication README and associated OSF repository.