LLM Experiments

LLM experiments: overview

SILNLP can translate with large language models (LLMs) as well as with the NLLB/MADLAD sequence-to-sequence models. There are two kinds of LLM experiment, selected by model_type at the top level of config.yml:

model_type	Page	What it does
local_llm (alias llm)	Local LLM fine-tuning	Downloads an open-weight decoder-only model (Gemma, TranslateGemma, Hunyuan-MT, ...) and fine-tunes it on your corpus with full fine-tuning, LoRA, QLoRA, DoRA or QDoRA. Runs on your own GPU or a ClearML GPU worker.
remote_llm	Remote LLM in-context learning	Does not train. Calls a hosted model (Claude, GPT, Gemini, OpenRouter, ...) through LiteLLM and shows it example translations from your training corpus in the prompt ("in-context learning").

"Remote" here means a hosted model behind an API. It has nothing to do with running a job remotely on a ClearML queue (--clearml-queue). In fact, run remote LLM experiments on your own machine, not on a ClearML queue. The work happens on the provider's servers, so a GPU worker adds nothing.

If model_type is absent, the experiment is treated as a local LLM only when model starts with google/gemma, google/translate-gemma, google/translategemma, tencent/Hunyuan or Hunyuan-MT. Otherwise it is a sequence-to-sequence (NLLB/MADLAD) experiment. A remote LLM experiment always needs model_type: remote_llm.

Both kinds are driven with the same tools as any other experiment (silnlp.nmt.experiment, preprocess, train, test, translate; see NMT: Usage), and use the same data configuration (see Configure a model).

Installation

The LLM libraries are an optional "extra":

poetry install -E llm

It installs bitsandbytes (4-bit quantization), sentence-transformers (embedding retrieval), litellm (hosted models) and rank-bm25 (BM25 retrieval). The VS Code dev container installs it automatically.

What LLM experiments have in common

• One language pair. An LLM experiment supports exactly one source language and one target language. More than one raises an error.
• No SentencePiece vocabulary. The defaults are data.tokenize: false, no vocabulary building, data.mirror: false, data.terms.train: false and data.aligner: fast_align. The model sees the raw detokenized parallel text written by preprocessing.
• Language names in prompts. The {src_lang} and {trg_lang} placeholders in prompts are filled from data.lang_codes, which maps an ISO code to the name shown to the model. A code with no entry is used as the name itself:

  data:
    lang_codes:
      eng: English
      spa: Spanish
  

  Unlike NLLB, these are not tokenizer language codes; write whatever the model will understand best.
• Prompts are configured the same way (below), in infer.prompt for both kinds and also train.prompt for local models.
• Evaluation (test) and translation (translate) work as for other models: test writes the usual scores and translate writes drafts.
• Validation split. If the experiment has no validation split, evaluation during training is switched off automatically.
• Multiple drafts. --multiple-translations works, but multiple_translations_method does not apply (see Producing multiple drafts). Each kind of LLM experiment has its own num_drafts default (1) and rules.
• Confidence scores (--save-confidences) are the mean token log probability of each translation. Each kind has its own restrictions.

Configuring the prompt

The prompt section is infer.prompt (used when translating and testing). A local model also has train.prompt (used to build training examples).

infer:
  prompt:
    system_message: "You are a Bible translator working from {src_lang} into {trg_lang}."
    instruction_template: |
      Translate the following text from {src_lang} to {trg_lang}.

      {examples} Now translate this text:

      {source}
    example_format: text
    num_examples: 5
    example_selection:
      method: bm25

Setting	Description
system_message	Optional system message. May use {src_lang} and {trg_lang}.
instruction_template	The user message. May use {src_lang}, {trg_lang}, {source} (the text to translate) and {examples} (the rendered examples).
example_format	How each example is rendered into {examples}; see below.
num_examples	How many examples to put in each prompt. 0 means no examples (zero-shot). The default is 0 for local models and 100 for remote models.
example_selection.method	How the examples are chosen: tfidf, bm25, embedding or coverage. The default is tfidf for local models and coverage for remote models.
example_selection.model	Only for embedding: the sentence-transformers model. Default sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2.

Notes:

• Templates are filled with Python's str.format, so a literal { or } in a template must be written {{ or }}.
• Any setting left unset gets a default for that kind of experiment. If you do not set instruction_template, you get a zero-shot template when num_examples is 0 and a few-shot template (with an {examples} section) when it is greater than 0. If you do set it, it is used as written.
• A warning is logged if num_examples is greater than 0 but the template has no {examples} (the examples would be silently dropped), or if num_examples is 0 but the template contains {examples} (it would render as nothing).
• params.prompt is no longer read. It logs a warning and is ignored; move its settings to infer.prompt (and train.prompt).
• num_examples must not be negative.

Example formats

example_format is either a string (the type) or a mapping with type and, for text, the options template, separator and escape.

Type	Output
text	One block per example, built from template, which may use {src_lang}, {trg_lang}, {source} and {target}. Blocks are joined with separator (default empty). If escape is true, &, < and > in the examples and in the text to translate are XML-escaped; quotes are left alone, because an apostrophe is a letter in some orthographies. The default is template: "Source ({src_lang}): {source}\nTranslation ({trg_lang}): {target}\n\n", with no separator and no escaping (remote models default to something different; see Remote LLM in-context learning).
json	A JSON list of {"source": ..., "target": ...} objects.
xml	One <example><source>...</source><target>...</target></example> element per example, XML-escaped.

example_format:
  type: text
  template: "{src_lang}: {source}\n{trg_lang}: {target}"
  separator: "\n\n"
  escape: true

How examples are selected

The examples come from the experiment's training data (never the validation or test split), as written by preprocessing. The source side of the training corpus is indexed, and for each text to translate the num_examples training verses with the most similar source are chosen.

• tfidf and bm25 are lexical. Text is lowercased and split into words with the sil-machine Latin word tokenizer, and punctuation is ignored. The index is rebuilt in seconds whenever it is needed.
• coverage aims for examples that between them show how each part of the text to translate is rendered, rather than just the closest few. It scores sources by character n-grams (2 to 5 characters), so it does not need word boundaries and suits scripts written without spaces. It takes the most similar examples outright (up to 20, or half of num_examples if that is fewer), then repeatedly adds the example that balances similarity against showing n-grams of the text that the chosen examples do not yet cover. Duplicate examples are dropped, and ties are broken by position, so the same corpus always gives the same prompt. Its index is rebuilt in seconds, like the lexical ones.
• embedding ranks by cosine similarity of sentence embeddings. Computing the embeddings of a large corpus is slow, so a remote LLM experiment saves them (example_retrieval_embeddings.npy and example_retrieval_meta.json) in the checkpoint folder and reuses them if the method, model and number of training examples are unchanged.
• When fewer examples are requested than the corpus holds, they are ordered from least to most similar, so the most relevant example is the one closest to the text to translate. When num_examples is at least the corpus size, every example is used in corpus order and no index is built.
• While training a local model, the verse being trained on is excluded from its own examples (leave-one-out).
• If num_examples is greater than 0 and the training files do not exist, the experiment stops with an error asking you to run preprocessing first. With num_examples: 0 the corpus is never read.