ktag extracts the most relevant tags and keyphrases from a body of natural language text. It is both a Go library and a command-line tool. It requires no external services, no API keys, and no model downloads: all scoring is done in-process using statistical methods.
This software is an experiment to understand the known limitations of current statistical keyword extraction approaches, I'm working on another tool which uses state-of-the-art methods to provide far superior results.
Library
go get github.com/karlbateman/ktag
CLI
go install github.com/karlbateman/ktag/cmd/ktag@latest
The primary entry point is Extract. It accepts a string of text and an
Options struct, and returns a slice of Tag values ordered by relevance
score descending.
package main
import (
"fmt"
"github.com/karlbateman/ktag"
)
func main() {
text := `Machine learning and artificial intelligence are transforming
industries worldwide. Deep learning models require large datasets
and significant computational resources.`
tags, err := ktag.Extract(text, ktag.DefaultOptions())
if err != nil {
panic(err)
}
for _, tag := range tags {
fmt.Printf("%s (%.2f)\n", tag.Text, tag.Score)
}
}DefaultOptions returns phrase-mode extraction with a count of 5 and no
sentiment weighting. Scores are normalised to [0, 1]; the top result always
has a score of 1.0.
// Single words, scored by TF-IDF.
opts := ktag.Options{Mode: ktag.ModeWords, Count: 10}
// Multi-word keyphrases, scored by RAKE.
opts := ktag.Options{Mode: ktag.ModePhrases, Count: 5}
// Multi-word keyphrases, scored by a YAKE-inspired algorithm.
opts := ktag.Options{Mode: ktag.ModeYAKE, Count: 5}When UseSentiment is enabled, candidates whose words carry strong positive or
negative sentiment (drawn from a curated subset of the AFINN-111 lexicon)
receive a score bonus. This is useful when the goal is to surface emotionally
significant phrases rather than purely frequent or distinctive ones.
opts := ktag.Options{
Mode: ktag.ModePhrases,
Count: 5,
UseSentiment: true,
}Additional words can be excluded from extraction without affecting the built-in stopword list. Values are normalised to lowercase before matching so casing does not matter.
opts := ktag.Options{
Mode: ktag.ModePhrases,
Count: 5,
AdditionalStopwords: []string{"company", "product"},
}By default, phrases contain at most three words. MaxPhraseLength overrides
this. A value of 1 is equivalent to word mode but still uses the phrase scoring
algorithm.
opts := ktag.Options{
Mode: ktag.ModeYAKE,
Count: 5,
MaxPhraseLength: 2,
}FilterNounPhrases uses a POS tagger
(via jdkato/prose) to remove phrase
candidates that contain a word appearing exclusively as a finite verb
(VBZ or VBD) in the source document, and never as a noun or adjective. This
eliminates predicate structures such as "freshness draws attention" while
preserving noun-like uses of the same surface form.
opts := ktag.Options{
Mode: ktag.ModeYAKE,
Count: 5,
FilterNounPhrases: true,
}This option applies to ModePhrases and ModeYAKE; it has no effect in
ModeWords.
ktag [flags] [text|file]
echo "text" | ktag [flags]
The positional argument is tried first as a file path. If the path does not exist it is used directly as input text. When no argument is given, the tool reads from stdin.
| Flag | Default | Description |
|---|---|---|
--mode |
phrases |
Extraction mode: words, phrases, or yake |
--count |
5 |
Number of tags to return |
--json |
false |
Output tags as a JSON array |
--scores |
false |
Append relevance scores to plain text output |
--sentiment |
false |
Enable sentiment-weighted scoring |
--stopwords |
`` | Comma-separated additional stopwords |
--filter-pos |
false |
Remove finite-verb-only phrase candidates (requires prose/v2) |
Extract the top five keyphrases from a file:
ktag article.txt
Extract ten single words with scores shown:
ktag --mode words --count 10 --scores article.txt
Extract YAKE keyphrases and output JSON:
ktag --mode yake --json article.txt
Pass text directly as an argument:
ktag --mode words "Go is a statically typed, compiled language."
Pipe from another command:
cat article.txt | ktag --mode phrases --sentiment
Exclude domain-specific terms:
ktag --stopwords "company,platform,service" article.txt
Extraction follows the same pipeline for all modes:
- Normalisation: The input text is lowercased, non-alphabetic characters (except hyphens and apostrophes) are removed, and whitespace is collapsed.
- Candidate extraction: The normalised text is split into candidate words or phrases.
- Scoring: Candidates are scored by the selected algorithm.
- Position boost: Candidates whose first word appears earlier in the document receive a small additive bonus. Words near the top of a document tend to be more representative of its topic.
- Sentiment boost (optional): The AFINN-111 magnitude of each candidate's words is added to its score.
- Ranking and deduplication: Candidates are sorted descending; in phrase modes, candidates that are whole-word substrings of a higher-ranked candidate are discarded to avoid redundant output.
- Score normalisation: Output scores are scaled to [0, 1] so the top result always equals 1.0.
Candidates are individual tokens. Tokens shorter than three characters are discarded. Morphological variants (post, posting, posts) are grouped by their Porter stem and a single representative surface form is kept, the one that appears most frequently, with alphabetical ordering as a tiebreaker. Each surviving word is scored by TF-IDF, where sentences within the document act as the IDF corpus. A word that appears in fewer distinct sentences receives a higher inverse document frequency, rewarding topic-specific vocabulary over words distributed evenly throughout the text.
Candidates are contiguous runs of non-stopword content words extracted from
within clause boundaries (the text is first split on commas, colons, full stops,
and similar delimiters to prevent cross-clause grouping). Single-word candidates
shorter than four characters are discarded unless no longer candidates exist.
Phrases that begin or end with an adverb ending in -ly are also discarded, as
these almost always mark a predicate or modifier rather than a topic phrase.
Scoring uses RAKE (Rapid Automatic Keyword Extraction). Each word in the candidate pool receives a score equal to its co-occurrence degree divided by its raw frequency. Words that appear predominantly inside multi-word phrases score higher than words that appear in isolation. A phrase's score is the sum of its constituent word scores. RAKE was introduced by Rose et al. (2010), "Automatic Keyword Extraction from Individual Documents."
The YAKE algorithm extends phrase-mode extraction with four per-word features that better capture what makes a word distinctive in a specific document:
- Casing: Words that appear with an initial capital mid-sentence (not at a sentence boundary) are likely proper nouns and receive a higher quality score.
- Position: Words that occur earlier in the document are preferred over words concentrated near the end.
- Specificity: Words that appear in fewer distinct sentences are more on-topic and rank higher.
- Coherence: Words that co-occur with a narrow set of neighbours appear in more consistent contexts and are rewarded accordingly.
A frequency component log-normalises raw term frequency relative to the document mean, preventing very common words from dominating. A phrase's score is the geometric mean of its constituent word quality scores, multiplied by phrase length. This linear length bonus keeps multi-word phrases competitive with single-word candidates without inflating the quality bar.
YAKE was introduced by Campos et al. (2018), "YAKE! Collection-Independent Automatic Keyword Extractor," ECIR.
type Options struct {
// Mode controls whether words or keyphrases are extracted.
// Valid values: ModeWords, ModePhrases, ModeYAKE.
Mode Mode
// Count is the maximum number of tags to return. Must be at least 1.
Count int
// UseSentiment enables AFINN-111 sentiment weighting.
// Candidates with strong positive or negative sentiment score higher.
UseSentiment bool
// AdditionalStopwords extends the built-in stopword list for this call.
// Values are normalised to lowercase before matching.
AdditionalStopwords []string
// MaxPhraseLength caps the number of words per keyphrase.
// A value of 0 or less is treated as 3 (the default).
// Applies to ModePhrases and ModeYAKE only.
MaxPhraseLength int
// FilterNounPhrases removes phrase candidates that contain a word
// appearing exclusively as a finite verb (VBZ or VBD) in the source
// document. Requires jdkato/prose/v2.
// Applies to ModePhrases and ModeYAKE only.
FilterNounPhrases bool
}Extract returns an error if the input text is empty
(after trimming whitespace) or if Count is less than 1. An unknown Mode
value also returns an error.
Each Tag carries a text string and a relevance score:
type Tag struct {
Text string `json:"text"`
Score float64 `json:"score"`
}Tags are returned in descending score order. The highest-scoring tag always has a score of exactly 1.0. All other scores are proportional to it.
The CLI renders tags as a comma-separated list by default. With --json the
full slice is written as a JSON array. With --scores, each item in the plain
text output includes the score in parentheses.
The algorithms in this library are heuristic and statistical in nature; they are well suited to exploratory tagging, content enrichment, and developer tooling, but anyone integrating ktag into a production data pipeline should validate its output against their domain before relying on it for downstream decisions.
All three algorithms operate on surface forms, co-occurrence patterns, and position signals. They cannot distinguish a word used ironically from one used sincerely, resolve ambiguous terms (a "java" candidate could be the language or the island), or recognise that two differently worded phrases refer to the same concept. An LLM or a knowledge-graph-backed tagger would handle these cases better; ktag trades that capability for speed and zero external dependencies.
RAKE and YAKE both rely on within-document statistics: co-occurrence degree,
sentence spread, and term frequency ratios. These signals are unreliable when
the document contains fewer than a few hundred words. On very short inputs
(a single sentence, a tweet, a heading) the rankings become largely arbitrary
and the scores compress toward one another. For short text, ModeWords tends to
degrade more gracefully than the phrase modes because TF-IDF has a weaker
dependency on document length.
The sentiment list included here is a curated subset of AFINN-111, selected for
technical writing and blog content. Many common sentiment words are absent, and
the scoring is not calibrated for other genres (legal text, academic prose,
customer reviews). Enabling UseSentiment on documents outside this domain may
produce unexpected ranking shifts.
Candidate phrases are delimited by stopwords and punctuation characters, not by
a parse tree. This means that a run of content words with no intervening
punctuation or stopwords will be treated as a single candidate regardless of the
grammatical structure within it. The FilterNounPhrases option partially
addresses this by removing candidates whose words appear exclusively as finite
verbs, but it operates on per-word POS distributions rather than constituent
structure, so it cannot catch all predicate phrases.
The proper-noun bonus is granted to any word that begins with a capital letter mid-sentence. In all-caps text, title-cased headlines, or non-English documents that capitalise common nouns (German, for example), many ordinary words will receive unwarranted boosts. The algorithm was designed for standard English prose and performs best there.
Two phrases are considered redundant only if one is a whole-word substring of the other after Porter stemming. Semantically equivalent phrases with no lexical overlap ("machine learning" and "neural networks") will both appear in the output even if they refer to the same topic. This is intentional, the algorithm has no way to know they are related, but it means the output is not deduplicated at the concept level.
The built-in list targets general English. Documents that are heavily
domain-specific (medical, legal, financial) will include many high-frequency
field terms that function as stopwords within that domain but are not on the
list. The AdditionalStopwords option handles this at the call site, but it
requires the caller to know in advance which terms to suppress.