An incremental corpus centroid that is still the mean after you remove things from it, with drift statistics that do not fire when the corpus simply gets broader.
import { CentroidTracker, compareDrift, standardizedSimilarity } from 'centroid-tracker';
const corpus = new CentroidTracker({ dimension: 768, requireUnit: true });
for (const doc of documents) corpus.add(doc.id, doc.embedding);
const baseline = corpus.snapshot();
corpus.remove('doc-4812'); // refuses if it was never added
corpus.add('doc-9930', embedding);
const report = compareDrift(baseline, corpus.snapshot());
report.directionDegrees; // how far the centre moved
report.concentrationChange; // how much broader the corpus got
standardizedSimilarity(baseline, embedding).z; // is this document unusual for that corpus
corpus.center(embedding); // x minus the arithmetic meanThe obvious implementation is a running sum over a count, with removal as subtraction. Here it is, in full, failing:
const sum = new Float32Array(dim);
add: for (d) sum[d] += v[d]; count++;
remove: for (d) sum[d] -= v[d]; count--;
mean: sum.map(s => s / count);Twenty thousand documents of magnitude 1e7, then three documents of [1, 2, 3, 4], then every large document removed. The mean of what remains is exactly [1, 2, 3, 4]. That accumulator returns:
[7992, -15139, -15141.33, -15143.67]
Two separate failures produced that. Float32 carries about seven decimal digits, so once the running sum passed 1e11 the small documents rounded away entirely on the way in. And rounding is not reversible, so subtracting the same twenty thousand vectors back out did not return the accumulator to where it started. What is left is cancellation residue.
Nothing downstream can see this. The result is finite, it has no NaN in it, its magnitude is unremarkable, and it is not the mean of anything.
Float64 does not fix it, it only moves the threshold. Add 1, add 1e17, remove 1e17, and a plain float64 sum returns 0.
This library accumulates in float64 with Kahan-Babuska-Neumaier compensation, keeping the bits each addition would have dropped in a parallel array and reading back hi + lo. The classical error bound is proportional to the size of the final sum rather than to the size of the partial sums, which is exactly the property add-then-remove needs. On the case above it returns [1, 2, 3, 4] exactly, and on the float64 case it returns 1.
The bound is not asserted, it is computed. relativeErrorBound() reports a conservative bound on the relative error of the current sum, derived from the running mass of absolute contributions. It stays around 1e-16 for realistic corpora and grows only where the workload is genuinely ill conditioned.
An implementation where the caller passes the vector to remove will accept the same document twice, or a document that was never added, and return a plausible centroid both times. The count goes negative eventually. Nothing checks.
remove(id) takes only an id, and subtracts the exact bytes that add(id, v) accepted. A second remove is an UNKNOWN_ID error naming the id and the current corpus size. Removing something never added is the same error. Re-adding an existing id is refused too, because "this is a duplicate document" and "this document was re-encoded" need opposite handling of the count and picking one silently leaves the other case wrong forever.
The stored vector is a copy. If the caller reuses a scratch buffer between add and remove, subtraction would otherwise remove a vector that was never added.
This is the part that ships to production and quietly lies.
For unit-normalised embeddings, the average cosine similarity of a corpus to its own centroid is not a measurement. It is an identity:
(1/n) Σ x̂ᵢ · μ̂ = ((1/n) Σ x̂ᵢ) · μ̂ = m · μ̂ = ‖m‖ = R
where m is the mean of the unit vectors and μ̂ = m/‖m‖. The mean similarity of a corpus to its centroid is the length of that centroid, and nothing else. That length is determined entirely by how spread out the corpus is. A tight corpus sits near 0.95. A healthily diverse one sits near 0.25, on the first day, with a perfectly healthy encoder.
So the dashboard that plots mean-similarity-over-time and alerts when it falls is plotting corpus diversity and calling it decay. In this repo's tests, a corpus of 300 documents on one topic reads R = 0.83. Ingest 900 documents on unrelated topics and it reads R = 0.20, every individual document's similarity to the centroid falls with it, and the centroid itself moved 6.6 degrees. Nothing happened. The corpus got broader, which is the thing you wanted.
The mirror failure is worse. Replace that corpus with 300 documents at 45 degrees from the original topic, at the same spread, and R comes back to within 0.0002 of where it was. The dashboard reports no drift at all while the corpus centre has rotated 45 degrees.
Three things follow, and they are the shape of the API:
The statistic is named for what it measures. concentration() returns R. Not coherence, not health, not similarity. It has a documented range of [0, 1] and a documented meaning: how tightly the corpus is gathered, which falls when you add variety.
compareDrift has no combined score. It reports rotation and concentration as separate fields and no verdict. directionDegrees is computed between unit directions and is therefore insensitive to either corpus becoming broader or narrower. concentrationChange is named for the arithmetic it performs. Collapsing the two into one number is the bug.
Per-document scores are standardised against the corpus's own scale. standardizedSimilarity returns z = (cosine - R) / σ, where R is what a typical member of that baseline actually scores and σ is the spread of member scores. Both terms move together under dilution, so the score stays put where the raw cosine collapses.
A fixed cosine threshold gets both directions wrong, and the tests pin both cases. Against a tight corpus (R = 0.96, σ = 0.009), a document at cosine 0.90 clears a 0.7 threshold while sitting seven standard deviations outside the corpus. Against a broad corpus (R = 0.30, σ = 0.17), all 200 of its own members fail that same threshold, and not one of them exceeds |z| = 4.
mean() is the arithmetic mean of the raw vectors, which is what mean centering subtracts. It is dominated by whichever documents have the largest norms, and that is correct for centering.
direction() is the normalised mean of the unit vectors. Document length does not vote. It is the only sensible reference for a cosine.
They are different vectors, and maintaining only one of them means answering one of the two questions wrong. A corpus of [3, 0, 0] and [0, 4, 0] has a direction of exactly 45 degrees between the axes and a raw mean that leans toward the second document.
meanExcluding(id) gives the leave-one-out mean, because centering a document by a mean that includes it leaks that document into its own baseline. At n = 1 the centered vector is exactly zero.
The module throws CentroidError with a code rather than returning a plausible number, wherever a plausible number would be a guess:
mean()on an empty corpus isEMPTY_CORPUS, not the zero vector. Zeros make centering a silent no-op.- An all-zero vector is
ZERO_VECTOR. It has no direction, and it is almost always an empty document or a failed encoder call. - A
NaNorInfinitycomponent isINVALID_VECTOR, naming the index. One of them poisons a dimension of the sum permanently, and removal cannot un-poison it. - An exactly opposed corpus has no mean direction, so
direction()isDEGENERATE_CORPUSrather than an arbitrary unit vector. - Standardising against a baseline with zero spread is
DEGENERATE_CORPUS, because every other vector would come back infinitely deviant. That is a statement about the baseline. compareDriftrefuses a live tracker where a snapshot belongs. Comparing a tracker to its own current centroid always reports no drift, because the corpus is centred on itself by construction.
Every vector is retained, so memory is O(n·d). A 100k-document corpus at 768 dimensions holds about 600 MB. This is the price of the identity guarantee: without the stored vector, remove cannot prove it is subtracting what was added. If you need a bounded-memory centroid and can accept that a double remove is undetectable, this is the wrong library.
snapshot() is O(n·d). The spread of per-document cosines is measured against a direction that moves on every add, so it cannot be maintained incrementally. Snapshots are meant for baselines and checkpoints, not for every document.
The incremental state and rebuild() need not agree bit for bit. Floating-point addition is not associative, and a removal changes the order in which the surviving terms were combined. They agree to within relativeErrorBound(), which is asserted in the tests. rebuild() replays insertion order and is the canonical value when a result has to be reproducible from the member set alone.
relativeErrorBound() covers the raw accumulator only, not the spherical one, and it is a bound rather than a measurement. A small bound proves the sum is fine; a large one means the workload is ill conditioned, not that the answer is currently wrong.
concentration() assumes the corpus is what you want it compared against. It says nothing about whether the encoder changed. Nothing here can detect encoder drift from embeddings alone, because a re-trained encoder produces a self-consistent corpus that looks entirely healthy by every statistic in this module. Detecting that requires re-encoding a fixed probe set with both models and comparing, which is outside what a centroid can see.
norm() overflows for components near the square root of Number.MAX_VALUE. No scaling is applied. Embeddings are nowhere near this, and adding a scaled norm would slow down the common path for a case that does not occur.
Deletion does not shrink the id map's allocation. A tracker cycled through many millions of ids will hold onto the Map growth. Call clear() and re-add, or build a new tracker.
npm install
npm test # 94 tests: cancellation, removal identity, the concentration identity, drift separationThe float32 accumulator and the naive float64 sum are both implemented in the test suite and run against the same inputs, so the failures described above are executed rather than asserted.
MIT