Audit: withdraw 2 statistical claims + 5 unreliable probes; package for review - #1
Merged
Conversation
Verified a clean clone + fresh venv on Python 3.14.2 / datasets 5.0.1 / pandas 3.0.5 / numpy 2.5.1. templates.json and phase3_raw.json regenerate byte-identical. Four issues blocked an outside reader from reproducing: - benchmark.py crashed on Windows with UnicodeEncodeError: the capability table printed U+2714/U+2718, which a default cp1252 console cannot encode. It died after writing results.json but before printing the 5/5-vs-0/5 result the README cites. Now ASCII. - outputs/compression_findings.md was hand-edited after generation and had gone stale against outputs/phase3_raw.json shipped beside it: the ELS top-5 tables disagreed (Administrative listed skips 2,3,7,6,5; the JSON says 21,31,14,44,28) and the 'skips with p < 0.01' count read 0 where the JSON says 1. Re-running the documented command silently overwrote it. Committed the script's real output so the two agree and re-running is a no-op. The headline null result is unaffected: Bonferroni-significant is still 0 of 495, and Zipf and RULING-parity figures are unchanged. - benchmark.py's docstring claimed it writes RESULTS.md. It never did. - README omitted the benchmark from 'How to Reproduce', pinned no Python version, and gave no way to tell a good run from a bad one. Added a venv step, runtimes, measured disk footprint, an expected-values table, and 'git diff --stat outputs/ should be empty' as the check. Also stop reporting a RULING-parity p-value for Letter, where n=1 adjacent-chunk pair cannot support one, and flag the small n on Administrative (17) and Royal Inscription (10).
A self-audit of how each p-value was computed found that the headline RULING-parity result (Royal p=0.002, Admin p=0.005) is an artifact of the wrong null hypothesis. It is withdrawn. The null pooled the tablet's tokens, shuffled them, and re-cut at the same lengths. That destroys all local structure, so beating it shows only that Sumerian is locally coherent - true of any natural language, and silent on <RULING> specifically. The null that isolates the question holds real token order and the exact multiset of chunk lengths fixed and permutes only where the cuts fall. Under it the effect disappears in every genre: Administrative obs 0.176 token-shuffle null 0.004 boundary null 0.343 p=0.78 Literary obs 0.731 token-shuffle null 0.018 boundary null 0.889 p=0.91 Lexical obs 0.538 token-shuffle null 0.037 boundary null 0.430 p=0.27 Royal obs 0.500 token-shuffle null 0.001 boundary null 0.500 p=1.00 Observed sits at or below the correct null throughout - opposite the reported direction. ruling_parity now runs both nulls side by side, builds the permutation distribution OF THE MEAN rather than comparing a mean against pooled individual null pairs, and applies (r+1)/(n+1). Also corrected the ELS framing. At 1000 permutations the smallest attainable p is ~0.001, roughly 10x the Bonferroni threshold of 0.000101, so that scan cannot return a Bonferroni-significant result by construction and '0 of 495' overstated its power. It now reports 3 nominal hits at p<0.01 against ~5 expected by chance, which is the informative comparison. Applied (r+1)/(n+1) there too so a p of exactly 0 can no longer clear the threshold un-earned. Unaffected: Zipf, compression delta, Phase 1 probe frequencies, and the kishib3 benchmark. Packaging for external review: CORRECTIONS.md (dated retraction log), REVIEWERS.md (where to attack it, including the parts I could not validate), CITATION.cff, and a claim-by-claim evidence-strength table in the README replacing the uniform 'evidence-backed' framing.
NORTHTEKDevs
changed the title
…over-catch Second audit pass, this time over every remaining quantitative claim. WITHDRAWN: 'Administrative and Royal are DSL-like (Zipf s=1.746/1.737) vs Lexical natural-language-like (s=1.114)'. It is a stream-length artifact. The streams differ ~60x in length (Literary 154,005 tokens, Lexical 2,508). Re-fitting every genre on equal-length blocks collapses a 0.632 spread to 0.096: Administrative 1.746 -> 1.187 Royal 1.737 -> 1.200 Literary 1.680 -> 1.187 Letter 1.607 -> 1.210 Lexical 1.114 -> 1.114 (already shortest) Holds under both contiguous-block and random-token subsampling, 25 draws. Compounding: zipf_fit is OLS on log-log rank-frequency, a biased power-law estimator whose R^2 is not a goodness-of-fit test (Clauset/Shalizi/Newman 2009); an MLE exponent disagrees in magnitude AND rank order. Dropping hapax (34-59% of types) moves exponents up to 0.17 and reverses some comparisons. Pipeline now emits the length control and MLE alongside OLS. SURVIVES: the compression-delta ranking was tested for the same confound rather than assumed safe, and holds at equal-length contiguous blocks (Royal > Admin > Literary/Letter > Lexical, swaps within overlapping sd). That control is now part of the pipeline. It must use CONTIGUOUS blocks -- scattered-position sampling destroys the adjacency zlib exploits and manufactures a collapse. The first version of this audit made that mistake and was redone. CITATIONS: README implication #1 cited P101440, P132611, P117793, P145759 under 'administrative tablets carry kisib3'. Only P101440 is administrative and sealed; the other three are Letters with no seal clause, attributing via u3-na-a-du11 + dub-sar + dumu instead. Now cited separately by mechanism. Verified the P101440 'Concrete Example' decomposition is accurate in every element. PROBES: quantified the over-catch. king_title (\blugal\b, 48.6%) is 54% personal-name elements and only ~12% actual titles -- flagged in code and CORRECTIONS.md as not citable, not silently patched, since name-vs-title disambiguation is beyond regex. year_formula (\bmu\b) matches inside mu-DU and mu-ni; tablet-level impact small (74.2% -> 70.6% strict), README now says ~71%. Confirmed exact: seal_of_PN 25.4%, speak_to_him 58.8%. CORPUS INTEGRITY: confirmed exact -- 91,606 tablets, 6,968,581 glyphs, 0 duplicate ids, 0 duplicate transliterations, sampling seeded at 42.
NORTHTEKDevs
changed the title
NORTHTEKDevs
added a commit
that referenced
this pull request
Aug 1, 2026
…plications
Phase 1 only ever measured how often a probe MATCHED. It never asked whether
the matches meant what the label claimed. They often do not. Added
scripts/phase1b_probe_validation.py so this is a reproducible pipeline stage
rather than an assertion, emitting outputs/probe_validation.{md,json}.
DO NOT CITE (precision):
god_dedication 6% nam-lugal 'kingship' / nam-en 'lordship' are abstract
nouns, not dedications; only nam-ti (10/182) qualifies
excess_diri 19% diri + month name = INTERCALARY MONTH, not a surplus
year_formula 34% '-' is a regex word boundary, so \bmu\b catches the
verbal prefixes mu-na-/mu-un-/mu-ni- (48.6% of matches)
witness_eye 41% igi-ni/zu-se3 = 'before him/you'; igi-nim = 'upper'
king_title 42% mostly lugal- as a PERSONAL NAME element; and even a
genuine hit is ambiguous, since lugal in Ur III admin
commonly means 'owner/master' rather than 'the king'
VALIDATE: received_by 100%, speak_to_him 100%, total_audit 100%,
seal_of_PN 96%, son_of_PN 95%. The README headline figures rest on these.
NEVER FIRES: credit_mu_DU searches mu-DU; the corpus writes mu-kux(DU) --
60 occurrences it never sees. phase1 does ,
so a dead probe vanishes from templates.json silently. The 'delivery/credit'
primitive was therefore never measured at all.
TWO IMPLICATIONS RESTATED, both falsified by the repo's own data:
#1 claimed 'witness clauses are common' and 'no important write is
anonymous, undated, or unattributed'. Genuine witness clauses: 0.4% of
Admin tablets. 23.2% of Admin tablets carry NEITHER seal, year, nor
witness; ~75% carry no seal. Sealing is a strong minority practice.
#3 claimed admin tablets 'close with su-nigin2' and 'nothing drifts
silently'. su-nigin2 is a precise probe but appears on 2.2% of Admin
tablets; la2-ia3 on 3.4%. Real device, described as if characteristic.
The general lesson, now stated in the report: ATTESTED != CHARACTERISTIC.
Added per-genre prevalence tables so a reader cannot repeat that slide.
Epistemic status: discriminators encode standard dictionary values applied
by a NON-SPECIALIST and are deliberately conservative, so these error rates
are LOWER BOUNDS. Not a substitute for Assyriological review; REVIEWERS.md
now asks specifically which of my TRUE buckets are wrong.
NORTHTEKDevs
added a commit
that referenced
this pull request
Aug 1, 2026
…plications
Phase 1 only ever measured how often a probe MATCHED. It never asked whether
the matches meant what the label claimed. They often do not. Added
scripts/phase1b_probe_validation.py so this is a reproducible pipeline stage
rather than an assertion, emitting outputs/probe_validation.{md,json}.
DO NOT CITE (precision):
god_dedication 6% nam-lugal 'kingship' / nam-en 'lordship' are abstract
nouns, not dedications; only nam-ti (10/182) qualifies
excess_diri 19% diri + month name = INTERCALARY MONTH, not a surplus
year_formula 34% '-' is a regex word boundary, so \bmu\b catches the
verbal prefixes mu-na-/mu-un-/mu-ni- (48.6% of matches)
witness_eye 41% igi-ni/zu-se3 = 'before him/you'; igi-nim = 'upper'
king_title 42% mostly lugal- as a PERSONAL NAME element; and even a
genuine hit is ambiguous, since lugal in Ur III admin
commonly means 'owner/master' rather than 'the king'
VALIDATE: received_by 100%, speak_to_him 100%, total_audit 100%,
seal_of_PN 96%, son_of_PN 95%. The README headline figures rest on these.
NEVER FIRES: credit_mu_DU searches mu-DU; the corpus writes mu-kux(DU) --
60 occurrences it never sees. phase1 does ,
so a dead probe vanishes from templates.json silently. The 'delivery/credit'
primitive was therefore never measured at all.
TWO IMPLICATIONS RESTATED, both falsified by the repo's own data:
#1 claimed 'witness clauses are common' and 'no important write is
anonymous, undated, or unattributed'. Genuine witness clauses: 0.4% of
Admin tablets. 23.2% of Admin tablets carry NEITHER seal, year, nor
witness; ~75% carry no seal. Sealing is a strong minority practice.
#3 claimed admin tablets 'close with su-nigin2' and 'nothing drifts
silently'. su-nigin2 is a precise probe but appears on 2.2% of Admin
tablets; la2-ia3 on 3.4%. Real device, described as if characteristic.
The general lesson, now stated in the report: ATTESTED != CHARACTERISTIC.
Added per-genre prevalence tables so a reader cannot repeat that slide.
Epistemic status: discriminators encode standard dictionary values applied
by a NON-SPECIALIST and are deliberately conservative, so these error rates
are LOWER BOUNDS. Not a substitute for Assyriological review; REVIEWERS.md
now asks specifically which of my TRUE buckets are wrong.
NORTHTEKDevs
force-pushed
the
fix/reproducibility
branch
2 times, most recently
from
August 1, 2026 04:40
dd5c95e to
ecd000a
Compare
…plications
Phase 1 only ever measured how often a probe MATCHED. It never asked whether
the matches meant what the label claimed. They often do not. Added
scripts/phase1b_probe_validation.py so this is a reproducible pipeline stage
rather than an assertion, emitting outputs/probe_validation.{md,json}.
DO NOT CITE (precision):
god_dedication 6% nam-lugal 'kingship' / nam-en 'lordship' are abstract
nouns, not dedications; only nam-ti (10/182) qualifies
excess_diri 19% diri + month name = INTERCALARY MONTH, not a surplus
year_formula 34% '-' is a regex word boundary, so \bmu\b catches the
verbal prefixes mu-na-/mu-un-/mu-ni- (48.6% of matches)
witness_eye 41% igi-ni/zu-se3 = 'before him/you'; igi-nim = 'upper'
king_title 42% mostly lugal- as a PERSONAL NAME element; and even a
genuine hit is ambiguous, since lugal in Ur III admin
commonly means 'owner/master' rather than 'the king'
VALIDATE: received_by 100%, speak_to_him 100%, total_audit 100%,
seal_of_PN 96%, son_of_PN 95%. The README headline figures rest on these.
NEVER FIRES: credit_mu_DU searches mu-DU; the corpus writes mu-kux(DU) --
60 occurrences it never sees. phase1 skips any zero-hit probe,
so a dead probe vanishes from templates.json silently. The 'delivery/credit'
primitive was therefore never measured at all.
TWO IMPLICATIONS RESTATED, both falsified by the repo's own data:
#1 claimed 'witness clauses are common' and 'no important write is
anonymous, undated, or unattributed'. Genuine witness clauses: 0.4% of
Admin tablets. 23.2% of Admin tablets carry NEITHER seal, year, nor
witness; ~75% carry no seal. Sealing is a strong minority practice.
#3 claimed admin tablets 'close with su-nigin2' and 'nothing drifts
silently'. su-nigin2 is a precise probe but appears on 2.2% of Admin
tablets; la2-ia3 on 3.4%. Real device, described as if characteristic.
The general lesson, now stated in the report: ATTESTED != CHARACTERISTIC.
Added per-genre prevalence tables so a reader cannot repeat that slide.
Epistemic status: discriminators encode standard dictionary values applied
by a NON-SPECIALIST and are deliberately conservative, so these error rates
are LOWER BOUNDS. Not a substitute for Assyriological review; REVIEWERS.md
now asks specifically which of my TRUE buckets are wrong.
NORTHTEKDevs
changed the title
The withdrawals were recorded in README/CORRECTIONS.md but the hand-authored
design documents still asserted them. A reader landing on outputs/*.md
directly would have found the retracted claims stated confidently, which
would have discredited the corrections themselves. Fixed:
reference_architecture.md '<RULING> is provably a logical row separator,
not a visual hint' -> struck through, with the
boundary-permutation result inline. Also the
'it is a DSL with reserved keywords' and
'Zipf s=1.75 -> typed DSL' passages.
summary.md 'Every clay tablet ends with kisib3 PN ... no
write is anonymous' -> struck through (actual:
25.4% sealed, 0.4% witnessed, 23.2% carry none
of seal/year/witness). Also the RULING parity
confirmation and the two ranking-table rows.
FULL_IDEAS.md item B3's parity p-values.
Documents are preserved rather than rewritten, with inline notices, so the
record of what was claimed stays visible.
templates.json now carries the probe audit in machine-readable form: an
_AUDIT block plus a per-row _audit_verdict on all 41 affected rows, so a
consumer parsing the JSON cannot pick up a discredited frequency without
seeing why it is discredited. Emitted by phase1_templates.py rather than
hand-edited, so regeneration stays byte-identical (verified run-vs-run).
Added scripts/check_integrity.py: every relative link resolves, every
retracted claim carries a correction notice ADJACENT to it, every
DO_NOT_CITE probe is stamped, corpus totals match. 78 checks.
Each check was validated by deliberately breaking it. That found one check
VACUOUS -- a +/-1200 char window let a top-of-file banner satisfy the
correction requirement for an assertion 60 lines below it. Tightened to
-300/+800 and re-verified that it now fails when the adjacent notice is
removed. A check that cannot fail proves nothing.
CI runs the integrity gate and asserts the benchmark's 5/5-vs-0/5 result on
every push. CITATION.cff version 0.2.0 states the retractions in its
abstract so anyone citing this reads them first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits. A full audit of every quantitative claim. Two statistical findings and five of ten regex probes did not survive. Every correction is reproducible from the pipeline, not asserted.
Statistical findings
RULING-parity — WITHDRAWN. The null pooled each tablet's tokens and shuffled them, destroying all local structure. Beating it shows only that Sumerian text is locally coherent — true of any natural language. Under a null that keeps real token order and chunk lengths and permutes only where cuts fall, the effect vanishes in every genre (p = 0.78 / 0.91 / 0.27 / 1.00), with observed values at or below the null.
Zipf-as-DSL — WITHDRAWN. A stream-length artifact. Streams differ ~60x (Literary 154,005 tokens; Lexical 2,508). At equal length a 0.632 spread collapses to 0.096 — every genre lands in 1.11–1.21. Compounding:
zipf_fitis OLS on log-log rank-frequency, a biased estimator whose R² is not a goodness-of-fit test (Clauset/Shalizi/Newman 2009); MLE disagrees in magnitude and rank order.Compression Δ — SURVIVES. Tested against the same confound rather than assumed safe. The genre ranking holds at equal-length contiguous blocks. (Scattered-position sampling destroys the adjacency zlib exploits — my first audit made that error and was redone.) Now the strongest positive statistic here.
ELS — power caveat. At 1,000 permutations the minimum p is ~0.001, ~10x the Bonferroni threshold, so the scan cannot return a Bonferroni-significant result by construction. Reframed as 3 nominal hits vs ~5 expected by chance.
Probe precision audit (new pipeline stage)
scripts/phase1b_probe_validation.py->outputs/probe_validation.md. Phase 1 only measured how often a probe matched, never whether matches meant what the label claimed.god_dedicationnam-lugal"kingship",nam-en"lordship" — abstract nounsexcess_diridiri+ month = intercalary monthyear_formulamu-na-,mu-un-,mu-ni-(48.6% of matches)witness_eyeigi-zu-še₃"before you";igi-nim"upper/north"king_titlelugal-as a personal-name elementValidating:
received_by100%,speak_to_him100%,total_audit100%,seal_of_PN96%,son_of_PN95% — the README headline figures rest on these.credit_mu_DUnever fires. It searchesmu-DU; the corpus writesmu-kuₓ(DU)— 60 occurrences it never sees. Phase 1 silently drops zero-hit probes, so the "delivery/credit" primitive was never measured.Two implications falsified by the repo's own data
šu-nigin₂" and "nothing drifts silently."šu-nigin₂is a precise probe but appears on 2.2% of Administrative tablets;la₂-ia₃on 3.4%.The lesson, now stated in the report with per-genre prevalence tables: attested is not the same as characteristic.
Also
Miscited tablets fixed (P132611/P117793/P145759 are Letters with no seal clause, cited under a seal claim). Reproduce path fixed —
benchmark.pycrashed on Windows (cp1252 vs U+2714);compression_findings.mdhad been hand-edited and disagreed with the JSON beside it. Added CORRECTIONS.md, REVIEWERS.md, CITATION.cff, and a Claims-and-Evidence-Strength table.Epistemic status of the probe audit: discriminators encode standard dictionary values applied by a non-specialist, deliberately conservative, so error rates are lower bounds. Not a substitute for Assyriological review — REVIEWERS.md asks specifically which TRUE buckets are wrong.
Verified: clean clone -> full pipeline ->
git diff --stat outputs/empty; benchmark exits 0.