lsdoc

A from-scratch, native-Rust parser for Logseq-flavored Markdown and Org → a typed, serde-serializable AST with source spans, behavior-equivalent to Logseq's mldoc at the granularity that matters for indexing and rendering.

It is the intended single source of truth for parsing in Tine (sibling outliner), including document-root outline recognition through the stable source-oriented API in OUTLINE.md. See SPEC.md for the full brief and DECISIONS.md for the design log (mldoc quirks, intentional deviations, complexity decisions).

Status

Markdown AND Org complete — render-level mldoc parity except for one intentional, semantically classified Markdown family (D48). D48 preserves valid dollar-delimited LaTeX directly inside ordinary Markdown emphasis. The differential gates report exact matches, classified intentional hits, and unclassified mismatches separately; a registry entry alone cannot suppress a mismatch. Org remains exact, and every format retains a zero-unclassified-diff invariant. Real graph/block gates, differential fuzz, deterministic complexity tests, and the perf suite cover correctness, linearity, and stack bounds. Milestone order (each gated by "zero unclassified oracle diffs on its slice + perf budgets hold"):

1. ✅ Harness / oracle / corpus / normalization + regression loop
2. ✅ Block structure (paragraphs, headings, lists, code fences, properties, quotes, hr, tables)
3. ✅ Inline core (text, emphasis, code, links/images, autolinks, escapes)
4. ✅ Logseq dialect inline ([[page]], #tag, ((uuid)), {{macros}}, math, timestamps)
5. ✅ Hardening (differential fuzz, perf + stack-overflow gate, real-graph diff)
6. ✅ Org mode (headlines, markers/priority/tags, org emphasis, [[t][l]] links, #+ directives/blocks, drawers)
7. ✅ Render-level parity (image-ness, link metadata/title, list checkboxes, org targets)
   • a blessed public API (lsdoc::ast + parse/refs), consumable as a git dependency

Using lsdoc as a library

The stable surface is lsdoc::ast (the serde-serializable AST — see AST.md for the field-by-field render contract), the source-oriented outline types documented in OUTLINE.md, and these entry points:

• parse(input, format) -> Vec<ast::Block> — render AST
• refs(input, format) -> ast::Refs — reference index
• parse_outline(input, format) -> Result<DocumentOutline, OutlineParseError> — parser-accepted document-root headers and exact source ranges, from one v2 block parse with no refs pass

It depends only on serde + serde_json and is consumed as a Cargo git dependency (AGPL-3.0):

lsdoc = { git = "https://github.com/martinkoutecky/lsdoc", rev = "…" }

Edition 2021 (MSRV ≈ 1.70). Tine renders every construct from the AST alone; the lsdoc-side integration prereqs (RENDER-PARITY-AND-INTEGRATION.md §1–§3) are complete. Remaining work is Tine-side (consume the AST, delete parseInline.ts, repoint refs.rs's inline half).

The oracle

Correctness is checked differentially against real mldoc (pinned latest mldoc@1.5.9), run under Node:

input string
  → mldoc Mldoc.parseJson (JSON AST)        # harness/, the reference
  → normalized "observable" projection      # block structure + inline tree + ref set
  ↕ compared against
  → lsdoc parse → same normalized projection

We do not bind to mldoc's exact internal node identity (some of it is legacy/quirky). We compare on a normalized projection: block kind/level/nesting/ properties, the ordered inline tree (kind + payload), and the OG-faithful ref set (page/block/tag/embed, UUID-gated as block.cljs does it). Spans are excluded from the comparison (mldoc emits no inline spans and its block spans are quirky); lsdoc tracks spans internally and verifies them with its own unit tests. Intentional intentional deviations live in DECISIONS.md and require both an exact registry entry (for deterministic corpus IDs) and a fail-closed semantic proof.

Correctness is necessary but not sufficient: a separate adversarial perf suite and a fuzz loop guard against O(n²)/O(2^n)/stack-overflow behavior that passes every correctness diff. No parser phase is worse than O(n log n) without a written justification in DECISIONS.md.

Check lsdoc against your own Logseq graph

Want to help? Run lsdoc and Logseq's own parser (mldoc) side-by-side on your graph and see where they disagree or how the speed compares. It all happens on your machine — nothing is uploaded. The only output is a local report file, and any divergence snippets in it are anonymized and re-verified before they land in the report, so you can read it and decide what (if anything) to share.

All you need is Node.js (v18+). No Rust, no compiler — the tool downloads a prebuilt lsdoc binary for your platform on first run.

git clone https://github.com/martinkoutecky/lsdoc
cd lsdoc
node tools/graph-check.mjs /path/to/your/logseq/graph --mode both

Point it at your graph root — the folder that contains pages/ and journals/. On the first run it auto-installs the reference parser (mldoc, via npm) and downloads the prebuilt lsdoc binary from the latest release, then writes graph-check-report.md in the current directory. Modes: --mode diff (only look for parser disagreements), --mode bench (only compare speed), --mode both (default). Add --help for more flags.

(If no prebuilt exists for your platform, it falls back to building from source, which then needs the Rust toolchain and a C compiler — build-essential on Linux, xcode-select --install on macOS.)

If you hit a snag, open an issue — a paste of the terminal output is enough to start.

Running

source scripts/env.sh                  # shared Rust toolchain on /aux (cargo 1.96)
cargo test                             # unit tests + fast perf/stack smoke
cargo test --release -- --ignored      # full-scale perf + stack-overflow gate

# Oracle harness (Node 20):
cd harness && npm install              # installs mldoc@1.5.9 (once)

# One-command differential regression loop (the dev gate):
#   corpus (inline + block + mined + real) → mldoc oracle → lsdoc → compare → report
node run.mjs                           # exits non-zero on any divergence
node run.mjs --no-gen                  # skip corpus regeneration
node fuzz.mjs [count] [seed]           # differential fuzz (panic + oracle-mismatch)
node fuzz-triage.mjs [count] [seed]    # bucket fuzz mismatches by structure
#   run.mjs writes divergences.json (drill-down) for compare.mjs mismatches.

Layout

• src/ — the parser crate (standalone; no dependency on Tine).
• harness/ — the live Node oracle + differential regression loop.
• bootstrap/ — the 2026-06-28 divergence-spike output: seed corpus, the block.cljs-faithful mldoc oracle, and FINDINGS.md (read it). Treated as the seed for harness/, not rebuilt.
• scripts/env.sh — sources the shared Rust toolchain.