Flat token buffer: lexer-fused parsing, flat mbe transcription, view-backed macro args [-1.07% incl tt-heavy]

[xmakro]

The full flat-token architecture. Five commits:

1. Replace the tree-walking token cursor with a pre-flattened buffer. bump is an index increment, lookahead is direct indexing, parser snapshots and lazy capture replay share the buffer via refcount bumps, and whole delimited sequences skip in one step through an open/close match table. Skipped invisible delimiters stay in the buffer as entries (filtered on consumption) so tree-level lookahead and nesting-depth queries keep byte-exact semantics.
2. Emit the flat buffer directly from the lexer. Primary parses never construct a token tree at all; the few cold callers that need one (proc-macro from_str, cmdline attrs, diagnostics fake streams) rebuild it from the buffer. Delimiter-recovery diagnostics are preserved byte-for-byte.
3. Transcribe macro output directly into the flat buffer. mbe transcription writes through a FlatSink; captured $x:tt fragments splice as buffer slices, restoring the Arc-clone-class sharing the old cursor had at parse_token_tree.
4. Pass macro arguments as lazy flat views. View-backed TokenStreams and bounded cursors let invocation arguments and expansion output parse straight from the buffer, removing the per-invocation flatten pass and the args tree rebuild (the two regression sources in the earlier prototype). Debug output is bounded to the viewed range and the pre-expansion KeywordIdents lint scans flat entries, so neither materializes every invocation in the crate. Attributes and macro-definition bodies stay eager so long-lived nodes do not pin token buffers.
5. Remove the now-dead tree-walking TokenCursor.

Performance (ThinLTO stage2, instructions:u, full scenario, measured on top of the whole perf stack):

• vs the old tree-walking cursor, including the tt-heavy crates (tt-muncher, token-stream-stress, html5ever, deep-vector, added to local coverage after an earlier measurement blind spot): -1.07% geomean, 14 cells improved >= 0.25%, 0 regressed (deep-vector -5.4 to -5.6%, tt-muncher -2.5%, syn check -0.66%). The intermediate cursor+fusion state regressed tt-muncher check +34%; the flat-first expansion step closed that hole.
• Synthetic uom-style tt pass-through stress (30k-token fragment through 16 rules): old cursor 98.83M instructions, this branch 98.69M. At parity or better even on the pure pass-through worst case.
• max-rss: -0.67% geomean for the flat machinery; the only increases are on tt-heavy stress crates (worst cell +5.6%) from expansion-buffer slices retaining prior-level buffers, bounded by macro recursion depth.

Correctness, validated standalone on this branch (no stack):

• x check compiler and src/librustdoc pass; tests/pretty 110/110.
• debug-assertions build bootstraps green and the full ui suite passes 21320 / 0 failed, zero diffs.
• tests/incremental 178/178; proc-macro input token fidelity (spacing/spans debug dump) and doc-comment desugaring byte-identical vs the old cursor.

Remaining follow-ups (not blockers): proc-macro output still arrives as a tree and pays one flatten per expansion (bridge-side flat emission is the next lever); mbe attr/derive rules materialize a tree at the boundary.