ROLE/INFO: master_repl_offset advances with no replica ever attached (Redis holds it at 0)

[TinDang97]

Divergence

With persistence enabled (the default when --dir is set and --appendonly is not disabled), moon advances master_repl_offset on every write even though no replica has ever connected. Redis creates the replication backlog lazily — the offset stays 0 until the first replica attaches — so the two disagree on a byte-compared ROLE.

Reproduce

$ moon --port 6492 --shards 4 --dir /tmp/d --disk-free-min-pct 0
$ redis-cli -p 6492 ROLE                 # fresh
1) "master"
2) (integer) 0
3) (empty array)
$ for i in $(seq 20); do redis-cli -p 6492 SET k$i v; done
$ redis-cli -p 6492 ROLE
1) "master"
2) (integer) 570                          <-- redis reports 0 here
$ redis-cli -p 6492 INFO replication | grep master_repl_offset
master_repl_offset:570

Real redis 8.6.1, same writes: master_repl_offset:0 and ROLE → :0.

Note --appendonly no masks it: the offset stays 0, which is why an isolated probe can read as clean.

Impact

Low but observable. Anything that reads the offset as "has this master made progress / is it a fresh master" sees a nonzero value on a standalone master. Both ROLE and INFO replication expose it.

How it surfaced

scripts/test-consistency.sh's ROLE on a master row (added by #471, which derives ROLE from real server state) has been red since it landed — the suite is not part of PR CI, so nothing caught it. Confirmed unrelated to shard routing: the divergence is a counter, reproduces at --shards 1, and my isolated fresh-server probe is byte-identical to redis.

Fix direction

Either gate offset advancement on a backlog/replica actually existing (matching redis's lazy-backlog semantics), or decide the divergence is intended and relax the parity row to compare the role name + replicas array only — but that decision should be explicit, not left as a permanently red assert.

[TinDang97]

Still reproduces on current main (native macOS arm64 release, --shards 4, persistence on),
side by side with a real redis-server oracle. The --appendonly no mask this issue warned about
is real — I confirmed it, which is why the probe pins the persistence-on configuration:

  moon --appendonly no    ROLE offset after 20 writes = 0    <- masks the bug
  moon (persistence on)   ROLE offset after 20 writes = 570
  redis-server (oracle)   ROLE offset after 20 writes = 0

It is four fields, not one

Comparing the whole INFO replication block after the same 20 writes, no replica ever attached:

field	moon	redis 8.x
role	master	master	same
connected_slaves	0	0	same
master_repl_offset	570	0	differs
repl_backlog_active	1	0	differs
repl_backlog_first_byte_offset	1	0	differs
repl_backlog_histlen	570	0	differs
repl_backlog_size	1048576	1048576	same

One root cause: moon behaves as though the replication backlog is always active, where Redis
creates it lazily on first replica attach. Fixing only master_repl_offset would leave a
standalone master still advertising repl_backlog_active:1 with a nonzero histlen — arguably a
worse state than now, because the three fields would then contradict each other.

Where the fix can and cannot go

master_repl_offset is not a reporting-only value in moon — it is the LSN space.
ReplicationState::issue_lsn (src/replication/state.rs) returns the pre-increment value as the
LSN that tags the corresponding AOF entry and backlog entry, and seed_master_offset restores it
from max(shards[*].max_lsn) after recovery so a restarted master cannot reissue an on-disk LSN
(per-shard AOF RFC § 2 Rule 3). So "stop advancing it" is not available; that would break AOF
entry identity and PSYNC continuation.

The change therefore belongs at the reporting boundary — src/command/identity.rs (ROLE),
src/command/connection.rs (INFO), src/replication/handshake.rs (the master INFO block, which
also hardcodes repl_backlog_active:1) — reporting the Redis-equivalent zeros until a backlog
genuinely exists, and the real values thereafter.

That needs one design decision stated before implementation: the gate is "has a replica ever
attached", not replicas.len() > 0. Redis keeps the backlog alive after the last replica
disconnects (repl-backlog-ttl), so flipping back to 0 when a replica drops would be a new
divergence introduced by the fix. ReplicationState has no such flag today.

Not attempting the fix in this pass: it touches the replication plane, which needs parity +
kill-9 + a both-directions VM A/B before it can be trusted, and I would rather land it with those
gates than without.

[TinDang97]

Grounded the two fix directions against the current code. The first one — "gate offset advancement on a backlog/replica actually existing" — is not safe in moon, and the reason is structural rather than a matter of taste.

master_repl_offset here is not Redis's backlog byte counter. It is the LSN allocator, and several invariants are written in terms of it:

• ReplicationState::seed_master_offset (src/replication/state.rs:320) seeds it from the max LSN observed across the per-shard AOFs before the server accepts writes. Per-shard AOF RFC §2 Rule 3: skip that and the next write issues an LSN already on disk, breaking the lsn → entry uniqueness the backlog merge depends on.
• It is bumped in lockstep with the per-shard axis (increment_shard_offset / issue_lsn) to hold Σ shard_offsets == total_offset(). Task fix: wave 0-4 gap closure — correctness, observability, CI, durability, compat #67 records what happens when the two axes diverge: a restarted --shards >= 2 master left wait_for_replicas blocked forever, reproduced twice by the 24h soak.
• check_and_initiate_failover (src/cluster/failover.rs:33) compares replicas by this offset, and CLUSTER SHARDS reports it (src/cluster/command.rs:722).

A standalone master that stopped advancing it until a replica attached would therefore issue duplicate LSNs on the first write after recovery. So the divergence in #536 is a reporting difference on top of a counter that must keep moving.

That leaves three real options, and the choice is about what master_repl_offset is meant to mean to a reader, so it should be made deliberately:

1. Report Redis's value, keep the internal one. Gate only the ROLE / INFO replication read sites (src/command/identity.rs:36,66, src/replication/handshake.rs:90,145, src/command/connection.rs:697) on "a replica/backlog has ever existed". Closest to byte-parity. Caveat worth deciding on: at first attach the reported value would jump 0 → accumulated, whereas Redis starts counting from 0 at that moment. Reporting current - baseline_at_first_backlog matches Redis more closely but then disagrees with the offset replicas actually use in PSYNC, which is arguably worse.
2. Accept the divergence, and relax the parity row to compare role name + replicas array — with the reason recorded next to the assert, not left implicit.
3. Keep as-is and leave the row red. Not acceptable — a permanently red assert trains people to ignore the suite.

I did not implement any of these: (1) touches replication reporting that failover and monitoring read, and this repo's bar for replication changes is parity + kill-9 + a VM A/B in both directions, which is not proportionate to a "low but observable" cosmetic field without a decision on which meaning is wanted.

Recommendation: (1) with the plain jump (not a rebased baseline), since the field is documented as "master's replication offset" and a monitor computing lag against a replica's ACK offset must see the same number the replica reports. But that is a call about intended meaning, so flagging rather than assuming.

[TinDang97]

New measurement for this issue, from an independent direction: ROLE on a standalone master with no replica ever attached.

scripts/test-consistency.sh has had a ROLE on a master parity check since before this issue was filed, but it never ran — the script aborted 40% earlier on an uninitialised TOTAL under set -u and exited 0 while doing it (fixed in #634). With the script completing, the check fails at every shard count:

                     redis 8.6.1        moon 0.8.5+ (fd838ede)
after ~380 writes    master             master
                     0                  258181
                     (no replicas)      (no replicas)

In isolation, before any write, the two are byte-identical on the wire — *3 $6 master :0 *0 on both. The divergence appears only after traffic: Redis leaves master_repl_offset at 0 while there is no replica and no backlog, Moon advances it per write regardless.

That makes the visible surface of this issue larger than INFO replication: ROLE's second element is the same number, and it is what a client uses to decide whether a server it just connected to has served anything. A monitoring agent comparing ROLE offsets across a fleet of standalone masters sees continuous "replication progress" on servers with no replication configured.

No change proposed here — this issue is still waiting on the decision about what master_repl_offset should mean. Recording the second surface so whichever answer is chosen covers both.

[TinDang97]

Still reproduces on `main` (verified 2026-08-23 in a throwaway worktree built from `54f71b46`, macOS aarch64, `--shards 1`):

```
ROLE -> master / 260242 / (empty array)
redis -> master / 0 / (empty array)
```

Surfaced again from the other direction while adding DEBUG DIGEST (#636): scripts/test-consistency.sh's ROLE on a master row is the single red row in an otherwise-green 456-row run, and has been since #471 added it.

I filed #678 for this before spotting this issue — closing that one as a duplicate of this.