Skip to content

AOF grows without bound: no automatic rewrite exists, and multi-shard BGREWRITEAOF is gated off by default #433

Description

@TinDang97

What

There is no automatic AOF rewrite in moon. auto-aof-rewrite-percentage / auto-aof-rewrite-min-size — the Redis knobs that trigger compaction — do not exist anywhere in src/:

$ grep -rn "auto_aof_rewrite\|auto-aof-rewrite" src/
(no matches)

The manual escape hatch is also unavailable on the multi-shard configurations moon is built for: BGREWRITEAOF is gated whenever per_shard_aof_active(num_shards) holds (--shards >= 2 + appendonly=yes, src/main.rs:988), because multi-shard rewrite lost ~38% of keys on restart (verified 2026-05-26, docs/runbooks/multi-shard-aof-rewrite.md). --experimental-per-shard-rewrite opts back in, but it is experimental by name and default-off.

Net effect by configuration:

Config Automatic rewrite Manual BGREWRITEAOF
--shards 1, appendonly yes none available
--shards >= 2, appendonly yes none gated off (unless --experimental-per-shard-rewrite)

Why it matters

The AOF is append-only with no compaction path, so it grows monotonically with write volume rather than with dataset size. Observed on a live instance: 4.8 GB appendonlydir growing ~1 GB/day against a 2.43 GB logical dataset — the file is mostly superseded writes that nothing will ever reclaim. On a long-lived instance this ends in a full disk, at which point moon's own MOONERR diskfull guard pauses writes.

The gating decision itself is correct — silently losing 38% of keys is far worse than an oversized file. The gap is that it leaves no supported compaction path for the default multi-shard deployment, and nothing surfaces that to the operator.

Possible directions

  1. Implement auto-aof-rewrite-percentage / -min-size for the single-shard path, where rewrite already works.
  2. Promote per-shard rewrite out of experimental once tests/crash_matrix_per_shard_bgrewriteaof.rs is trusted, then wire automatic triggering to it.
  3. Interim, cheap: report AOF size vs. dataset size in INFO persistence and warn on a large ratio, so operators can act before the disk does. (Blocked on INFO persistence reports aof_enabled:0 unconditionally while --appendonly defaults to yes #432 — the same INFO block currently reports aof_enabled:0.)

Provenance

Found while diagnosing disk growth on a live instance during the c10k hardening campaign (#20); out of scope for that work, filed separately as promised in #431.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions