Runbook — multi-shard AOF (per-shard layout)¶

Status: Resolved. The startup refusal gate introduced in v0.1.13 was lifted in v0.1.13-patch (PR #129) once the per-shard AOF replay path was shipped and verified (CRASH-01-LITE: 200/200 SIGKILL recovery, 0 data loss).

Background (historical context)¶

Prior to PR #129, Moon refused to start with --shards >= 2 + --appendonly yes because the single-writer AOF implementation lost ~50 % of writes on SIGKILL in that configuration. The fix was a full per-shard AOF architecture (Option B): each shard owns its own writer task and recovery walks every shard's segment manifest independently.

If you are running Moon ≤ v0.1.13 and hit the old startup error, see the Upgrading section below.

Current architecture (v0.1.13-patch / PR #129 and later)¶

Per-shard AOF layout¶

<persistence_dir>/
  appendonlydir/
    moon.aof.manifest          ← top-level manifest (layout: PerShard)
    shard-0/
      moon.aof.1.base.rdb      ← base snapshot for shard 0
      moon.aof.1.incr.aof      ← incremental log for shard 0
    shard-1/
      moon.aof.1.base.rdb
      moon.aof.1.incr.aof
    shard-N/
      ...

Each shard's writer task appends to its own .incr.aof file. On restart, Moon opens the manifest, discovers all shard directories, and replays each shard's log independently. Shard replay is parallel — recovery time does not grow linearly with shard count.

Durability invariants¶

appendfsync	Guarantee
`always`	Write is on durable storage before +OK (AppendSync rendezvous)
`everysec` (default)	Fsync runs every second; at most ~1 s of writes at risk on crash
`no`	OS decides when to flush; fastest but weakest guarantee

BGREWRITEAOF in per-shard mode¶

BGREWRITEAOF is fully supported on PerShard layouts (#433, un-gated 2026-08): the per-shard fan-out folds every shard cooperatively (C4 snapshot via the shard event loop), advances each shard's manifest entry, and the last writer performs a single synchronized seq commit + old-generation prune. Exactness (no dropped, no double-applied writes across a rewrite that straddles a live write stream + SIGKILL) is pinned by tests/crash_matrix_per_shard_bgrewriteaof.rs. --experimental-per-shard-rewrite, the old opt-in, is deprecated and a no-op.

Automatic rewrite (#433)¶

The AOF compacts itself, Redis-style: when its on-disk size exceeds auto-aof-rewrite-min-size (default 64mb) AND has grown auto-aof-rewrite-percentage% (default 100) over its size after the last rewrite, a background rewrite is dispatched automatically. Set the percentage to 0 to disable automatic rewrites (manual BGREWRITEAOF always works). Watch INFO persistence → aof_base_size / aof_current_size to see the trigger's inputs.

Upgrading from v0.1.13 (old TopLevel AOF) to per-shard layout¶

If you have an existing deployment with a TopLevel AOF manifest (single writer, layout: TopLevel) and want to migrate to per-shard layout:

Option A — cold migration (recommended, zero-risk)¶

Stop the server.
Run BGSAVE on the last healthy instance, or copy dump.rdb from --dir.
Remove appendonlydir/ entirely.
Restart with --shards N --appendonly yes. Moon creates a fresh per-shard manifest. Recovery loads from RDB; the incremental AOF starts empty.

Option B — in-place migration (future tooling)¶

An offline migration CLI subcommand is planned for v0.2. Until then, use Option A.

Safety guard — TopLevel manifest with multi-shard startup¶

If Moon detects an existing TopLevel AOF manifest at startup with --shards >= 2, it refuses to start with exit code 2 and prints the following to stderr:

REFUSING TO START: legacy TopLevel AOF manifest at <path> detected with
--shards N (>= 2). This combination silently loses data for shards 1..N-1.
See docs/runbooks/multi-shard-aof-rewrite.md for migration instructions.

(Exact text may include additional context such as the manifest path and shard count substituted in; the key phrase to match in alerting is REFUSING TO START.)

This is intentional — a TopLevel log does not capture per-shard ordering, so replaying it on a multi-shard instance would produce incorrect key routing.

Resolution: Follow Option A above (remove appendonlydir/ and restart).

Deprecated flag: --unsafe-multishard-aof¶

The --unsafe-multishard-aof flag was introduced in v0.1.13 as an escape hatch to acknowledge the known ~50 % data-loss risk. It is now deprecated and will be removed in v0.2:

The underlying bug is fixed — the flag no longer suppresses any safety gate.
Passing it emits a [DEPRECATED] warning at startup.
If you have scripts or systemd units that pass --unsafe-multishard-aof, remove that flag — it is a no-op.

Monitoring and telemetry¶

INFO Persistence fields added in PR #129¶

aof_backpressure_dropped:<N>   ← count of writes dropped due to full AOF channel

A non-zero value indicates the AOF writer is falling behind write throughput. Investigate disk I/O or increase aof-rewrite-min-size.

Prometheus / alerting¶

A dedicated gauge for aof_backpressure_dropped is planned for v0.2. Until then, monitor via INFO persistence polling.

Crash recovery verification¶

Run the CRASH-01-LITE suite to verify your configuration recovers cleanly:

# From the moon-dev OrbStack VM:
cargo test --release crash_01_lite 2>&1 | tail -20

Expected: 200/200 entries recovered across all shards after SIGKILL.

Escalation¶

If you observe data loss after a crash on v0.1.13-patch or later, collect:

appendonlydir/moon.aof.manifest contents
appendonlydir/shard-*/ file sizes and modification times
Server log from the crashed process (look for AOF writer task exit reason)
INFO persistence output from the recovered instance

File a P0 with these artifacts attached.