From 7c9b13051db17e01b8ddfd6d74773dae236ff615 Mon Sep 17 00:00:00 2001 From: veg Date: Sun, 31 May 2026 02:03:35 +0000 Subject: [PATCH] Make 'nothing archived' a distinct exit (3) and fsync the dir on publish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two further hardening findings: - Destroy-on-success was unsafe: the script exited 0 for empty/no-archive lists too, so the README's &&-gated loop could remove a list that produced no backup artifact. Give the no-file outcomes a distinct exit code 3 ("nothing to archive" — not an error). The destroy loop now removes only on exit 0 (archive written) + a successful member dump, and leaves exit-3 lists intact unless ALLOW_NO_ARCHIVE=1. Safe by default: an unexpected "no archive" can no longer authorise a delete. - Publish wasn't crash-durable: the handoff fsynced file contents but not the parent directory, so the rename's directory entry could be lost to a crash after success was reported and the list removed. fsync the destination directory after os.replace(), before returning success. Verified with stubbed mailman-web: NONE/EMPTY -> exit 3 (no file), FILE -> exit 0 (0600, content intact, source temp unlinked), bad address -> exit 2. sh -n + shellcheck clean, embedded python compiles. --- README.md | 56 ++++++++++++++++++++++++++++++++++----------------- mm3-hk-export | 23 +++++++++++++++++---- 2 files changed, 56 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index c12600c..e849b44 100644 --- a/README.md +++ b/README.md @@ -46,18 +46,23 @@ mm3-hk-export announce@lists.example.com /root/mm3/announce.mbox mm3-hk-export --gzip announce@lists.example.com /root/mm3/announce.mbox.gz ``` -## Outcomes +## Outcomes and exit codes -The script never writes a partial or empty file, and exits `0` on all three: +The script never writes a partial or empty file. The **exit code** tells a +caller exactly what happened, most importantly whether a backup was produced: -| Situation | HyperKitty state | Result | -|---|---|---| -| Archive has messages | row with N emails | writes the mbox, prints `wrote N messages to …` | -| Archived but empty | row, 0 emails | no file, prints an "empty archive" notice | -| No archive at all | no `MailingList` row | no file; consults core `archive_policy` to say whether the list is `never`-archived or simply has nothing archived yet | +| Situation | HyperKitty state | Result | Exit | +|---|---|---|:--:| +| Archive has messages | row with N emails | writes the mbox, prints `wrote N messages to …` | `0` | +| Archived but empty | row, 0 emails | no file, prints an "empty archive" notice | `3` | +| No archive at all | no `MailingList` row | no file; consults core `archive_policy` to say whether the list is `never`-archived or simply has nothing archived yet | `3` | +| Usage / bad address | n/a | usage message | `2` | +| Genuine error (e.g. `mailman-web` failing) | n/a | underlying output on stderr | `1` | -A genuine error (e.g. `mailman-web` failing) exits non-zero with the underlying -output on stderr. +Exit **`3`** ("nothing to archive") is deliberately distinct from `0` ("archive +written"). A genuinely empty or never-archived list is a normal, expected +outcome, but it is not a successful backup, so a backup-and-destroy caller +can refuse to delete a list whose archive was never captured (see below). ## In a backup-and-destroy loop @@ -65,19 +70,32 @@ output on stderr. mkdir -p /root/mm3 for list in list1 list2; do addr="$list@domain.tld" - # Only destroy a list once BOTH the archive export and the member dump - # succeed. mm3-hk-export exits 0 (writing no file) for a list with no - # archive, so &&-gating still lets genuinely-empty lists through — it only - # blocks remove on a real failure (mailman-web error, disk full, bad mv). - if mm3-hk-export "$addr" "/root/mm3/$list.mbox" \ - && mailman-wrapper members "$addr" > "/root/mm3/$list.members"; then - mailman-wrapper remove "$addr" - else - echo "mm3-hk-export: backup FAILED for $addr — leaving list intact" >&2 - fi + + mm3-hk-export "$addr" "/root/mm3/$list.mbox" + case $? in + 0) ;; # archive written; safe to proceed + 3) # nothing to archive (empty / no archive): + # no backup file exists, so by default do + # not destroy. Override only if you mean it. + if [ "${ALLOW_NO_ARCHIVE:-0}" != 1 ]; then + echo "skip: $addr has no archive to back up; not removing" >&2 + continue + fi ;; + *) echo "skip: $addr export failed; not removing" >&2; continue ;; + esac + + mailman-wrapper members "$addr" > "/root/mm3/$list.members" \ + || { echo "skip: $addr member dump failed; not removing" >&2; continue; } + + mailman-wrapper remove "$addr" done ``` +The loop destroys a list only when its archive was actually written +(exit `0`) and its member roster was dumped. A list with nothing to archive +(exit `3`) is left intact unless you opt in with `ALLOW_NO_ARCHIVE=1`, so a +transient or unexpected "no archive" can never silently authorise a delete. + Export before `remove`, and never `remove` on a failed backup. (HyperKitty's archive lives in a separate database and is not deleted by core's `remove`, but a failed export must still halt the destroy step.) diff --git a/mm3-hk-export b/mm3-hk-export index f40f23e..c1f2721 100755 --- a/mm3-hk-export +++ b/mm3-hk-export @@ -6,9 +6,11 @@ # Output is byte-for-byte what the web "Download" button produces (same # Email.as_bytes() code path). With -z/--gzip the stream is gzip-compressed, # matching the web download's .mbox.gz exactly (and ".gz" is appended to the -# output name if absent). If the list has no archived messages — including the -# case where HyperKitty never created a MailingList row for it — nothing is -# written and the script exits 0 with a notice. +# output name if absent). If the list has no archived messages (including the +# case where HyperKitty never created a MailingList row for it), nothing is +# written and the script exits 3, a distinct "nothing to archive" status rather +# than an error. A backup-and-destroy caller can then tell "backed up" from +# "nothing was backed up" and refuse to delete an uncaptured list. # # Runs the export through Debian's mailman-web wrapper, which executes the # Django site as the correct web user. The mbox is created by that @@ -128,9 +130,14 @@ case "$status" in echo "mm3-hk-export: $addr has no HyperKitty archive (archiving disabled, or nothing ever archived); nothing written." >&2 ;; esac + # Nothing archived: exit 3 (distinct from 0=archive-written) so a + # backup-and-destroy caller can refuse to delete a list whose archive + # was never captured. Not an error, just a normal recoverable outcome. + exit 3 ;; EMPTY) - echo "mm3-hk-export: $addr has a HyperKitty archive but 0 messages — nothing written." >&2 + echo "mm3-hk-export: $addr has a HyperKitty archive but 0 messages; nothing written." >&2 + exit 3 ;; FILE) path=$(printf '%s' "$line" | cut -d'|' -f4) @@ -163,6 +170,14 @@ try: os.fsync(w.fileno()) os.replace(dtmp, out) # atomic same-dir rename, root-owned, 0600 dtmp = None # consumed by the rename + # fsync the containing directory so the new directory entry is crash-durable + # before we report success: a caller that then destroys the list must not be + # able to lose the backup to a crash that drops the unsynced rename. + dirfd = os.open(os.path.dirname(out) or ".", os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(dirfd) + finally: + os.close(dirfd) except BaseException: if dtmp is not None: try: os.unlink(dtmp)