The SHA-256 double-hash and TOCTOU defenses (O_NOFOLLOW/O_NONBLOCK, S_ISREG, fsync, in-process durable mkdir) guarded against a hostile same-uid process racing the /var/tmp handoff — a threat that does not match this deployment (a small trusted collective, no hostile local users), and the cost showed up as a 10-15s silent gap after the progress bar (two full passes over the archive).
Replace the ~60-line python3 publish block with plain shell: mkdir -p, mktemp in the destination dir, cp, atomic mv -f, rm the source. The two-stage handoff and atomic rename stay (the web user cannot write into a root-owned destination, and the rename keeps $out from being partial). The exporter no longer hashes either. README updated to match.
The export loop can run for a while on large lists. It already knows the
message count up front, so draw a braille spinner + progress bar (frames
and blue borrowed from warez) from inside the Python loop. Written to
stderr so it never pollutes the captured MM3HK| stdout, gated on
isatty() so cron/piped runs stay silent, throttled to ~10 fps, cleared
in finally. Progress bytes never touch the mbox, so its SHA-256 is
unaffected.
Drop the backup-and-destroy headline framing (the script only ever
exports; removal is the caller's opt-in) and cut the ELI5 hand-holding
aimed at someone who isn't the admin of this box. Collapse exit codes to
a table, demote removal to a short exit-code-gated snippet. No code change.
Review found three robustness gaps:
- set -e + command substitution let a mailman-web that exits 3 abort the
script with exit 3, colliding with the reserved "nothing archived" code
so a caller could treat a failed export as empty. Wrap the invocation in
`... ) || { ...; exit 1; }` so every exporter failure normalizes to 1
and can never be read as 3.
- Crash durability missed a newly-created output directory: only the leaf
dir was fsynced after the rename, not the parent holding that new dir
entry. Create the output tree in-process and fsync each new level into
its parent before reporting success.
- A same-uid process could swap the /var/tmp temp for a FIFO; an
O_RDONLY open without O_NONBLOCK blocks forever before the regular-file
check, hanging the job instead of failing closed. Open with
O_NOFOLLOW|O_NONBLOCK, then fstat and reject anything non-regular.
Verified: mailman-web exit 3 -> script exit 1 (NONE still 3, garbage 1);
FIFO source -> rejected in <5s, no hang; publish into a missing nested
dir creates it durably and writes 0600; existing-dir path unregressed.
The O_NOFOLLOW + S_ISREG handoff blocks a
symlink swap but still accepts a *regular-file* swap — a process holding
the web uid can replace/truncate the /var/tmp temp with different content
before root copies it, and root would publish it and exit 0, which the
destroy loop reads as authorization to remove the list.
The exporter now hashes the bytes it wrote (read back through its own
held fd — the original inode, immune to any name swap) and returns the
SHA-256 on the FILE status line. Root copies the source while hashing and
refuses to publish unless the digest matches; a swap/truncation/partial
write yields a mismatch, a non-zero exit, and no removal.
This does NOT defend a fully compromised mailman-web process (it controls
the source DB the archive is built from) — that is inherent and now
documented as out of scope. It closes same-uid swaps, partial writes, and
accidental corruption while staying in the existing architecture.
Verified: held-fd hash equals an independent by-name read (gzip + plain);
correct digest publishes (0600, content intact, source unlinked); wrong
digest, post-hash swap, and empty digest are all rejected with nothing
published; end-to-end NONE/EMPTY still exit 3, bad digest exits 1.
Two further hardening findings:
- Destroy-on-success was unsafe: the script exited 0 for empty/no-archive
lists too, so the README's &&-gated loop could remove a list that
produced no backup artifact. Give the no-file outcomes a distinct exit
code 3 ("nothing to archive" — not an error). The destroy loop now
removes only on exit 0 (archive written) + a successful member dump,
and leaves exit-3 lists intact unless ALLOW_NO_ARCHIVE=1. Safe by
default: an unexpected "no archive" can no longer authorise a delete.
- Publish wasn't crash-durable: the handoff fsynced file contents but not
the parent directory, so the rename's directory entry could be lost to
a crash after success was reported and the list removed. fsync the
destination directory after os.replace(), before returning success.
Verified with stubbed mailman-web: NONE/EMPTY -> exit 3 (no file),
FILE -> exit 0 (0600, content intact, source temp unlinked), bad
address -> exit 2. sh -n + shellcheck clean, embedded python compiles.
The prior handoff checked the /var/tmp
temp file by name (-L/-f) and then reopened it by name with cp: a
process holding the web uid could swap the path for a symlink in that
window and make root copy an arbitrary root-readable file into $out.
Replace the check-then-reopen with a single root-side python3 step that
opens the source ONCE with O_NOFOLLOW (a swapped symlink makes the open
fail), verifies it is a regular file via the open fd, copies from that
fd into a fresh root-owned mkstemp temp in the destination dir, fsyncs,
and atomically os.replace()s it into place. The path is never
re-resolved by name, so the TOCTOU is closed rather than narrowed.
Paths are passed as argv (heredoc single-quoted), so nothing is
interpolated into the snippet. Verified: regular file publishes 0600
with src removed and no leftover temp; symlink source is rejected
(ELOOP) with dest untouched; existing backups are atomically replaced.
Address three hardening findings:
- Gate the README backup-and-destroy loop so `mailman-wrapper remove`
only runs after both the export and member dump succeed; a failed
backup now leaves the list intact instead of falling through to an
irreversible delete. Empty/no-archive lists still pass (exit 0 by
design), so only real failures block removal.
- Replace the curl/REST archive_policy lookup with `mailman-wrapper
shell`, reading archive_policy straight from core's DB via
getUtility(IListManager). Eliminates REST credential handling
entirely (no secret in argv / /proc), drops the curl dependency, and
removes the @->. list-id conversion.
- Make the final publish a safe privilege handoff: reject non-regular /
symlinked temp files, copy the web-user mkstemp output into a
root-owned temp file on the destination filesystem, chmod 600, then
atomically rename into place. Fixes inherited web-user ownership and
the cross-filesystem mv that could leave a partial or clobbered
backup.
Scriptable equivalent of HyperKitty's web Download button for
backup-and-destroy loops. Runs the export through Debian's mailman-web
wrapper (HyperKitty lives in the web DB, separate from core), reproducing
the web view's Email.as_bytes() output.
- Handles the three archive states without writing partial/empty files:
populated (writes mbox), row-but-empty, and no row at all.
- For a missing row, best-effort core REST lookup distinguishes
archive_policy=never from 'nothing archived yet'.
- -z/--gzip matches the web .mbox.gz (deterministic header, mtime=0).
- Privilege-safe: web user writes a 0600 mkstemp file, root moves it.
- Address validated before interpolation into the embedded Python.
