Exporting Mailman3's mailing-list archives through HyperKitty on the CLI

Find a file

veg 4069f26769 Normalize exporter failures, durable mkdir, reject swapped FIFO Review found three robustness gaps: - set -e + command substitution let a mailman-web that exits 3 abort the script with exit 3, colliding with the reserved "nothing archived" code so a caller could treat a failed export as empty. Wrap the invocation in `... ) \|\| { ...; exit 1; }` so every exporter failure normalizes to 1 and can never be read as 3. - Crash durability missed a newly-created output directory: only the leaf dir was fsynced after the rename, not the parent holding that new dir entry. Create the output tree in-process and fsync each new level into its parent before reporting success. - A same-uid process could swap the /var/tmp temp for a FIFO; an O_RDONLY open without O_NONBLOCK blocks forever before the regular-file check, hanging the job instead of failing closed. Open with O_NOFOLLOW\|O_NONBLOCK, then fstat and reject anything non-regular. Verified: mailman-web exit 3 -> script exit 1 (NONE still 3, garbage 1); FIFO source -> rejected in <5s, no hang; publish into a missing nested dir creates it durably and writes 0600; existing-dir path unregressed.	2026-05-31 12:28:55 +00:00
mm3-hk-export	Normalize exporter failures, durable mkdir, reject swapped FIFO	2026-05-31 12:28:55 +00:00
README.md	Verify export integrity with SHA-256 before publishing	2026-05-31 12:28:09 +00:00

veg 4069f26769 Normalize exporter failures, durable mkdir, reject swapped FIFO

Review found three robustness gaps:

- set -e + command substitution let a mailman-web that exits 3 abort the
  script with exit 3, colliding with the reserved "nothing archived" code
  so a caller could treat a failed export as empty. Wrap the invocation in
  `... ) || { ...; exit 1; }` so every exporter failure normalizes to 1
  and can never be read as 3.

- Crash durability missed a newly-created output directory: only the leaf
  dir was fsynced after the rename, not the parent holding that new dir
  entry. Create the output tree in-process and fsync each new level into
  its parent before reporting success.

- A same-uid process could swap the /var/tmp temp for a FIFO; an
  O_RDONLY open without O_NONBLOCK blocks forever before the regular-file
  check, hanging the job instead of failing closed. Open with
  O_NOFOLLOW|O_NONBLOCK, then fstat and reject anything non-regular.

Verified: mailman-web exit 3 -> script exit 1 (NONE still 3, garbage 1);
FIFO source -> rejected in <5s, no hang; publish into a missing nested
dir creates it durably and writes 0600; existing-dir path unregressed.

2026-05-31 12:28:55 +00:00

mm3-hk-export

Normalize exporter failures, durable mkdir, reject swapped FIFO

2026-05-31 12:28:55 +00:00

README.md

Verify export integrity with SHA-256 before publishing

2026-05-31 12:28:09 +00:00

README.md

mm3-hk-export

Export a private HyperKitty (Mailman 3) list archive to an mbox file from the command line: the scriptable equivalent of clicking the web UI's Download button, so it fits a backup-and-destroy loop without manual browser steps.

Why this exists

HyperKitty has no hyperkitty_export management command; the only built-in export is the web "Download" view (gated by the HYPERKITTY_MBOX_EXPORT setting). This script reproduces that view's exact output by calling the same Email.as_bytes() code path on each archived message, driven through Debian's mailman-web wrapper: no web login, no direct database reconstruction.

The archives live in the mailman-web / HyperKitty database, which is separate from Mailman core. That is why the export must go through mailman-web (not mailman / mailman-wrapper, whose shell only sees core).

Requirements

Debian-style Mailman 3 with the mailman-web wrapper on PATH (Django site at /usr/share/mailman3-web/, HyperKitty installed).
Run as root: mailman-web drops privileges to the web user to read the archive, and the script (as root) moves the finished file into place.
Optional, only to enrich the "not archived" notice: the mailman-wrapper core CLI (for a best-effort archive_policy lookup via mailman shell, reading core's DB directly, no REST, no credentials). Missing it just falls back to a generic message.

Usage

mm3-hk-export [-z|--gzip] <list@domain> [output]

output defaults to ./<localpart>.mbox (./<localpart>.mbox.gz with -z).
-z / --gzip gzip-compresses the stream, matching the web download's .mbox.gz byte-for-byte (deterministic header, mtime=0); .gz is appended to the output name if absent.

Examples:

mm3-hk-export announce@lists.example.com /root/mm3/announce.mbox
mm3-hk-export --gzip announce@lists.example.com /root/mm3/announce.mbox.gz

Outcomes and exit codes

The script never writes a partial or empty file. The exit code tells a caller exactly what happened, most importantly whether a backup was produced:

Situation	HyperKitty state	Result	Exit
Archive has messages	row with N emails	writes the mbox, prints `wrote N messages to …`	`0`
Archived but empty	row, 0 emails	no file, prints an "empty archive" notice	`3`
No archive at all	no `MailingList` row	no file; consults core `archive_policy` to say whether the list is `never`-archived or simply has nothing archived yet	`3`
Usage / bad address	n/a	usage message	`2`
Genuine error (e.g. `mailman-web` failing)	n/a	underlying output on stderr	`1`

Exit 3 ("nothing to archive") is deliberately distinct from 0 ("archive written"). A genuinely empty or never-archived list is a normal, expected outcome, but it is not a successful backup, so a backup-and-destroy caller can refuse to delete a list whose archive was never captured (see below).

In a backup-and-destroy loop

mkdir -p /root/mm3
for list in list1 list2; do
    addr="$list@domain.tld"

    mm3-hk-export "$addr" "/root/mm3/$list.mbox"
    case $? in
        0) ;;                       # archive written; safe to proceed
        3)                          # nothing to archive (empty / no archive):
                                    # no backup file exists, so by default do
                                    # not destroy. Override only if you mean it.
            if [ "${ALLOW_NO_ARCHIVE:-0}" != 1 ]; then
                echo "skip: $addr has no archive to back up; not removing" >&2
                continue
            fi ;;
        *)  echo "skip: $addr export failed; not removing" >&2; continue ;;
    esac

    mailman-wrapper members "$addr" > "/root/mm3/$list.members" \
        || { echo "skip: $addr member dump failed; not removing" >&2; continue; }

    mailman-wrapper remove "$addr"
done

The loop destroys a list only when its archive was actually written (exit 0) and its member roster was dumped. A list with nothing to archive (exit 3) is left intact unless you opt in with ALLOW_NO_ARCHIVE=1, so a transient or unexpected "no archive" can never silently authorise a delete.

Export before remove, and never remove on a failed backup. (HyperKitty's archive lives in a separate database and is not deleted by core's remove, but a failed export must still halt the destroy step.)

Caveats

Reconstruction, not the raw original. HyperKitty rebuilds each message from parsed fields: From, To, Subject, Date, Message-ID, In-Reply-To, body, and attachments. Headers like Cc, References, List-*, and custom X-* are not preserved. This is inherent to HyperKitty; the manual web download has the identical limitation.
Memory. Messages are streamed with a DB iterator and written incrementally, avoiding the web export's known blow-up on large lists.
Re-import. The output is valid mbox (proper From separators, From-mangled bodies) and can be re-imported with hyperkitty_import.
Handoff integrity. The archive is written by the privilege-dropped web user into /var/tmp, then published by root. Root opens that file with O_NOFOLLOW, copies it while computing a SHA-256, and refuses to publish unless the digest matches the one the exporter computed over the bytes it wrote, so a same-uid process that swaps, truncates, or corrupts the temp file between export and publish is caught (the run fails, nothing is published, and the destroy loop will not remove the list). A fully compromised mailman-web process is out of scope: it builds the archive from a database it already controls, so no file-level check downstream can vouch for the content it produced.