mm3-hk-export/README.md

mm3-hk-export

Export a private HyperKitty (Mailman 3) list archive to an mbox file from the command line: the scriptable equivalent of clicking the web UI's Download button, so it fits a backup-and-destroy loop without manual browser steps.

Why this exists

HyperKitty has no hyperkitty_export management command; the only built-in export is the web "Download" view (gated by the HYPERKITTY_MBOX_EXPORT setting). This script reproduces that view's exact output by calling the same Email.as_bytes() code path on each archived message, driven through Debian's mailman-web wrapper: no web login, no direct database reconstruction.

The archives live in the mailman-web / HyperKitty database, which is separate from Mailman core. That is why the export must go through mailman-web (not mailman / mailman-wrapper, whose shell only sees core).

Requirements

• Debian-style Mailman 3 with the mailman-web wrapper on PATH (Django site at /usr/share/mailman3-web/, HyperKitty installed).
• Run as root: mailman-web drops privileges to the web user to read the archive, and the script (as root) moves the finished file into place.
• Optional, only to enrich the "not archived" notice: the mailman-wrapper core CLI (for a best-effort archive_policy lookup via mailman shell, reading core's DB directly, no REST, no credentials). Missing it just falls back to a generic message.

Usage

mm3-hk-export [-z|--gzip] <list@domain> [output]

• output defaults to ./<localpart>.mbox (./<localpart>.mbox.gz with -z).
• -z / --gzip gzip-compresses the stream, matching the web download's .mbox.gz byte-for-byte (deterministic header, mtime=0); .gz is appended to the output name if absent.

Examples:

mm3-hk-export announce@lists.example.com /root/mm3/announce.mbox
mm3-hk-export --gzip announce@lists.example.com /root/mm3/announce.mbox.gz

Outcomes and exit codes

The script never writes a partial or empty file. The exit code tells a caller exactly what happened, most importantly whether a backup was produced:

Situation	HyperKitty state	Result	Exit
Archive has messages	row with N emails	writes the mbox, prints wrote N messages to …	0
Archived but empty	row, 0 emails	no file, prints an "empty archive" notice	3
No archive at all	no MailingList row	no file; consults core archive_policy to say whether the list is never-archived or simply has nothing archived yet	3
Usage / bad address	n/a	usage message	2
Genuine error (e.g. mailman-web failing)	n/a	underlying output on stderr	1

Exit 3 ("nothing to archive") is deliberately distinct from 0 ("archive written"). A genuinely empty or never-archived list is a normal, expected outcome, but it is not a successful backup, so a backup-and-destroy caller can refuse to delete a list whose archive was never captured (see below).

In a backup-and-destroy loop

mkdir -p /root/mm3
for list in list1 list2; do
    addr="$list@domain.tld"

    mm3-hk-export "$addr" "/root/mm3/$list.mbox"
    case $? in
        0) ;;                       # archive written; safe to proceed
        3)                          # nothing to archive (empty / no archive):
                                    # no backup file exists, so by default do
                                    # not destroy. Override only if you mean it.
            if [ "${ALLOW_NO_ARCHIVE:-0}" != 1 ]; then
                echo "skip: $addr has no archive to back up; not removing" >&2
                continue
            fi ;;
        *)  echo "skip: $addr export failed; not removing" >&2; continue ;;
    esac

    mailman-wrapper members "$addr" > "/root/mm3/$list.members" \
        || { echo "skip: $addr member dump failed; not removing" >&2; continue; }

    mailman-wrapper remove "$addr"
done

The loop destroys a list only when its archive was actually written (exit 0) and its member roster was dumped. A list with nothing to archive (exit 3) is left intact unless you opt in with ALLOW_NO_ARCHIVE=1, so a transient or unexpected "no archive" can never silently authorise a delete.

Export before remove, and never remove on a failed backup. (HyperKitty's archive lives in a separate database and is not deleted by core's remove, but a failed export must still halt the destroy step.)

Caveats

• Reconstruction, not the raw original. HyperKitty rebuilds each message from parsed fields: From, To, Subject, Date, Message-ID, In-Reply-To, body, and attachments. Headers like Cc, References, List-*, and custom X-* are not preserved. This is inherent to HyperKitty; the manual web download has the identical limitation.
• Memory. Messages are streamed with a DB iterator and written incrementally, avoiding the web export's known blow-up on large lists.
• Re-import. The output is valid mbox (proper From separators, From-mangled bodies) and can be re-imported with hyperkitty_import.