mm3-hk-export/README.md

mm3-hk-export

Export a private HyperKitty (Mailman 3) list archive to an mbox file from the command line: the scriptable equivalent of clicking the web UI's Download button, so it fits a backup-and-destroy loop without manual browser steps.

Why this exists

HyperKitty has no hyperkitty_export management command; the only built-in export is the web "Download" view (gated by the HYPERKITTY_MBOX_EXPORT setting). This script reproduces that view's exact output by calling the same Email.as_bytes() code path on each archived message, driven through Debian's mailman-web wrapper: no web login, no direct database reconstruction.

The archives live in the mailman-web / HyperKitty database, which is separate from Mailman core. That is why the export must go through mailman-web (not mailman / mailman-wrapper, whose shell only sees core).

Requirements

• Debian-style Mailman 3 with the mailman-web wrapper on PATH (Django site at /usr/share/mailman3-web/, HyperKitty installed).
• Run as root: mailman-web drops privileges to the web user to read the archive, and the script (as root) moves the finished file into place.
• Optional, only to enrich the "not archived" notice: the mailman-wrapper core CLI (for a best-effort archive_policy lookup via mailman shell, reading core's DB directly, no REST, no credentials). Missing it just falls back to a generic message.

Usage

mm3-hk-export [-z|--gzip] <list@domain> [output]

• output defaults to ./<localpart>.mbox (./<localpart>.mbox.gz with -z).
• -z / --gzip gzip-compresses the stream, matching the web download's .mbox.gz byte-for-byte (deterministic header, mtime=0); .gz is appended to the output name if absent.

Examples:

mm3-hk-export announce@lists.example.com /root/mm3/announce.mbox
mm3-hk-export --gzip announce@lists.example.com /root/mm3/announce.mbox.gz

Outcomes

The script never writes a partial or empty file, and exits 0 on all three:

Situation	HyperKitty state	Result
Archive has messages	row with N emails	writes the mbox, prints wrote N messages to …
Archived but empty	row, 0 emails	no file, prints an "empty archive" notice
No archive at all	no MailingList row	no file; consults core archive_policy to say whether the list is never-archived or simply has nothing archived yet

A genuine error (e.g. mailman-web failing) exits non-zero with the underlying output on stderr.

In a backup-and-destroy loop

mkdir -p /root/mm3
for list in list1 list2; do
    addr="$list@domain.tld"
    # Only destroy a list once BOTH the archive export and the member dump
    # succeed. mm3-hk-export exits 0 (writing no file) for a list with no
    # archive, so &&-gating still lets genuinely-empty lists through — it only
    # blocks remove on a real failure (mailman-web error, disk full, bad mv).
    if mm3-hk-export "$addr" "/root/mm3/$list.mbox" \
        && mailman-wrapper members "$addr" > "/root/mm3/$list.members"; then
        mailman-wrapper remove "$addr"
    else
        echo "mm3-hk-export: backup FAILED for $addr — leaving list intact" >&2
    fi
done

Export before remove, and never remove on a failed backup. (HyperKitty's archive lives in a separate database and is not deleted by core's remove, but a failed export must still halt the destroy step.)

Caveats

• Reconstruction, not the raw original. HyperKitty rebuilds each message from parsed fields: From, To, Subject, Date, Message-ID, In-Reply-To, body, and attachments. Headers like Cc, References, List-*, and custom X-* are not preserved. This is inherent to HyperKitty; the manual web download has the identical limitation.
• Memory. Messages are streamed with a DB iterator and written incrementally, avoiding the web export's known blow-up on large lists.
• Re-import. The output is valid mbox (proper From separators, From-mangled bodies) and can be re-imported with hyperkitty_import.