initial public commit

.flake8

@@ -0,0 +1,9 @@
+[flake8]
+# E266 too many leading '#' for block comment [ I like marking disabled code blocks with '### ' ]
+# E402 module level import not at top of file [ usually on purpose. might use individual overrides instead? ]
+# E701 multiple statements on one line [ still quite readable in short forms ]
+# E713 test for membership should be ‘not in’ [ disagree: want `not a in x` ]
+# E714 test for object identity should be 'is not' [ disagree: want `not a is x` ]
+# W503 line break before binary operator [ gotta pick one way ]
+extend-ignore = E266,E402,E701,E713,E714,W503
+max-line-length = 140

.gitignore

@@ -0,0 +1 @@
+venv

LICENSE

@@ -0,0 +1,19 @@
+Copyright © Universität Stuttgart
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,124 @@
+# Postfix (cluster) queue manager
+
+CLI tool to manage postfix queues (optionally across multiple hosts through ssh).
+
+It uses [trio](https://trio.readthedocs.io/en/stable/) for asynchronous operations (i.e. parallel handling of many nodes in a cluster).
+
+It also has proper readline support with history of commands.
+
+## Workflow
+
+Start with `health` to get a feeling for the queue state.
+
+Show the list of deferred mails with `list`, or show all mails from a certain user (maybe `health` showed a suspicious user) with `list from bob@example.com`.
+
+Note that `list` (contrary to `list-verbose`) only shows the last (remaining) recipient of a mail (and how many targets are remaining).
+
+Build up filters to match only certain mails - e.g. start with `select from MAILER-DAEMON` to search for bounces, and then limit those to no-reply targets with `select to ~reply`.  Use `list` or `list-verbose` to see which mails remain, and what the problem is.
+
+Use `show mailid` to show headers of suspicious emails, or `show -hb mailid` to show the body too if really necessary.
+
+Use `hold` to hold all currently selected mails (e.g. spam/...), and then delete them later with `delete`. Or use `delete mailid` to delete specific mails directly.
+
+Use `expire` to return mails to the sender (because you think the error from `list-verbose` is permanent, even if postfix doesn't know it yet).
+
+`delete` and `expire` operations prompt for confirmation after showing affected mails.
+
+## List of commands
+
+There is an interactive help command.
+
+```
+# help
+clear                          Clear all filters
+clear <arg>                    Clear filter with given index (zero based; first filter is at index 0)
+current                        Show current filters
+delete                         Delete selected mails from the hold queue
+delete <arg>                   Delete mails with given IDs from the queues
+delete-all                     Delete selected mails from all queues
+exit                           Exit pqm (or press Ctrl-D on empty prompt)
+expire                         Expire mails and flush them (return error to sender).
+expire <arg>                   Expire mails (return error to sender) with given IDs and flush them.
+expire-noflush                 Expire mails (return error to sender on next delivery attempt).
+expire-noflush <arg>           Expire mails (return error to sender on next delivery attempt) with given IDs.
+expire-release                 Expire mails and flush them (return error to sender); release if mails are on hold.
+expire-release <arg>           Expire mails (return error to sender) with given IDs and flush them; release if mails are on hold.
+flush                          Flush (deferred) selected mails in the queue: attempt to deliver them
+flush <arg>                    Flush (deferred) mails with given IDs: attempt to deliver them
+flush-all                      Flush the queue: attempt to deliver all queued mail.
+health                         Show generic stats for queues and overly active address
+help                           Show all available commands
+help <arg>                     Show detailed help for a command
+hold                           Put selected mails on hold
+hold <arg>                     Put mails with given IDs on hold
+list [<arg>]                   List all mails (single line per mail); if no filter is configured hide active and hold queue
+list-all [<arg>]               List all mails (including active + hold queue, single line per mail)
+list-verbose [<arg>]           List all mails (verbose output); if no filter is configured hide active and hold queue
+list-verbose-all [<arg>]       List all mails (including active + hold queue, verbose output)
+pop                            Remove last filter
+release                        Release mail that was put "on hold".
+release <arg>                  Release mails with given IDs that were put "on hold".
+requeue                        Requeue mail (move to "maildrop"; get reprocessed)
+requeue <arg>                  Requeue mail with given IDs (move to "maildrop"; get reprocessed)
+select <arg>                   Add filter for mails that other commands work on
+show <arg>                     usage: show [-b] [-e] [-h] ID [ID ...]
+```
+
+Available filter expressions are show in the help for `select`:
+
+```
+# help select
+>>>> select <args>
+
+Add filter for mails that other commands work on
+
+Filter syntax:
+- `from bob@example.com`, `from alice@example.com,bob@example.com,@example.net`, `from "alice@example.com" "bob@example.com"`
+  Multiple address can be given to match any of them; either unquoted and comma separated, or quoted and whitespace
+  separated. In the unquoted case the (comma separated) pattern must not include any whitespace.
+  An address starting with `@` only checks the domain name and ignores the local part.
+- `from ~regex.*@example.(com|net)`, `from ~"regex.*@example.(com|net)" "other@example.com`
+  To use regular expressions either put `~` in front of an unquoted pattern (need to repeat for each regex in a
+  comma separated pattern list) or before the quotes of a quoted pattern.
+- `to ...` and `address ...` (matching both to and from) work the same as `from ...`
+- `queue hold,deferred`
+  Comma separated list of queues a mail must be in. For single queue names `queue` can be omitted, e.g. `hold`.
+- `$expr1 and $expr2`
+- `$expr1 or $expr2`  (`a and b or c` is parsed as `a and (b or c)`)
+- `($expr)`
+- `not $expr`
+```
+
+The select expressions use the data provided by `postqueue -j` - which is why it doesn't support filtering by subject/other headers.
+
+## Installation
+
+Dependencies:
+- python3, probably >= 3.10
+- [trio](https://trio.readthedocs.io/en/stable/)
+- [pyparsing](https://github.com/pyparsing/pyparsing/)
+- [PyYAML](https://github.com/yaml/pyyaml)
+
+On Debian: `apt install python3-trio python3-pyparsing python3-yaml`
+
+Put the pqm script into your path and make it executable.
+
+## Configuration
+
+Configuration is optional; by default it tries to use `~/.config/pqm.yaml` and `/etc/pqm.yaml` (picking the first one that exists).
+
+Also see `pqm.example.yaml` in this repository.
+
+### Source (hosts load load mails from)
+
+By default it will use the local queue; it probably needs to run as root for that.
+
+Use `remote-sources` in the config to use remote hosts instead ("cluster" mode).  Each remote source has a unique alias that is used in the `pqm` output (by default the first DNS label in the hostname).
+
+`pqm` uses `ssh -oBatchMode=yes root@remotesource ...` to execute postfix tools on the remote nodes; this means you'll probably want a public-key based authentication to the postfix nodes, and the server keys must already be trusted.
+
+You'll also want to make sure the ssh login is fast; `UseDNS no` (should be the default) on the server and ed25519 keys should help with that.
+
+## Queue IDs
+
+Each mail in the postfix queues has an ID; when using the local source `pqm` will use this ID directly.  With remote sources it will prepend `alias/` to the ID.

check-lints.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+set -e
+
+cd "$(dirname "$(readlink "$0")")"
+
+rc=0
+
+run() {
+    # remember last failure
+    if "$@"; then :; else rc=$?; fi
+}
+
+export PYTHONDONTWRITEBYTECODE=1
+
+if [ ! -d venv ]; then
+    virtualenv -p python3 venv
+    ./venv/bin/pip install trio pyparsing PyYAML mypy flake8 trio-typing types-PyYAML
+fi
+
+run ./venv/bin/flake8 pqm
+run ./venv/bin/mypy pqm
+
+exit $rc

pqm.example.yaml

@@ -0,0 +1,14 @@
+---
+# when remote-sources are not given / are empty: defaults to localhost (without ssh)
+# each remote source is used as ssh target (with `root@` prefix); custom ssh
+# option (separate port, ...) should be configured through ssh config files.
+remote-sources:
+- mx1.example.com
+- mx2.example.com
+- mxout1.example.com
+- mxout2.example.com
+- mxintern1.example.com
+- mxintern2.example.com
+# prefix with "alias:" if you don't want the first DNS label to be the alias
+- orgmx1:mx1.example.org
+- orgmx2:mx2.example.org