In designing ezmlm, Dan J. Bernstein has used the unix philosophy of small component programs with limited and well defined functions. Requests for specific functions can then be met by the addition of new programs.
Thanks to the program execution mechanism Dan built into qmail, it is easy to execute several small programs per delivery in a defined sequence. It is also very easy to add shell scripts for further customization.
Dan J. Bernstein has written ezmlm in C. It is written for speed and reliability even in the face of power loss and NFS. These features are augmented to a large extent by the ruggedness of the qmail (also by Dan) delivery mechanism (see qmail-command(8)).
ezmlm uses some routines and techniques that still are not frequently seen in many mailing list managers. For example, subscriber E-mail addresses are stored in a hash so that searches require reading only, at most, 2% of the E-mail addresses. ezmlm has a optional message archive, where messages are stored 100 per directory, again to allow more efficient storage and retrieval. Important files are written under a new name and, only when safely written, moved in place, to assure that crashes do not leave the list in an undefined state.
In addition, ezmlm has a number of new inventions. One of these is bounce detection, which generates an automatic warning containing information identifying the messages which have bounced, followed by a probe message to the E-mail addresses for which mail has bounced. If the probe bounces, the address is unsubscribed. Thus, the system won't remove E-mail addresses due to temporary bounces: it takes 12 days after the first bounce before a warning is sent, and another 12 days of bounces after the warning bounce before the probe message is set.
Another Dan J. Bernstein invention is the use of cryptographic cookies based on a timestamp, address, and action. These are used to assure that the user sending a request to subscribe or unsubscribe really controls the target address. It is also used to prevent forgery of warning or probe messages to make it exceedingly difficult to subvert the bounce detection mechanism to unsubscribe another user.
See qmail(7), qmail-local(8), qmail-command(8), envelopes(5), and dot-qmail(5). Briefly, qmail having resolved the delivery address delivers it via the .qmail file that most completely matches the address. This file may be a link to another file, as is the case in ezmlm lists. qmail then delivers the message according to successive lines in this file forwarding it to an address, storing it, or piping it to a program. In the latter case, the program is expected to exit 0 leading delivery to proceed to the next line in the .qmail file, or 99 leading to success without delivery to succeeding lines. An exit code of 100 is a permanent error leading to an error message to the SENDER. An exit code of 111 is used for temporary errors, leading to re-delivery until successful or until the queue lifetime of the message has been exceeded.
Delivery granularity is the .qmail file and re-deliveries start at the top. Thus, if the message fails temporarily at a later line, the delivery according to an earlier line will be repeated. Similarly, qmail may have made deliveries successfully according to most of the .qmail file and then fail permanently. The SENDER is informed that the delivery failed, but not about at which point.
ezmlm takes advantage of these basic mechanisms to build a fast, efficient, and very configurable mailing list manager from a set of small independent programs.
See ezmlm(5) and the man pages for the different programs (listed in ezmlm(5)).
See ezmlm(5).
Messages to the list are delivered to a .qmail file, usually ~/.qmail-listname which is linked to DIR/editor. Here, the message is first delivered to ezmlm-reject(1) which can reject messages based on subject line contents, MIME content-type, and message body length. It also by default rejects all messages that do not have the list address in the ``To:'' or ``Cc:'' header. This eliminates most bulk spam. If the list is set up for restrictions based on envelope SENDER, the next delivery is to one or more instances of ezmlm-issubn(1). If the messages passed this check, it is usually delivered to ezmlm-send(1) for distribution. If the list is message moderated, it is instead delivered to ezmlm-store(1) which queues the message and sends out a moderation request. ezmlm-gate(1) is used by some other setups. It will for message moderated lists invoke ezmlm-send(1) directly if the message is from a specific set of SENDERs, and in other cases ezmlm-store(1) to send the message out for moderation.
If the list is configured for digests, DIR/editor also contains an ezmlm-tstdig(1) line followed by an ezmlm-get(1) line. If ezmlm-tstdig(1) determines that the criteria are met for digest generation, it exits with an exit code of 0, causing the ezmlm-get(1) line to be executed leading to a digest mailing. Otherwise, ezmlm-tstdig(1) exits 99, resulting in the remainder of the DIR/editor file to be ignored too long. The digest is not related to the message being delivered, but the delivery is used to trigger execution of the relevant programs.
In addition, DIR/editor contains a number of house-keeping functions. These are invocations of ezmlm-warn(1) to send out bounce warnings and and (if the list is moderated) ezmlm-clean(1) to clean the moderation queue of messages that have been ignored. Again, these functions are not related to the specific message delivered, but the delivery itself is used as a convenient ``trigger'' for processing.
Replies to moderation requests are channeled to DIR/moderator. This file contains an invocation of ezmlm-moderate(1) which invokes ezmlm-send(1) for accepted messages and sends out a rejection notice for rejected messages. It also sends error messages if the message is not found or already accepted/rejected contrary to the moderation message. Thus, if you accept a message already accepted, no error message is sent. ezmlm-clean(1) is also invoked from DIR/moderator for house keeping.
Administrative requests for both list and digest lists are captured by ~/.qmail-listname-default linked to DIR/manager. Here they are delivered first to ezmlm-get(1) which processed archive retrieval requests, exiting 99 after successful completion which causes the rest of the delivery lines to be ignored. If the request is not for ezmlm-get(1) it rapidly exits 0. This leads to invocation of ezmlm-manage(1) which handles subscriber database functions, help messages, and (if configured) editing of DIR/text/ files. Again, ezmlm-warn(1) lines are included for bounce directory processing.
If configured, an ezmlm-request(1) line is present. This program constructs
valid ezmlm requests from command in the subject lines of messages sent
to listname-request@host and exits 99. These requests are mailed
and will then return to be processed by one of the other programs.
Bounces to the list are handled by DIR/bouncer. For the digest list this is DIR/digest/bouncer. The two were combined in previous versions, which is still supported. As this leads to problems with list names ending in ``digest'', the functions are separate with lists set up or edited with ezmlm-idx>=0.32. The bounce is first delivery is to ezmlm-weed(1) which removes delivery delay notification and other junk. The second to ezmlm-return(1) which analyzes valid bounces storing the information in DIR/bounce/ for the list and DIR/digest/bounce/ for the digest. This is the information that ezmlm-warn(1) (invoked from DIR/editor and DIR/manager) uses and processes for automatic bounce handling. ezmlm-return(1) will also unsubscribe a subscriber from whom a probe message has bounced.
These are processed by DIR/owner and delivered to DIR/mailbox by default. It is better to put the real owner address in this location. This can be done manually, via editing of ezmlmrc(5), or via the ezmlm-make(1) -5 switch. Again, some house-keeping functions are also executed.
ezmlm subscriber E-mail addresses are stored within DIR/subscribers/ as a hashed set of 53 files. The hash calculated from the address determines which of the 53 files and address is stored in. Thus, to find out if an address is a subscriber, ezmlm has to read at most about 2% of the E-mail addresses. The hash function insures that E-mail addresses are reasonably evenly distributed among the 53 files.
Addresses in the files in DIR/subscribers/ are stored as strings starting with ``T'', followed by the address, followed by a zero byte. This is the same format as taken by qmail-queue(8) on file descriptor 1. Thus, subscriber lists can be directly copied to qmail without any further processing.
With ezmlm-idx>=0.32 you can use an SQL server for the subscriber databases. Please see the SQL section ( ezmlm support for SQL datbases).
rfc822 states that the host part of an address is case insensitive, but that
case of the local part should be respected and the interpretation of it is the
prerogative of the machine where the mailbox exists.
Thus, ezmlm preserves the case of the local part,
but converts the host part to lower case. ezmlm proper also bases the hash on
the case of the local part, so that USER@host and
user@host are not
(usually) stored in the same file.
Locally, deliveries are most often case insensitive, i.e. mail to
USER@host and
user@host
are delivered to the same mail box. A consequence of this is that
many users use E-mail addresses with different case interchangeably.
The problem is that when USER@host is subscribed,
ezmlm will not find that address in response
to an unsubscribe request from
user@host. This is even more problematic
when E-mail addresses have been added by hand to e.g. moderator lists.
ezmlm-idx>=0.22 changes address storage to make comparisons case insensitive
and store E-mail addresses based on the hash of the all lower case address. Case is
maintained for the local part. Thus, if USER@host is subscribed, mail
is set to USER@host,
but user@host is recognized as a subscriber
and an unsubscribe request from
user@host will remove USER@host from
the subscriber list.
To maintain backwards compatibility with old subscriber lists, a second
lookup is made for partially upper case E-mail addresses in some cases. This will
find USER@host subscribed with a case sensitive hash as well.
If may be useful to move all old mixed case E-mail addresses
to the ``new'' positions.
Without this, USER@host subscribed with the old system will be
able to unsubscribe as USER@host, but not as user@host.
After the
repositioning, s/he will be successfully able to use any case in an
unsubscribe request, e.g. UsEr@host. To do this:
% ezmlm-list DIR | grep -G '[A-Z]' > tmp.tmp
% xargs ezmlm-sub DIR < tmp.tmp
This works, because subscribing an address, even if it already exists, will assure that it is stored with a case insensitive hash. On some systems, the grep ``-G'' switch need/should not be used.
This mode of operation is automatically set up if you specify the
ezmlm-make(1) ``-u'' switch. Since there may be some addresses that should be
allowed to post, but are not subscribers of list or list-digest,
ezmlm-make(1) sets
up an additional address database in DIR/allow/.
Use ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) to manipulate
these addresses. If the list is configured for remote administration (see
How remote administration works),
you can add/remove addresses from the DIR/allow/ database by
mailing list-allow-subscribe@listhost
and list-allow-unsubscribe@listhost, respectively. Other commands
that access subscriber databases work in the same manner.
To similarly restrict archive access, use the ezmlm-make(1) ``-g'' switch.
Since SENDER is under the control of a potential attacker, it is not secure to use tests of SENDER for anything important. However, when replies are always sent to SENDER (such as for archive access), a check of SENDER can prevent the sending of information to E-mail addresses not in the database.
To test sender, use the program ezmlm-issubn(1). It will return 0 (true for the shell, success for qmail deliveries) if SENDER is in at least one of a set of subscriber databases. If not, it will return 99 (false for the shell: success, but skip remainder of .qmail file for qmail deliveries). The basedirs of the subscriber lists (i.e. the directories in which the ``subscriber'' dirs are located) are given as arguments. ezmlm-issubn(1) can take any number of arguments.
Thus, to permit an action if SENDER is a subscriber to the list in any of DIR/, DIR/digest/, or DIR/allow/ and exit silently, put the following into the relevant .qmail file:
|/usr/local/bin/ezmlm/ezmlm-issubn DIR DIR/digest DIR/allow [...]
|/path/action_program
Restricting your list to posts from your subscribers is as easy as that. If your ezmlm binaries are in a different directory, you may have to modify the ezmlm-issubn(1) path.
ezmlm-issubn(1) has a ``-n'' switch which ``negates/reverses'' the exit code. To do an action if SENDER is NOT a subscriber of any of the lists:
|/usr/local/bin/ezmlm-ezmlm-issubn -n DIR/deny [dir2 ...]
|/path/other_program
To automatically configure the list with a blacklist address database in
DIR/deny, use the ezmlm-make(1) ``-k'' switch. If the list
is configured for remote administration (see
How remote administration works) and if you
are a remote administrator,
you can manipulate the ``deny'' database remotely by sending mail
to list-deny-subscribe-user=userhost@listhost, etc.
Each ezmlm list has it's own ``key'' created by ezmlm-make at setup time. This key is stored in DIR/key, and you can improve it by adding garbage of your own to it. However, changing the key will make all outstanding cookies invalid, so this should be done when the list is established.
When ezmlm receives an action request, such as ``subscribe'', it constructs a cookie as a function of:
Dan has based these cookies on cryptographic functions that make it very unlikely that a change in any part of the cookie or the items will result in a valid combination. Thus, it is virtually impossible to forge a request even for someone who has a number of valid requests to analyze. Since the algorithm ezmlm uses is available, the security rests on the key (and the correctness of the algorithm). Anyone who knows the key for your lists can easily construct valid requests.
As ezmlm-make(1) doesn't use a truly random process to generate the key, it is theoretically possible that someone with sufficient knowledge about your system can guess your key. In practice, this is very unlikely, and the safety of the system is orders of magnitude higher than that of other mechanisms that you may rely on in your list management and mail transport (exclusive of strong encryption, such as PGP).
Moderator E-mail addresses are stored just like ezmlm subscriber addresses, in a set of up to 53 files within the subscribers subdirectory of the list's basedir/. For subscribers, the basedir/ is the list directory itself, i.e. DIR/. For moderators, the default is DIR/mod/, which can be overridden by placing a basedir name (starting with a ``/'') in DIR/modsub, DIR/remote, or DIR/modpost for subscription moderation, remote administration, and message moderation, respectively. This permits the use of one moderator database for multiple lists. Note: Subscription moderators and remote administrators are always the same addresses. If both DIR/modsub and DIR/remote contain paths, only the DIR/modsub path is used.
Subscription moderation is a simple extension of the ezmlm subscribe mechanism. Once the user has confirmed the subscribe request, a new request is constructed with a different action code. This is sent out to the moderator(s). When a moderator replies with a valid request and cookie combination, the user is subscribed. The user is then also welcomed to the list. Other moderators won't know that the request has already been approved. If other moderators reply to the request, no notification of the duplicate action is sent to the subscriber of the duplicate action. Ezmlm knows that this is a repeat request since the target address is already a subscriber.
The moderators are not informed about the result, unless there was an error (subscribing a target that is already a subscriber is not considered an error). This cuts down the number of messages a moderator receives. Any list moderator knows (or should know) the qmail/ezmlm/unix paradigm: if you're not told otherwise, your command was carried out successfully. This may be counterintuitive to those used to some other operating systems, but in our experience it doesn't take long to get used to the reliability and efficiency of U*ix/qmail/ezmlm.
Subscription moderation is enabled by creating DIR/modsub and adding the subscription moderator to DIR/mod/:
% ezmlm-sub DIR/mod moderator@host
The term ``remote administration'' is used to denote the ability of a
list administrator by E-mail
to add or remove any E-mail address from the subscriber
list without the cooperation of the user. Normally, when
user@userhost sends a message to
list-subscribe-other=otherhost@listhost to
subscribe other@otherhost, the confirmation request goes
to other@otherhost. However, if remote administration is enabled and
user@userhost is a moderator, a confirmation request (with a different
action code) is sent back to user@userhost instead. The reply
from the administrator is suppressed in the welcome message sent to the new
subscriber (other@otherhost). This protects the identity of the remote
administrator.
Remote administration is enabled by creating DIR/remote and adding the remote administrator E-mail address(es) to DIR/mod/:
% ezmlm-sub DIR/mod remoteadm@host
Remote administrators can add and remove addresses to the digest list, the ``allow'' list (user aliases for lists using SENDER restrictions on posting and archive access), and if used the ``deny'' list containing addresses that are denied posting rights to the list. The latter is easy to circumvent and intended to block errant mail robots, rather than human users.
ezmlm-store(1), invoked in DIR/editor, receives messages for message moderated lists. If DIR/modpost does not exist, ezmlm-store(1) just calls ezmlm-send(1) and the message is posted to the list as if it were not moderated. If DIR/modpost exists, ezmlm-store(1) places the message in DIR/mod/pending/. It also sends a moderation request to all the moderators. Included with this request is a copy of the message. The ``From:'' and ``Reply-To:'' E-mail addresses contain codes for ``reject'' and ``accept'', together with a unique message name (derived from the message timestamp and process id) and a cookie based on these items. When a moderator replies, ezmlm-moderate(1) is invoked via DIR/moderator. ezmlm-moderate(1) validates the request, and if the request is valid and the message is found in DIR/mod/pending/, it carries out the requested action.
If the request is ``reject'' the post is returned to SENDER with an explanation and an optional moderator comment. If the request is ``accept'' the message is posted to the list via ezmlm-send(1). As the request is processed, a stub for the message is created in DIR/mod/rejected/ or DIR/mod/accepted/ for ``reject'' and ``accept'' requests, respectively.
If a valid reply is received but the message is no longer in DIR/mod/pending/, ezmlm-moderate(1) looks for the corresponding stub in DIR/mod/rejected/ and DIR/mod/accepted/. If the stub is found and the fate of the message was the one dictated by the new request, no further action is taken. If, however, no stub is found or the request and the actual message fate do not match, a notification is sent to the moderator. This scheme was chosen to impart a maximum of information with a minimum of messages. Also, it is the least demoralizing setup for multiple moderator lists, where it is important not to notify subsequent moderators that their work was in vain since the action of the first responding moderator has already resulted in processing of the message.
If a message is not ``rejected'' or ``accepted'' it remains in DIR/mod/pending/ until it times out. Cleanup of both messages and stubs is accomplished by ezmlm-clean(1) which is invoked through both DIR/editor and DIR/moderator for message moderated lists. ezmlm-clean(1) looks at the timestamp used to generate the message/stub name. If it is older than 120 hours (configurable in a range of 24-240 hours, by placing the value in DIR/modtime) it is removed. Unless suppressed with the ezmlm-clean(1) ``-R'' switch, the SENDER of the message is notified.
By default, the E-mail addresses of message moderators are stored as a subscriber list with a basedir of DIR/mod/. This can be changed to any other basedir by placing the name of that directory with a leading ``/'' in DIR/modpost. Although the default basedirs for message moderation and subscription moderation/remote administration are the same, both the functions and actors are entirely independent.
The structure of the ezmlm list archive is described in the ezmlm(5) manual page. Basically, the message is stored in DIR/archive/n/m, where ``n'' is the message number divided by 100 and ``m'' the remainder (2 digits). The first message is stored in DIR/archive/0/01.
The ezmlm-idx(1) adds the option (default) of a message index to ezmlm. The ``From:'' line, the subject, the author's E-mail address and name and the time of receipt are logged for each message as it is received. The subject is ``normalized'' by concatenating split lines and removing reply-indicators such as ``Re:''. A hash of the normalized subject with all white space removed is also stored. The hash for any message within a thread is almost always the same and is used together with the order of receipt to connect a set of messages into a ``thread''. A hash is needed due to the inconsistent handling by MUAs of white space in rfc2047-encoded subject headers.
The message index is stored as DIR/archive/n/index, where ``n'' is the message number mod 100. Thus, the directory DIR/archive/52/ stores messages 5200 through 5299 and the file ``index'' which contains the index for those messages.
The message index can be retrieved with the -index command (see ezmlm-get(1)). You can also retrieve a range of messages, a specific thread, or generate a message digest (see ezmlm-get(1)). Each of these commands can be disabled or restricted as desired by the list owner.
The ezmlm-idx(1) can be used at any time to either reconstruct an existing index or create one an index for an existing message archive. without one.
A ezmlm thread is just a message number-ordered set of messages with identical ``normalized'' subject entries. This is a very reliable method for threading messages. It does not rely on any variably present ``In-Reply-To:'' or ``References:'' headers. If the subject changes, the continuation becomes a separate thread very close to the original thread in a digest. ezmlm uses this mechanism to return message sets threaded and with a thread and author index, unless specifically told not to do so with the ``n'' format specifier. Naturally, lists set up without a message index (using the ezmlm-make ``-I'' switch) do not maintain thread information.
A ``digest'' is just an ordered collection of messages from a list, usually sent out regularly depending on the time and traffic volume since the last digest. Digest subscribers thus can read messages as ``threads'' once daily, rather than receiving a constant trickle of messages.
As a major change in ezmlm-idx-0.30, the digest is no longer a totally separate ezmlm-list, but a part of the main list. This has security advantages, makes setup and administration easier, saves space, and allows a consistent way for subscribers of both ``list'' and ``list-digest'' to retrieve missed messages from a single archive.
The digest of the list ``list'' is always called ``list-digest''. To set up
a list with a digest, simply use the ezmlm-make(1) ``-d'' switch. You subscribe
to and unsubscribe from a digest the same way as for the main list, except that
the request is sent to e.g. list-digest-subscribe@host rather than
to list-subscribe@host.
Any option such as remote admin or subscription moderation that is active for the list applies also to the digest list. Any restrictions in posts or archive retrieval set up for the list, automatically accept both subscribers of the main list and of the digest list.
The changes in ezmlm-idx>=0.30 allow all programs to service both list and list-digest functions. All digest-specific files are stored in DIR/digest/. Digest list subscriber addresses in DIR/digest/subscribers/ and digest list bounce information in DIR/digest/bounce/. Text files are shared between list and digest. To get the local part of the list or list-digest name in a context sensitive manner, use ``<#l#>'' (lower case ``L'') in the text file.
In order to generate digest, the list needs to be archived and indexed (both default). You can retrieve sets of messages from the message archive. Such sets are always returned to the SENDER of the request. ``Digests'' are a special form of such a set/request. First, there are no restrictions on the number of messages that can be in a digest (which is balanced by the requirement for a ``digest code'' that needs to be specified in order to create a digest based on a mailed request). Second, special files (DIR/digissue and DIR/dignum) keep track of the digest issue and the message number, amount, and time when the last digest was created. Thus, the system is adapted to make it easy to create the regular collections of messages commonly referred to as ``digests''.
Digest can be generated in several different ways:
ezmlm-get can be invoked on the command line, or via a script from e.g. crond(8):
% ezmlm-get DIR
This is the default and does not require and additional setup. It works well with most lists. The only possible advantage is for very low traffic lists and for lists where it is important that a digest be sent out at a specific time (as DIR/editor digests are triggered only when messages are received).
In DIR/editor, ezmlm-get(1) needs to be combined with ezmlm-tstdig(1) so that digests are generated only if certain criteria are met (in this case, more than 30 messages, 64 kbytes of message body or 48 hours since the latest digest). Add these lines after the ezmlm-send line in DIR/editor:
|/usr/local/bin/ezmlm/ezmlm-tstdig -t48 -m30 -k64 DIR || exit 99
|/usr/local/bin/ezmlm/ezmlm-get diglist@host DIR || exit 0
% ezmlm-make -d DIR dot local host [code]
This is useful only if you want digests at specific times, and you do not have access to crond(8) on the list host. ezmlm-get(1) is in it's normal place in DIR/manager before ezmlm-manage(1), but a digest code is specified in the ezmlm-get(1) command line. To trigger digests requires a regular trigger messages generated from e.g. crond(8) (see below), but this can be done from _any_ host, not only the list host. ezmlm-make(1) sets up ezmlm-get(1) this way if a digest ``code'' is given as the 5th ezmlm-make(1) command line argument. However, you need to set up the trigger messages separately (see below):
% ezmlm-make DIR dot local host code
|/usr/local/bin/ezmlm/ezmlm-tstdig -t24 -m30 -k64 DIR || exit 99
The default setup in the ezmlmrc(5) file included in the distribution
is the DIR/editor triggered setup described above.
If you in addition use ezmlm-cron(1) or crond(8) directly to generate trigger
messages to list-dig.code@host, you can get regular digests (via
the trigger messages and DIR/manager),
with extra digest sent when traffic
is unusually high (via the ezmlm-tstdig/ezmlm-get limits set in
DIR/editor).
This works best when the time argument on the ezmlm-tstdig(1) command line
is the same as the trigger message interval, and the other ezmlm-tstdig(1)
parameters are set so that they are only rarely exceeded within the normal
digest interval.
ezmlm-tstdig(1) looks at DIR/num and DIR/dignum
to determine how many
messages and how much traffic (in terms of bytes of message body) has arrived
to the list since the latest
digest. It also determines how much time has passed since the last digest
was generated. If any of the
criteria specified by command line switches exists, ezmlm-tstdig(1) exits 0,
causing the invocation of the next line in the .qmail file. If not,
ezmlm-tstdig(1) exits 99 causing qmail to skip the rest of the .qmail
file. ezmlm-tstdig(1) looks at LOCAL to determine if
it is invoked in the command line, in DIR/editor,
or in DIR/manager. In
the latter two cases, ezmlm-tstdig(1) verifies that the list local address
is correct. If invoked in DIR/manager, ezmlm-tstdig(1) exits 0 for all
action requests except list-dig,
so that is does not interfere with the
normal functions of ezmlm-get(1) and ezmlm-manage(1). ezmlm-tstdig(1) uses
DIR/tstdig as a flag to avoid problems caused by starting
the program when another copy is already running.
ezmlm-make(1) automatically configures ezmlm-tstdig(1) with the parameters ``-t48 -m30 -k64'', which can be overridden with the ``-3'' switch.
ezmlm uses the concept of sublists. Sublists are regular ezmlm lists, except that they only accept messages from their parent list, which is placed in the file DIR/sublist.
sublists are used to split the load of a large mailing list among several
hosts. All you need to do to set up a local sublist of e.g.
the qmail@list.cr.yp.to list is to create a ezmlm list,
and put ``qmail@list.cr.yp.to'' into DIR/sublist of you list,
and subscribe the sublist to the main qmail list. Now anyone can subscribe
to your local list which handles its own bounces, subscribe requests, etc.
The load on the main list is only the single message to your local list.
Sublists will not add their own mailing list header and they will not add a subject prefix. Normally, sublists will use their own message number, rather than that used by the main list. With ezmlm-idx>=0.23, sublists that are not archived and not indexed, will instead use the main list message number. This way, bounce messages from the sublist can refer the subscriber to the main list archive. This is not done for indexed/archived sublists for security reasons (an attacker could overwrite messages in the sublist archive).
With ezmlm-idx>=0.31, there is support for using ezmlm as a sublist of a mailing list run by another mailing list manager. To set this up, set up a normal ezmlm sublist, then edit DIR/editor so that the ezmlm-send line contains the command line option ``-h X-Listprocessor-Version:'' (before DIR). As the header text, you need to use a header that the main list manager adds to messages. Now your sublist will accept only messages from the main list requiring that they come from that list and contain the header specified.
ezmlm-idx>=0.313 also has added protection against the malicious subscription of the ezmlm list to mailing lists run by other list managers. If the ezmlm-reject(1) line in DIR/editor has ``-h'' and ``DIR'' on it, ezmlm-reject(1) will read DIR/headerreject and reject messages that have any header specified in that file. See the ezmlm-reject(1) man page for suitable headers.
Often you create a local sublist of a list that you do not control. Local users know to subscribe to your local list. However, occasionally, you want to run your own list as a main list and a series of sublists per geographic site, or split onto several hosts if the list is too large to be handled by a single computer. You may also want to split the load of a ``well known'' list host that is getting overwhelmed with traffic. ezmlm supports sublists, but here the fact that the user has to interact with the correct sublist is a problem. What if the user doesn't remember which sublist s/he is subscribed to? What if you change the name of a sublist host or move a sublist to a different host?
ezmlm-idx&-0.32 adds ezmlm-split(1), which allows sublisting transparent
to the user. This program is invoked before ezmlm-manage(1) in
DIR/manager. If it detects a subscribe or unsubscribe
command, it will forward the command to the appropriate sublist based
on a ``split file'' DIR/split. This file contains entries, one per
line, of the format:
domain:lo:hi:sublistname@sublisthost
edu:::othersub@otherhost
:1:26:third@thirdhost
id.wustl.edu
it would be ``edu.wustl''. The domain is considered to match if the characters
in the split file match. It is advisable to use only the last part of the
domain for compatibility with the SQL version
version (see section
ezmlm support for SQL datbases).
Thus, any address *@*.domain with a hash between ``lo'' and ``hi''
inclusive
would match the first line and be forwarded to sublistname@sublisthost.
*@*.edu (independent of hash)
would match the second line and be forwarded to
othersub@otherhost. Of remaining requests, a request for any
target address with a hash between 1 and 26 would be forwarded to
the sublist third@thirdhost. Remaining requests would be passed
on to the local list.
The domain is useful for ``geographic'' splitting, and the hash for load splitting (within a domain). The user interacts only with the main list, and does not need to know from which sublist s/he is serviced.
ezmlm-idx sublists use the message number of the main list message if they are not indexed. This allows sublists to in bounce messages refer the subscriber to the main list archive. Use ezmlm-make(1) in conjunction with ezmlmsubrc(5) to set up the sublists. See man pages for further details.
Since the addresses are stored locally, the system is very fast and robust, but it is difficult to add new sublists. ezmlm-split(1) -D supports parsing addresses on stdin and splitting them to stdout (see man page). Thus, if you divide the domain of some sublist(s) onto a net set of sublists, you can use ezmlm-list(1) to collect the addresses, ezmlm-split -D with the new split file to split them, then after clearing the local subscriber databases use ezmlm-sub(1) to add the correct addresses to each new sublist. The section on SQL support describes an alternative way of managing sublists (see section ezmlm support for SQL datbases).
Rfc2142 (standards track) says that for each mailing list list@host,
there MUST be an administrative address list-request@host. This
is not the default for ezmlm, but can be added with
ezmlm-make(1) ``-q'', which adds a ezmlm-request(1) line before the
ezmlm-manage(1) line in DIR/manager. This address is used to
manage commands in the ``Subject:'' line, by translating them into
appropriate ezmlm command messages.
When migrating from other mailing list managers which use this method to issue list commands, configuring ezmlm to respond to such commands may be useful. In addition, some software manufacturers sell MUAs and mail gateways that are unable to correctly transport rfc822-compliant Internet mail with certain characters in the local part of the address.
ezmlm-request(1) services the list-request@host address per
rfc2142 (standards track). It is usually invoked in DIR/manager
before ezmlm-get(1) and
ezmlm-manage(1). It ignores all requests that are not for the list-request
address. For requests to the list-request@host address,
ezmlm-request(1) parses
the ``Subject:'' line. If a ezmlm command address starting with the contents
of DIR/outlocal
(e.g. list-get45) is on the command line,
ezmlm-request(1) generates the
corresponding full ezmlm request message. If the subject does not start with
the contents of DIR/outlocal, ezmlm-request(1) prefixes the line
with the contents of DIR/outlocal, thereby
building a complete ezmlm command. If a host name is specified, it must
match the contents of DIR/outhost, i.e. ezmlm-request(1) in this
function will only generate command messages for the local list.
Thus, a subject of ``subscribe'' to list-request@host
will be auto-magically
rewritten as a message to list-subscribe-userlocal=userhost@host.
Similarly, any ezmlm command
or ``Reply-To:'' address can be pasted into the subject field and sent to
list-request@host.
ezmlm-request(1) does not validate the command name, but
invalid commands result in a ``help'' message in reply via ezmlm-manage(1). This
allows ezmlm-request(1) to also service custom commands,
like list-faq@host that you may have created for your list.
If the ``Subject:'' is empty or does not start with a letter, ezmlm-request(1) will attempt to interpret the first message body line that starts with a letter in the first position.
When ezmlm-request(1) has successfully processed a ''request'' command, it exits 99 to skip the rest of DIR/manager.
To set up a list to include ezmlm-request processing, use the ezmlm-make(1) ``-q'' switch. The default is to not do this.
ezmlm-idx>=0.23 allows alternate names for all user commands. This can be
used to e.g. make a message to list-remove@host to result in
an ``unsubscribe'' action. This may help migration from other mailing list
managers and in non-English environments. The use of aliases allows ezmlm
to respond to new command names, while always responding correctly to the
standard commands. If ezmlm-request(1) is used it will automatically
be able to deal with any commands you set up for the list, within ezmlm or as
separate programs.
See
Multiple language support
on how to set up command aliases.
The qmail/ezmlm mechanism makes it very easy to add your own commands. You can add them to DIR/manager, but this requires great care in terms of ordering and exit codes. Easier is to set them up separately with a .qmail-list-command file.
Let's assume you want to allow anyone to determine how many subscribers
are subscribed to your list with the command list-count@host.
Just create a program to do the work:
#!/bin/sh
DTLINE='Delivered-To: list-count@host processor'
grep "$DTLINE" > /dev/null &&
{ echo "This message is looping"; exit 100; }
{
echo "$DTLINE"
cat <<EOF
From: list-help@host
To: $SENDER
Subject: list@host subscriber count
Current number of subscribers:
EOF
ezmlm-list ~/DIR | wc -l
} | /var/qmail/qmail-inject -f list-return- "$SENDER"
exit 0
A user with shell access can always manipulate subscriber lists with ezmlm-sub(1), ezmlm-unsub(1), and ezmlm-list(1) for the lists s/he owns.
Sometimes a remote administrator requires a list of subscriber E-mail addresses. At the same time, the list should be kept out of the hands of spammers and all unauthorized entities. By default, ezmlm does not allow remote subscriber list retrieval. You can enable the ``-list'' command for remote retrieval of a subscriber list by using the ezmlm-make(1) ``-l'' switch or by adding the ``-l'' switch to the ezmlm-manage(1) line in DIR/manager. With this switch, ezmlm will permit retrieval of a subscriber list, but only to remote administrators. Subscribers cannot get the list membership, and any outsider would have to be able to read a remote administrator's mail to get the list. Note: This option is not functional unless the list is configured for remote administration, i.e. the ezmlm-make(1) ``-rl'' switches need to both be used.
The list returned is unsorted for efficiency reasons. You can easily sort it or use your mail reader to find a specific entry. The number of subscribers is shown at the bottom of the list. To get the number of subscribers from the command line, use:
% ezmlm-list DIR | wc -l
The same conditions that enable remote administrators to retrieve a
subscriber list (see
subscriber_list
) also enable the remote admin
to retrieve the subscription log, i.e. the log of changes made to the subscriber
list. The command is list-log@host. The entries are of the
form ``date timestamp dir event address comment''. ``dir'' is ``+'' for addition
of an address, ``-'' for removal, ``event'' is empty for
normal (un)subscribe ``manual'' for changes made with ezmlm-(un)sub,
and ``probe'' for removals via bounce handling. ``address'' is the subscription
address, and ``comment'' is empty or the subscribers ``From:'' line. The
log can be used to look at recent additions/removals and to try to track down
a subscriber address from e.g. the name on the ``From:'' line. The log is
written on a best-effort basis. In contrast to the subscriber database,
entries in the log may be lost at a system crash.
The remote administrator can do a case-insensitive search through the log with
the command list-log.xxx@host, where ``xxx'' is any sequence of
letters/numbers that must occur on a line in order for that line to be
included in the reply. A ``_'' is a wild card and should be used for special
characters as well. Thus, to search for any entry with a host
name of host* mail list-log._host and to find entries
for ``Keith John...'' etc, use list-log.keith_john.
For SQL-enabled lists, this command searches the ``list_slog'' table.
If a list is set up with the ezmlm-make(1) ``-n'' switch, or if the ``-e'' switch is added to the ezmlm-manage(1) line in DIR/manager, ezmlm allows remote administrators to edit the text files that make up most of the ezmlm responses. Of course, this will work only if remote administration is enabled for the list. Replies are sent only if the target address is a remote administrator. Thus, ezmlm does not rely on SENDER (easily forged) but on the notion that only the recipient receives the message. This is a reasonable assumption for remote administrators that receive mail on the local system.
With this switch, ezmlm replies to the -edit command with a list of the
files in DIR/text/.
Only files where editing seems reasonable are included in
the list. The remote administrator can edit any file in DIR/text/
by sending e-mail containing the new text to
-edit.file where ``file''
is the name of the file replaced (edited). The file must exist and
the name consist of only lower case letters and '-'. Any '-' (hyphen)
must be
substituted by a '_' (underscore). For remote administrator convenience, the
substitution has been made in the list of files sent in reply to the
-edit command.
In reply to this command, ezmlm sends a message with the file and editing instructions. A ``cookie'' based on the date, file name, and contents of the file is added to the ``Reply-To:'' address. The cookie becomes invalid as soon as the file has been changed, or after 27 hours, whichever is shorter. Also, the cookie cannot be used to edit any other file, even if the other file has exactly the same contents. If you sent an edit request, and decide not to edit the file, you can simply delete the message.
To apply standard changes to all your text files it is easier to edit ~/.ezmlmrc. To reset the list's text files back to their default contents (as specified by ezmlmrc(5)), use the ezmlm-make(1) ``-e'' switch together with any other switches used to set up the list.
First of all, it is against a number of RFCs to modify the ``Subject:'' header of messages. However, it is frequently requested by users who have seen it on other list managers. Second, it is many times worse to have a prefix that changes from message to message, such as a prefix with the message number. However, a number of lists, especially in Japan, use this feature and in its absence these lists might be unable to take advantage of ezmlm. Thus, while we recommend against using a prefix, ezmlm-idx supports it.
To add a subject prefix, just put the text into DIR/prefix. The only format that makes any sense is ``list:'' or ``(list)'' or such.
The message number prefix is activated by putting e.g. ``(list-#)'' into DIR/prefix. ``#'' is replaced by the message number. ezmlm refuses to make more drastic changes in the subject of a message. As a consequence, the message number prefix is added only when the subject does not already contain a prefix. Thus, replies will have the message number of the original message. Doing anything else and still supporting rfc2047-encoded subjects in the archive threading (much more important) would require decoding the subject, removing/editing the prefix, and re-encoding the subject. This is far too invasive.
The entire thread can always be retrieved by sending a message to
list-thread-x where ``x'' is the message number in the prefix of
any message in the thread.
Ezmlm messages are sent with an envelope sender (``Return-Path'') that directs bounces to DIR/bouncer and also via ``VERP'' contain information about the intended recipient. Thus, programs run from DIR/bouncer know the subscriber for whom the message bounced. ezmlm-weed(1) is used to weed out delivery delay notification and other junk. For others ezmlm-return(1) decides if the address is a subscriber. If so, it saves the first bounce message and a list of bounced-message numbers. ezmlm-warn(1) executed from e.g. DIR/editor goes through these bounce files. If it finds any that are older than 1,000,000 seconds (about 11.6 days) it sends a warning message to the subscriber. If this warning message bounces, ezmlm-return(1) sets up a "warning flag" for the subscriber. If ezmlm-warn(1) finds a warning flag older than 11.6 days, it sends a "probe" to the subscriber. If ezmlm-return(1) receives a bounced probe, the subscriber is automatically unsubscribed.
The ezmlm-warn(1) ``-t'' switch can be used to change the time-out (in days). The ezmlm-warn(1) ``-d'' switch causes processing of ``list-digest'' bounces rather than ``list'' bounces. ezmlm-weed(1) and ezmlm-return(1) can handle bounces for either list.
ezmlm-warn(1) also removes any files in the bounce directory that are older than 3 times the bounce time-out.
ezmlm-warn(1) is normally run from DIR/editor. This can take quite a lot of resources, if there are a large number of bouncing addresses (>>1000) on a busy list, since by default all bounces are stored in a single directory and ezmlm-warn(1) examines all of them with each invocation. ezmlm-idx->=0.32 changes bounce handling to improve performance for large lists. Bounces are stored in subdirectories of DIR/bounce/d/, one per 10,000 seconds. The corresponding address hashes are stored in 16 subdirectories of DIR/bounce/h/. Instead of looking at all bounces, ezmlm-warn(1) processes only the bounces in DIR/bounce/d/ subdirectories that are ``due''. In addition, ezmlm-warn(1) uses DIR/bounce/lastd as a simple lockout, to assure that it will do work only at most once every 5.5 hours. (Times are scaled to the ezmlm-warn(1) ``-t'' argument if used.) Together, these changes assure that bounce handling will scale well in the default configuration, even for very large lists.
The -info and -faq commands simply reply with the contents of the DIR/text/info and DIR/text/faq files. Edit these files directly or remotely (see How to remotely edit dir/text files). The DIR/text/info file should start with a single line that is meaningful as is and describes the list. This will be used in later versions to allow automatic assembly of the global ``list-of-lists'' (see How to set up a global list address like majordomo@host or listserv@host).
Sometimes, it is desirable to have a host- or user-wide address that can list available mailing lists.
ezmlm-request(1) can be used to set up a global address, such as
ezmlm@host which allows the user to see and interact with a
number of different mailing lists. This is especially useful when your
users are used to other mailing list managers,
such as ``majordomo'' or ``listproc''. ezmlm-request(1) is set up to
answer requests to the address (see
How to set up a global list address like majordomo@host or listserv@host).
There, it interprets the first line of the message body as a command. It
will reply directly to ``lists'' and ``which'' commands. All other commands
will be used to construct messages to the respective lists. Where other
mailing list managers use synonyms of ezmlm commands, ezmlm-request(1)
recognizes these and translates them to the corresponding ezmlm commands.
ezmlm-request(1) will build commands also of unrecognized commands. Thus,
if you create new commands for a list, ezmlm-request(1) will automatically
support them.
If the user does not specify the complete list address, ezmlm-request(1) will attempt to complete the name. See the ezmlm-reject(1) man page for more info.
If you are a user and have crond(8) access, if you do not need to get digests at specific times, or if you are a system administrator setting up lists, there is no reason for you to use ezmlm-cron(1). If you are a system administrator not allowing users crond(8) access or a user that needs digests at specific times, but without crond(8) access, read on.
ezmlm-cron(1) is a very restrictive interface to crond(8).
ezmlm-cron(1) can be used
to create digest trigger messages. If a list is set up with a digest
code (see ezmlm-make(1) and ezmlm-get(1)) ezmlm will generate a digest
from the list joe-sos@host sent to
to subscribers of joe-sos-digest@dighost when receiving a message
to joe-sos-dig-code@host where ``code'' is the
digest code. ezmlm-cron(1) can be
used to generate such messages at regular intervals.
The file ezcronrc is set up by the sysadmin and controls what trigger
messages specific users may set up via ezmlm-cron(1).
Usually, the ezcronrc of that use will have an entry like ``user:user-:host:10'' allowing ``user'' to create trigger messages for up to 10 lists with names starting with ``user-'' and on the host ``host''.
To list the ezcronrc line controlling your use of ezmlm-cron(1):
% ezmlm-cron -c
% ezmlm-cron -l
list@host
every morning at 0230:
% ezmlm-cron -t 02:30 -i24 list@host code
To delete the entry above:
% ezmlm-cron -d list@host
% ezmlm-cron -t 16:16 -i24 list@host code
ezmlm lists allow almost infinite customization. The component build, together with the qmail delivery mechanism makes it possible to create any variant of list function imaginable. However, this complexity makes it somewhat daunting to the average user wanting to set up a mailing list. ezmlm-make(1) allows automated list setup, while permitting a large amount of configurability.
At first glance, ezmlm-make(1) has many complicated options. However, these can be applied iteratively through the ezmlm-make(1) edit mechanism. Also, they are intended to be relatively complete so that execution of ezmlm-make(1) by e.g. a GUI can be used to safely set up and edit any list.
ezmlm-make(1) reads its command line arguments and switches, then creates the list directory. If the ``-e'' edit or ``-+'' sticky edit switches are not specified, ezmlm-make(1) will fail if the directory already exists. The directory argument must be an absolute path starting with a slash. The dot-qmail file argument, if specified, must also be absolute.
ezmlm-make(1) next reads ezmlmrc(5) located in the /etc/ directory with a default install. If not found, the file in the ezmlm binary directory will be used. The second ezmlm-make command line argument specify the root name of the .qmail files. If the ezmlm-make(1) ``-c'' switch is used, ezmlm-make(1) will look in that directory for a .ezmlmrc file and use it instead. If this file does not exist, ezmlm-make(1) will print a warning and use the previously discussed ezmlmrc(5) files in the same order. You can also use ``-C ezmlmrc.alt'' to use ezmlmrc.alt as the ezmlmrc(5) file. Again, ezmlm-make(1) will fall back to the others with a warning, if the specified ezmlmrc(5) file is not found.
When not run in ``-e edit'' or ``-+'' sticky edit modes, ezmlm-make(1) first creates the list directory. It also as the last step of its action creates DIR/key containing the key used for cookie generation.
The ezmlmrc(5) file consists of a number of file names relative to the list directory, followed by conditional flags (see ezmlm-make(1) and ezmlmrc(5) for details). If all the conditional flags (controlled by the corresponding command line switches) are true, the lines that follow are entered into the named file. There are also tags to erase files. Tags in the format <#X#> (where ``X'' is any number, except ``1'' and ``2'') are replaced by the corresponding ezmlm-make(1) switch argument. The ezmlm-make(1) command line arguments and the ezmlm binary path can be similarly substituted into the text. Thus, ezmlmrc(5) controls (within reason) the entire operation of ezmlm-make(1). ezmlmrc(5) is also set up so that no messages or file containing list state information are lost. Therefore, ezmlm-make(1) can be used to safely edit existing lists. The only caveat is that the list state is undefined while editing is in progress. Thus, it is advisable to prevent mail delivery by setting the ``sticky'' bit on the user's home directory while editing lists.
ezmlm-make(1) will create the file DIR/config. This files saves all the flags that were set at the last execution of ezmlm-make, as well as all the switch and command line arguments. When editing a list, only ``DIR'' and the non-default letter switches need to be specified. Other command line arguments and the ``digit switch'' arguments are read from DIR/config. To remove a digit switch, simply use it with two single quotes as the argument.
You can also easily determine how a list was set up by looking at DIR/config.
Rather than restrict you to a single E-mail address (user@host), qmail
in the default setup gives you control over an infinite number of addresses
user-*@host. Of course, you (normally) have no way of controlling
elsewhere@host since that could lead to overlap between
users' ``e-mail address space''. As a consequence, all you mailing lists
have to be named user-xx@host where ``user'' is your user name
and ``xx'' is anything. You cannot create e.g. mylist@host, only
user-mylist@host. To create the list user-list@host do:
% ezmlm-make ~/list ~/.qmail-list user-list host
There are two way to create lists with names not starting with your user name: First, qmail can be set up so that you control a virtual domain (see below). Second, the system administrator can set up lists with arbitrary names within the ~alias/ directory.
If you use qmail>=1.02 and ezmlm-idx>=0.32, lists under virtual domains work just like other lists and require no adjustments. You can choose any local name for the list and the ezmlm-make(1) argument ``local'' is that name; ``host'' is the name of the virtual domain.
All non-default switches, ezmlm-issubn(1) setups, etc, can be made standard for new lists by customizing the ezmlm-make(1) configuration file named ``ezmlmrc''. A default ezmlmrc(5) is installed in the ezmlm binary directory. If installed, a system-wide customized ezmlmrc file in /etc/ezmlmrc (or symlinked from there) overrides this. Installing a ~/.ezmlmrc file in a user dotdir and using the ezmlm-make(1) ``-c'' switch allows further per user customization (see Customizing ezmlm-make operation).