This chapter tells you how to deal with common problems and introduces you to the power and technical aspects of ezmlm/idx. If you want to customize your list, please use the man pages and FAQ included in the ezmlm and ezmlm-idx packages (see More information).
As the owner of an ezmlm list you will appreciate the high degree of speed and automation of ezmlm. Even with large numbers of subscribers, the parallelism of ezmlm/qmail results in rapid delivery of the message to the subscribers. This makes it easy to have discussions among list members. With slower list managers the discussion is often limited to a small set of users who send mail on the topic to each other, ``Cc:ing'' the message to the list. When the other subscribers receive the message, the discussion is way past this point
Features available only if ezmlm-idx is used are marked ``(*)''..
ezmlm -subscribe and -unsubscribe commands result in a handshake which essentially eliminates the possibility of people other than the user (and possibly remote administrators) adding or removing the user's address from the list.
For the list mailinglist@example.org,
mail to the mailinglist-help@example.org address returns a message
with all the relevant ezmlm commands for subscription and message archive
related functions. Archive access is simple and powerful.
Together these features make it very easy to set up and manage an ezmlm
mailing list. The rare messages to the list-owner address
(mailinglist-owner@example.org, in the example) are usually due to
subscribers who have subscribed under one address, and now use another.
Even they can (un)subscribe without owner intervention, but some
subscribers may be unable/unwilling to read the ezmlm help message and
follow the instructions therein.
In addition to the regular subscriber address databases for the list and list-digest, ezmlm may use up to three other address databases. One of these the ``allow'' database is housed in DIR/allow/
File names are shown in boldface. Directories end with a slash. ``DIR/'' is the list directory.. In case posts or access to the archive is restricted to subscribers, addresses present in DIR/allow/ are also accepted. If the subscriber
jonesj@softx.com
sometimes posts from john@example.net, you can just add the second
address to DIR/allow/ to permit this. The previous chapter describes
how to do this via the ``allow'' command extension (see
Adding subscriber aliases).
If you have shell access, you can simply:
% ezmlm-sub ~/listDIR/allow john@example.net
% ezmlm-list ~/listDIR/allow
Another address database is DIR/deny/. If the list was configured with the ezmlm-make(1) ``-k'' switch, posts from addresses in this database are rejected. This is easy to circumvent, but very handy if a subscriber creates a mail loop (ezmlm/qmail has excellent loop detection, but some subscriber vacation programs generate a new message to the list for every message received and ezmlm has no way of knowing that this is not a legitimate post). For such a subscriber address, you can just temporarily add the sending address to DIR/deny/, which is some cases is preferable to simply unsubscribing the user. If this database is configured, remote administrators can access it via the ``deny'' command extension (see Preventing an address from posting).
The final database is DIR/mod/ in which moderator and remote administrator addresses are stored. Remote administrators may themselves (un)subscribe any address. Remote administration (see Remote administration) is configured with the ezmlm-make(1) ``-r'' switch. The same addresses may also be subscription moderators. If configured (``-s'' switch) a user can subscribe only if one or more of the moderators approve. Messages to the list can also be moderated. This is configured with the ``-m'' switch. Again, the moderators are stored in DIR/mod/, although it is possible to configure the list so that message and subscription moderators are different sets of addresses (see More information).
To add a moderator/administrator:
ezmlm-sub ~/DIR/mod john@example.net
ezmlm lists are set up with a one line ezmlm-make(1) command. Existing lists can be modified using the ezmlm-make ``-e'' or ``-+'' (*) switches, without loss of archive, subscriber database, or other state information.
The operation of ezmlm-make is controlled by ezmlmrc(*). To customize ezmlm list setup on a host-wide or user (virtual domain)-wide basis, copy this file from the ezmlm binary directory to /etc/ezmlmrc of ~user/.ezmlmrc and edit it. (See More information.)
ezmlm offers virtually infinite possibilities for customization. The more common types of lists can be created and configured with a single ezmlm-make(1) command. For more complicated lists and a discussion of how ezmlm works, see More information).
Normal mailing lists are public, i.e. anyone can subscribe/unsubscribe, and access the list archive. Occasionally, you may want to set up a non-public list, where (un)subscribe actions are possible only from the command line or remotely by the administrator(s). This is controlled by the ezmlm-make(1) ``-P'' switch.
The digest is a collection of messages, produced about every two days. If more than 30 messages or 64 kbytes of ``message body'' have arrived since the latest digest, the digest is sent earlier. All these criteria can be changed (see More information). Also, lists can be set up to produce digests at a specific time each day, every other day, etc (see More information).
Some users do not get involved in discussions on the list but rather want to read messages on a specific topic. For these users, a digest with sorted messages is much preferable to receiving the messages one at a time. These users would subscribe to the digest list, rather than to the list itself.
To set up a digest list with the list, all you need to do is to
use the ezmlm-make ``-d'' switch. The digest of the list list@host
is always called list-digest@host. Thus for the list
mailinglist@example.org, the digest list would be
mailinglist-digest@example.org.
The digest can be sent in several formats. The default, MIME with removal of ``non-essential'' headers is virtually always the best. A ``format-specifier'' can be used with the ezmlm-get(1) ``-f'' switch to override this. ``v'' returns MIME with all headers, and ``r'' a rfc1153-like non-MIME format (see More information). You can use DIR/digheaders to specify the message headers included for archive excerpts when using the ``m'' (default) format.
ezmlm mailing lists are by default set up to keep an archive of all
the messages sent to the list. This archive is used to create digests.
Users can also access messages in the archive,
using the -get, -thread(*),
and -index(*) addresses (see above).
If the users
would like to receive messages in a non-default format, they may
do so by adding a format specifier(*) to the command address name.
Thus, to receive messages 45-65 from mailinglist@example.org in
the ``v'' (virgin - MIME with all headers) format, the user would send
a message to mailinglist-getv45-65@example.org(*).
ezmlm mailing lists can be set up with many restrictions, both for posts and archive access. Through customization of ezmlm, there are many variations. The more common ones are accessible through ezmlm-make(1) switches when the list is created or modified (see More information).
By default, ezmlm rejects all messages that have an empty subject or a subject consisting of a command name (such as ``subscribe'') only. The lists will also reject messages that do not have the list address in the ``To:'' or ``Cc:'' header(*). This removes the majority of spam without affecting normal posts. In addition, lists may be configured to reject messages below or above a certain size(*), measured on the message body. ezmlm can also reject messages containing certain MIME content types(*). It may make sense to reject messages consisting only of ``text/rtf'' on lists with users on multiple platforms, many of which would be unable to read the message. Similarly, restricting the maximal size of messages to e.g. 20000 bytes may save inexperienced users from quoting entire digests or sending large programs or such as an attachment. All these restrictions are handled by ezmlm-reject(1) in the DIR/editor file in the list directory (see More information).
To set up lists with subscription moderation so that each -subscribe request must be approved by a moderator, use the ezmlm-make(1) ``-s'' switch.
Lists set up with the ezmlm-make(1) ``-u'' switch allow posts only from subscribers of the list or list-digest. Use of the ``-m'' switch makes all posts moderated (see below). A combination of these switches (``-um'') has a special meaning: posts from subscribers go directly to the list, whereas posts from others are sent out for moderation. Also, messages that are ignored (neither accepted nor rejected) by the moderators, usually result in an informative error message to the sender after about 5 days. However, for ``-um'' lists, such posts are ignored. This gives the moderator the choice of accepting the message, rejecting it with the accompanying notification of the sender, and silently ignoring it (most appropriate for e.g. junkmail/spam).
Like normal subscribers, digest subscribers are allowed to post.
For some lists, you may not want non-subscribers to be able to read archived messages. This restriction can be placed on a list with the ezmlm-make(1) ``-g'' switch. Since ezmlm always sends archive excerpts to the sender of the request this ensures that only persons able to read mail to a subscriber address can receive archive excerpts. All such persons can already read the list traffic. The same is true for digest subscribers who also have access to the digest.
The DIR/allow/ directory holding the ``allow'' addresses can be useful to give archive access to addresses not in the list or list-digest subscriber databases. Addresses in this database are treated as subscribers, both for archive access and posts, but of course they will not receive list messages.
You can also make the list non-public. This can be done by removing DIR/public or by using the ezmlm-make ``-P'' switch (see More information). This results in a list where only remote administrators can access the archive. You can eliminate the archive altogether, but it may be required for other purposes (such as digest generation).
All these additions can be made default for all lists by modifying ezmlmrc (see More information).
You can add small ``trailer'' to each message. This usually would contain information on how to unsubscribe from the list, or where the list html archive is kept. To add a trailer, simply put the text in the file text/trailer in the list directory. You can use the ezmlm-make(1) ``-t'' switch to set up the list with a simple trailer that gives ``unsubscribe'' instructions. ezmlm will substitute the tags <#h#>, <#l#>, and <#n#> with the list host name, local name, and the current message number, respectively. The current message number for digests is the number of the first message in the digest. You can use this to e.g. add a line ``to get this thread, mail <#l#>-thread<#>@<#h#>''. This will always point to the archive of the current list with the current message number, and thus retrieve the thread containing the current message.
DIR/headerremove contains headers that are removed from all messages before they are sent to the subscribers. Some of these must be removed in order for the list to function. You can add others (see More information). If you add a header that should not occur multiple times, it is good to add the same header here to have it removed from incoming messages, if it exists.
Headers in DIR/headeradd are added to posts before they are sent to subscribers. They are also added to the archived version of the message.
ezmlm will substitute the tags
<#h#>,
<#l#>, and
<#n#> with the list host name, local name, and the current
message number, respectively.
For the digest of the list
mailinglist@example.org the local name is ``mailinglist-digest''.
The current message number for digests is the number of the first message
in the digest.
You can use this to e.g. add a sequence number header by
putting ``X-Seq: <#n#>'' into DIR/headeradd.
ezmlm can strip out MIME parts from multipart MIME messages. Some mail clients use ``multipart/alternative'' to send both plain text and html versions of the same message. On many lists this is considered annoying and wasteful (messages are usually 2-3 times as large). You can have ezmlm remove the html part from such messages by adding the line ``text/html'' to DIR/mimeremove.
ezmlm can also add a ``prefix'' to the subject line of each message
that does not already contain it. This was common on old lists, when
mail readers were unable to sort mail, except by subject. It still
popular for historical reasons, even though it violates mail standards.
Use this only if you feel that it is necessary, and for novice users
where ``look'' is more important than correctness. If you do use this
make it the list name and short, e.g. ``[mailinglist]'' (well, that's
from the example, in real life this is too long and too long a name
for a mailinglist as well. ml@example.org with ''ml''
would be better).
The prefix with the list name can be configured with the
ezmlm-make(1) ``-f'' switch. Alternatively, just place the text in
the file DIR/prefix.
For a real thrill, you can use ``[ml-#]''.
Here, ``#'' will be
replaced by the message number, labelling all messages with the
message number of the first message in the thread. This is not only
a mail standard violation, but will inconvenience many users and
html archiving programs. Still, it may be useful and help novice
users use the archive efficiently (mail ml-thread-#@example.org)
where ``#'' is the number in the prefix of the message to get
the entire thread from the archive). This works also with partially
rfc2047-encoded subjects. (See the ezmlm-send(1) man page for more info.)
See
Adding a trailer to posts for a better
way to point the recipients to the thread retrieval function.
Many old mailing list managers allow users to list the addresses of
all subscribers. This used to be OK, but has lately become a means
for junkmailers/spammers to acquire large collections of addresses.
Consequently, ezmlm does not allow this. Still, it may be useful for
the list owner to be able to get such a list to help a user unsubscribe
or to remotely backup the subscriber list.
ezmlm will supply such a list if the
list was set up with the ezmlm-make(1) ``-r'' and ``-l'' switches
and the recipient of the list is a ``remote administrator'' (see
Remote administration). The command
is -list. Thus, for the digest of mailinglist@example.org
a message to mailinglist-digest-list@example.org would return the
list of subscribers, but only if the sender is a remote administrator
and the -list function is enabled.
The remote administrator right apply to all address databases of the list (subscribers of list and list-digest, ``allow'' and ``deny''), except the moderator addresses. Moderators' addresses cannot be accessed remotely.
The subscription log is stored in DIR/Log. An entry is made for each modification made to the subscriber database. For subscription by E-mail, the subscribers ``From:'' line is logged. In contrast to the subscriber database, the log is not written in a crash-proof manner. Thus, information might be lost if the system crashes while the log is being updated.
Remote administrators can list this log with the -log command or search it for specific entries with the -log.xxx command, where ``xxx'' are letters or numbers to be matched exactly, or an underscore (``_'') to match any character. Since the information is highly overlapping with the subscriber list, it is enabled by the same commands, i.e. the ezmlm-make ``-l'' switch. In addition, it requires remote administration to be enabled (``-r''). See Allowing remote admins to get a subscriber list for more info.
Use the ezmlm-make(1) ``-n'' switch to allow remote editing of text files to remote administrators. Editing is allowed for all existing text files. Thus, in order to allow editing of the DIR/text/info file (sent in response to the -info command), you must at least have created such a file from the shell command line, or via ezmlmrc (default for new lists). See More information for more information.
A list of editable files is contained in the DIR/text/edit-list
file. This file itself for the list mailinglist@example.org
is editable via
mailinglist-edit.edit_list@example.org.
This file has no effect on which
files can be edited, but it useful since it contains the instructions
returned in reply to the -edit command. Hyphens in file names
need to be replaced by underscores in the edit commands.
See
More information for more information.
Qmail/mail systems are fault tolerant by virtue of a queuing mechanism where a message is removed from the queue only when the recipient mail transfer agent (MTA) has acknowledged receipt of the message. With this, the responsibility for the message has been transferred from the sending MTA to the receiving MTA. For messages that remain in the queue, the sending MTA will retry the transmission. After a set period of time, the message will be bounced, i.e. returned as ``undeliverable''. If the recipient does not exist (which may be caused by a misconfiguration of e.g. the domain naming service) the message is returned immediately as ``recipient unknown''.
ezmlm uses the subscriber address and message number encoded within the sender address to keep track of which message have ``bounced'' for which recipient. Since messages can bounce for a variety of reasons, ezmlm doesn't remove a subscriber from the list after a single bounce. Rather, it waits about 12 days after the first bounce, then sends a ``warning'' message to the subscriber. The warning message lists the numbers of the messages missed. The user can retrieve these from the archive. If this warning message bounces as well, ezmlm waits another 12 days, then sends a ``probe''. First when this message bounces is the subscriber removed from the subscriber database. This mechanism guarantees that a subscriber won't be removed due to a temporary error, while assuring that all non-existent or permanently failing addresses will be removed eventually.
Some users never learn and don't want to spend the time to figure out how
to unsubscribe. As a remote administrator(*), you can unsubscribe them entirely
on your own (see
Remote administration).
You can also do this from the command line. Alternatively, you can as a normal
user send a -unsubscribe request for the subscriber address and
let the subscriber confirm. Another alternative is to tell the user to send
a message to listname-help@listhost and ask the user to follow the
instructions, which are very explicit.
Other users have attempted to unsubscribe, but failed. They usually will tell
you what they've tried, and forward a list message to you.
Most likely, they are subscribed with an address other than their
normal sender address. If you're lucky, the subscriber address is in the users
E-mail to you. If not, you can ask the user to give you a list of possible
addresses. You can then send mail to listname-list@listhost(*) (if
enabled) and use ``find'' or ``grep'' to search through the list for parts
of the users addresses. If remote subscriber listing is not available, try:
% ezmlm-list DIR | grep -i 'part_of_user_address'
Another possibility is to search the subscription log for a part of the subscribers name, usually shown in the ``From:'' header (see Searching and listing the subscription log).
Subscriber-only lists allow posts and archive access only to subscribers of the
main list or the digest list. If a subscriber's posts or archive access
attempts are rejected, the subscriber is working with a sender address that
is not the one s/he is subscribed under. Usually, the user's sender address
will be in the E-mail to you. ezmlm-idx keeps a database of ``allow'' addresses
which are treated like subscribers for the purpose of subscriber-only
restrictions, but are not sent posts. To add an address, send mail to
listname-allow-subscribe-user=userhost@listhost. If you're a remote
administrator the confirmation request will come to you, otherwise to
user@userhost. Replying to the request will add the address. The
user should now be able to post and access the archive. To remove an address,
user the same procedure, but with the -unsubscribe command. Users
can manipulate the database themselves, but this may be too complicated for
all but advanced users.
If you set up the list with ezmlm-make ``-umr'' messages from addresses that are not subscribers will be sent for moderation. This allows you to ignore junk mail and approve other posts, as well as add the address to the ``allow'' address database, all without the subscriber noticing (see Placing restrictions on subscribers for more info).
Qmail and ezmlm have built-in advanced mail loop prevention. However, some auto-responders are misconfigured and create a new message to the list for every message received. Since this is a completely new message, ezmlm has no way of knowing that it is not a post. This is very rare, but may happen. The symptom is that a ``John is on vacation'' messages will arrive at the list, go out to subscribers, only to be followed by another ``John is on vacation'' message, and so on.
To stop this, look at the sender of the ``vacation'' message. Unsubscribe the address.
Even rarer is that there is a mail loop somewhere on the Internet, and that it sends of messages to your list. Unsubscribing doesn't help. Your only recourse is to block the sender of the message from posting. Easiest is to use the ``deny''(*) function with the ezmlm-make(1) ``-k'' switch, then add the sender address:
% ezmlm-sub DIR/blacklist badname@badhost
listname-deny-subscribe-badname=badhost@listhost. You could also
add a line:
|/bin/echo -- "$SENDER" |/usr/bin/grep -iv '^badname@badhost$' || exit 99
ezmlm/idx lists by default(*) reject messages that do not have the list address in the ``To:'' or ``Cc:'' headers. Most ``bulk E-mail'' does not pass this test. If you still receive too much junk mail to your list, you can set up your list as a subscriber-only(*) list. This creates a small overhead of users who try to post from addresses that are not their subscriber address. See above on how to deal with this.
Since the message sender is easy to fake, this is not secure. A determined SPAMMER can subscribe to the list or just send mail ``From:'' a subscriber address. The next level of countermeasure is to have a message moderated(*) (and possibly subscription moderated(*)) list. The overhead is higher - someone has to approve each message - but it is much more secure. ezmlm-idx has support for multiple moderators making it easy to set up a list with fast and efficient moderation, which is a very reliable way to keep junk mail off a list.
Several sites on the Internet subscribe to mailing lists, archive the messages, and make them publicly available. This may not be desirable, as the this often exposes the addresses of the subscribers. The subscribers, while happy to discuss their work on a peer list, may not appreciate messages from the general public asking for follow-ups to messages posted to the list.
ezmlm lists, by default(*), are set up with a ``X-No-Archive: yes'' header. For older lists or lists using ezmlm alone, just add this header to DIR/headeradd. Well-behaved archiving servers respect this. If you want to maintain full control, the only recourse is to make the list subscription moderated(*). Of course, the messages are no more ``secure'' than the least reliable subscriber. Remove the ``X-No-Archive: yes'' header for DIR/headeradd if you want to allow well-behaved archivers to collect your messages.
Like other list changes, you can use ezmlm-make with the ``-e'' edit switch together with other desired switches and arguments. An existing list can be made message moderated with:
% ezmlm-make -em DIR
The archive is a set of files, one per message, kept in the archive subdirectory of the list directory. The directory archive/0/ contains messages 1-99, archive/1/ messages 100-199, etc. The files within the directories are named by the last 2 digits of the message number, ``01-99'' for archive/0/ and ``00-99'' for others. In addition, each set of messages is indexed(*) in the file index stored with the corresponding message files.
With time, the archive grows and it may become necessary to remove some messages. When the list is at message ``12301'' one may wish to remove messages 1-9999 from the archive. Easiest is to remove messages as entire subdirectories. For the above, do:
% cd ~/
% tar -cvf listarch1_9999.tar DIR/archive/? DIR/archive/1?
% gzip listarch1_9999.tar
% rm -rf ~/DIR/archive/? ~/DIR/archive/1?
Save listarch1_9999.tar.gz in a safe place. (Usually, you would choose a better name for the backup ``tar'' file). Nothing else needs to be done. If instead you remove individual messages and use ezmlm-idx, it is useful, but not necessary to do:
% ezmlm-idx ~/dir
To restore the messages saved in the example above to the archive, simply:
% cd ~/
% gunzip listarch1_9999.tar.gz
% tar -xvf listarcg1_9999.tar.gz
% ezmlm-idx ~/dir