O'Reilly: [Chapter 23] 23.5 How the Queue Is Processed

23.5 How the Queue Is Processed

Over time, messages may gather in the queue awaiting delivery. They remain there until sendmail performs a queue run to process the queue. The sendmail program can be told either to process the queue periodically (when run as a daemon) or to process the queue once, then exit. Each time sendmail processes the queue, it also performs a series of operations that are intended to improve the efficiency with which it delivers messages.

First the queue directory is opened for reading. If that directory cannot be opened, sendmail syslog(3)'s the following message at LOG_CRIT and exits:

orderq: cannot open "/var/spool/mailq" as "."

This error is usually the result of a user running sendmail in an unsafe manner, with a -C command-line argument, for example. It can also result from sendmail attempting to open an NFS-mounted queue directory, where root is mapped to nobody.

Next, the qf files are read to gather their priorities and times (the P and T lines). If a qf file cannot be opened, it is quietly ignored unless the -d41.2 debugging command-line switch (see Section 37.5.145, -d41.2) is specified. That switch causes sendmail to print the following error message:

orderq: cannot open qfMAA12345 (reason)

Prior to V8.7 sendmail, there was a hard limit on the number of messages that could be processed at any time. If more than QUEUESIZE (defined in conf.h, typically 1000) messages were in queue, only the first QUEUESIZE (1000) of them would be processed! Ordinarily, this was not a problem. But it could quickly become one if your queue were clogged with a huge number of undeliverable messages (where the first 1000 continued to be deferred). In that case the only solution is to temporarily move the 1000 messages out of the way by hand (see Section 23.7.1, "Handling a Down Site") and clear the queue. The only way to detect this situation is to print the queue (see Section 23.4).

V8.7 and above sendmail dynamically allocates memory to process the queue. If more than QUEUESIZE messages are found, sendmail will print the following notice and process them:

grew WorkQ for queue_directory to bytes

As an alternative to this dynamic behavior, V8.7 and above sendmail offers a hard limit that is somewhat like the old version but is site tunable with the MaxQueueRunSize option (see Section 34.8.38). After all the qf files have been gathered, they are sorted in order of cost. Messages with the lowest value of the P line have the highest priority (lowest cost) and are processed first.

Beginning with V8.7, sendmail also offers the QueueSortOrder option (see Section 34.8.51, QueueSortOrder), which allows you to sort by priority (as before), by priority and host-name, or by date queued. Once all the messages have been sorted, sendmail processes each in turn.

23.5.1 Processing a Single Message

A single queued message has a single sender but may have many recipients. When processing a queued message, sendmail attempts to deliver it to all recipients before processing the next queued message.

The first step in processing a queued message is to lock it so that concurrent runs of sendmail do not attempt to process it simultaneously (see Section 23.2.3.1, "Current style file locking"). Then the qf file is opened and read. The sender and all the recipients are gathered from the corresponding S and R lines.

For each recipient, delivery is attempted. If delivery is successful, that recipient's address is removed from the sendmail program's internal list of recipient addresses. If delivery fails, that address is either left in the list or bounced, depending on the nature of the error.

After all recipients have been either delivered, bounced, or left in the list, sendmail reexamines that list. If there are no recipients left in it, the message is de-queued (all of the files in the queue directory that compose it are removed). If any recipients are left, the M line is assigned the error message of the last failed message, and the qf file is rewritten with the list of remaining recipients. Finally, the qf file is closed, thus freeing its lock.

Under V8 sendmail the CheckpointInterval (C) option (see Section 34.8.7, CheckpointInterval (C)) causes checkpointing of this process. When this option has a positive value, the qf file is rewritten after that value's number of recipients have been processed. For example, consider a mail message to five recipients. If the CheckpointInterval (C) option is set to a value of 2, the qf file is rewritten after the first two recipients have been processed, then again after four, and again after they all have been processed. This keeps the qf file reasonably up-to-date as protection against sendmail being improperly killed or the machine crashing.