Print-friendly Version

Preventing Denial of Service Attacks in Sun Java System Messaging Server

This article describes some of the ways you can prevent Denial of Service (DoS) attacks and similar Spam abuses on Sun Java System Messaging Server. Use this article to learn more about:

• Common techniques for preventing DoS attacks to the SMTP server and preventing abuse from internal users
• Mitigating abuse from clients who use DSL connections and who send lots of email using "email bombers" directly to the platform

Note: Keep in mind that when it comes to "stopping" Spam, often a "soft" technique can be more effective than a "hard" technique. Consider a jujitsu analogy: rather than attempting to directly stand in the way of the Spam, one sticks one's foot out and lets the Spam trip over it (and crash to the floor due to its own momentum). For Spam, that might translate to letting Spam in but sidelining it (.HELDing it), "tar-pitting" suspected Spam transactions (intentionally responding very slowly during the SMTP dialogue), and using other similar "soft" techniques.

About DoS Attacks

A DoS attack happens when an organization has trouble accessing the network resources it normally uses. Typically, that network service is email, which has become unavailable or has lost connectivity. Usually done intentionally and maliciously, DoS attacks can also happen accidentally, though this is rare. A common form of a DoS attack is to send more traffic to a network address than the system buffers can handle. The effect of this attack is to make it difficult for legitimate requests for a session to get established.

Common Techniques to Use

Limiting Specified IP Address Connections to the MTA

A particular IP address can be limited to how often it connects to the Messaging Server MTA by using the shared library, conn_throttle.so in the Port Access mapping table. Limiting connections by particular IP addresses may be useful for preventing excessive connections used in DoS attacks.

See Chapter 17, Mail Filtering and Access Control, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Extraordinarily large lists of recipient addresses are often a characteristic of unsolicited bulk email. The holdlimit keyword tells the MTA that messages coming in the channel that result in more than the specified number of recipients should be marked as .HELD messages and enqueued to the reprocess channel (or to whatever channel is specified via the expandchannel keyword). The files will sit unprocessed in the reprocess queue awaiting manual intervention by the MTA postmaster.

See Chapter 12, Configuring Channel Definitions, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Using conn_throttle from a FROM_ACCESS Mapping

Use conn_throttle from a FROM_ACCESS mapping rather than from PORT_ACCESS, and then use $D delays to delay SMTP responses and/or $H to .HELD the incoming message(s) rather than just rejecting ($N) the messages once the throttle limit has been reached.

Another approach is to start delaying and/or .HELDing at a "starting to get suspicious" threshold level, and then go ahead and reject at a higher "fairly certain it's abuse" threshold level.

See Chapter 17, Mail Filtering and Access Control, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Using the alternatechannel and alternaterecipientlimit Channel Keywords

The MTA provides the ability to retarget messages that exceed a specified limit on the number of recipients, message size, or message lines to an alternate destination channel. This is implemented as a set of the following channel keywords, alternatechannel, alternateblocklimit,alternatelinelimit, and alternaterecipientlimit, that can be placed on any destination channel. A message that exceeds any of these thresholds will be enqueued to the alternate channel instead of the original destination channel.

Use the alternatechannel and alternaterecipientlimit channel keywords to force routing of message copies that have "lots" of recipients to an alternate destination channel (which is perhaps configured to process messages in a more bottle-necked fashion due to a more restrictive maxjobs setting, or at "off-peak" times of day).

See the section titled "Limits on Messages, Quotas, Recipients, and Authentication Attempts," in Chapter 12, Configuring Channel Definitions, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Enforcing Use of SMTP Authentication

SMTP Authentication, or SMTP Auth (RFC 2554) is the preferred method of providing SMTP relay server security. Enforce use of SMTP Auth to submit messages, log the authenticated submitter (LOG_USERNAME=1 in the option.dat file), and then impose sanctions on any such submitters observed to be submitting Spam.

See Chapter 19, Configuring Security and Access Control, in the Sun Java System Messaging Server 2005Q4 Administration Guide, and Chapter 13, Planning Messaging Server Security, in the Sun Java System Communications Services 2005Q4 Deployment Planning Guide for more information.

Configuring Per-channel Recipient Limits

Configure per-channel recipient limits by using recipientlimit and recipientcutoff. recipientlimit specifies the total number of recipient addresses that will be accepted for the message. recipientcutoff compares the total number of recipients that were presented to the MTA to the specified value. No message will be accepted for delivery if the limit of the value is exceeded. Both keywords accept a single integer argument. The default for both is infinite unless the corresponding channel keyword is specified.

Recipient limits can also be set on a sender or sender's domain. This is done by specifying a user or domain LDAP attribute with the appropriate MTA option: LDAP_RECIPIENTLIMIT, LDAP_RECIPIENTCUTOFF,LDAP_DOMAIN_ATTR_RECIPIENTLIMIT, LDAP_DOMAIN_ATTR_RECIPIENTCUTOFF, and adding the attribute to the sender's user entry or domain entry.

This technique can be counterproductive unless you're also going to impose transaction limits and/or some other form of limiting. Otherwise, just limiting the number of recipients submitted per transaction may merely get rid of one of the easier-to-detect clues that a message is Spam.

Note: These keywords were introduced in Messaging Server 6, and so do not exist for Messaging Server 5 installations.

See the section titled "Limits on Messages, Quotas, Recipients, and Authentication Attempts," in Chapter 12, Configuring Channel Definitions, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Checking "Email Bombers" for Proper Host Name

Check if email "bombers" are presenting a proper host name on their initial HELO/EHLO line and if not, "tar-pit" (or .HELD or reject).

Note: This feature was introduced in Messaging Server 6, so that the FROM_ACCESS mapping table probe includes the presented host name.

Configuring Spam Filter Packages

Add a third-party Spam filter package to your Messaging Server deployment. For example,  SpamAssassin is freeware, however, other choices exist. 

See Chapter 14, Integrating Spam and Virus Filtering Programs Into Messaging Server, in the Sun Java System Messaging Server 2005Q4 Administration Guide for more information.

Finding and Deleting Spam on the System

Use the imsimta qtop command (and then imsimta qclean command) to find (and then delete) Spam that has made it into your system and is still in your MTA's channel queues.

Upcoming Enhancements in the Next Messaging Server Release

A new MeterMaid facility is planned in the upcoming Messaging Server release, which provides a similar functionality as conn_throttle, but spans an entire Messaging Server deployment (multiple hosts and multiple SMTP server processes per host accessing the same repository of throttling information), unlike conn_throttle where the information is only per-process (hence with conn_throttle, a spammer who reconnects and happens to reconnect to a different SMTP server process can once again send messages).

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.