SMTP STARTTLS in sendmail/Secure Switch

Last Update 2009-06-22

[Terminology] [Compilation] [Configuration] [Operation] [Misc]


Sendmail Secure Switch/ sendmail 8.11/ 8.12 support SMTP STARTTLS as defined in RFC 2487 which is based on TLS. This document describes the necessary steps to use this feature.


TLS can provide authentication (identification of the communication partner), privacy/confidentiality (communication is not intercepted or eavesdropped), and integrity (message has not been modified). TLS uses different algorithms for encryption, signing, message authentication etc. An introduction to these can also be found in books about cryptography and in RFC 2246 and its list of references. Some important terms used here are: For those familiar with public key encryption (and signing), a cert can be considered a signed public key (with associated data to identify an entity).

Initial Setup

If sendmail should use STARTTLS as server and as client, you have to (names starting with conf refer to m4 variable names used in a .mc file):
  1. Install one or more CA certs in confCACERT. Notice: do not list too many root CAs in that file, otherwise the TLS handshake may fail; e.g.,
    error:14094417:SSL routines:SSL3_READ_BYTES:
    sslv3 alert illegal parameter:s3_pkt.c:964:SSL alert number 47
    You should probably put only the CA cert into that file that signed your own cert(s).
  2. Install zero or more CA certs in confCACERT_PATH with (symbolic) links of their hashes pointing to them:
    ln -s $C `openssl x509 -noout -hash < $C`.0
    (or sslc instead of openssl)

    These CA certificates are required to successfully authenticate another entity. The signature of the certificate presented by the other side is checked against these CAs. If one of them issued the certificate, the authentication is considered successful. Moreover, during the TLS handshake, the DNs of the CA certificates listed in confCACERT are sent to the client so it can properly select a certificate that is signed by one of those CAs.

  3. Get a cert (make sure the CN is the fully qualified name of your host) and install it as confSERVER_CERT and the private key as confSERVER_KEY (make sure the file is only readable by root or the trusted user). For simplicity, use the same filenames for confCLIENT_CERT and confCLIENT_KEY, respectively.
    Note: the private key must not be encrypted. This is required for unattended startup of sendmail. Otherwise someone/something would have to enter the passphrase each time sendmail is started as server or client.
  4. If you run sendmail 8.11 or later and your OS doesn't have /dev/urandom(4), then you need to setup a source to seed the pseudo random number generator. For Solaris 7 and 8, you may have a look at a kernel module for /dev/random (please check yourself whether it is ok to use!), or see whether Sun has a package called SUNWski for your OS. It is strongly advised to use at least EGD (Entropy Gathering Daemon) and compile sendmail with the flag EGD, and point confRAND_FILE to the socket used by EGD (use `egd:' as prefix!). If neither /dev/urandom(4) nor EGD are available, you have to make sure that useful random data is available all the time in confRAND_FILE (use `file:' as prefix). If the file hasn't been modified in the last 10 minutes before it is supposed to be used by sendmail the content is considered obsolete. In this case, the PRNG for TLS is only seeded with other random data if the DontBlameSendmail option InsufficientEntropy is set. This is almost always not sufficient for security.
You can create your own CA: Lutz Jänicke, Gregory Neil Shapiro, and Martin Ouwehand (for Apache, but most of the steps can be used for an MTA too) wrote explanations how to do it.

Extract from a .mc file:

define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl

Initial Test

Start the sendmail daemon, connect to it and see whether it comes up with
in the EHLO response:
% telnet localhost 25
Connected to localhost
Escape character is '^]'.
220 ESMTP Sendmail Sendmail 8.12.0/8.12.0; Sun, 30 Sep 2001 10:47:28 -0700 (PDT)
ehlo localhost Hello localhost [], pleased to meet you
250 HELP
If it doesn't, check your logfile whether any security problems are listed (unsafe files). If this doesn't reveal any problems, increase the LogLevel to 14 and try again.


STARTTLS can be used to allow relaying based on certificates, and to restrict incoming or outgoing connections. For this purpose, several rulesets are available which require some new macros and the access map.

New Macros

New macros for SMTP STARTTLS are
holds the DN of the CA (the cert issuer).
holds the DN of the cert (called the cert subject).
the TLS/SSL version used for the connection: TLSv1, SSLv3, SSLv2.
the cipher suite used for the connection, e.g., EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC-SHA, DES-CBC-MD5, DES-CBC3-SHA, RC2-CBC-MD5, RC4-MD5
the key length (in bits) of the symmetric encryption algorithm used for the connection.
holds the result of the verification of the presented cert. Possible values are:
verification succeeded
no cert presented
cert presented but could not be verified, e.g., CA is missing.
STARTTLS has not been performed
temporary error occurred, e.g., the other side replied with 454.
protocol error occurred
problems during the handshake at the TLS level. In this case the connection will be dropped.
the name of the server of the current outgoing SMTP connection.
the address of the server of the current outgoing SMTP connection.


SMTP STARTTLS can allow relaying for senders who have successfully authenticated themselves. This is done in the ruleset RelayAuth. If the verification of the cert failed ({verify} != OK), relaying is subject to the usual rules. Otherwise the DN of the issuer is looked up in the access map using the tag CERTISSUER. If the resulting value is RELAY, relaying is allowed. If it is SUBJECT, the DN of the CERT subject is looked up next in the access map. using the tag CERTSUBJECT. If the value is RELAY, relaying is allowed.

To make things a bit more flexible (or complicated), the values for {cert_issuer} and {cert_subject} can be optionally modified by regular expressions defined in the m4 variables _CERT_REGEX_ISSUER_ and _CERT_REGEX_SUBJECT_, respectively. To avoid problems with those macros in rulesets and map lookups, they are modified as follows: each non-printable character and the characters '<', '>', '(', ')', '"', '+' are replaced by their HEX value with a leading '+'. For example:
/C=US/ST=California/ Mail (Cert)/
is encoded as:

To allow relaying for everyone who can present a cert signed by /C=US/ST=California/ simply use:

CERTIssuer:/C=US/ST=California/		RELAY

To allow relaying only for a subset of machines that have a cert signed by /C=US/ST=California/ use:

CERTIssuer:/C=US/ST=California/		SUBJECT
CERTSubject:/C=US/ST=California/		RELAY

Of course it is also possible to write a simple rulesets that allows relaying for everyone who can present a cert that can be verified, e.g.,

R$*		$: $&{verify}
ROK		$# OK

Allowing Connections

The rulesets tls_server and tls_client are used to decide whether an SMTP connection is accepted (or should continue).

tls_server is called when sendmail acts as client after a STARTTLS command (should) have been issued. The parameter is the value of {verify}.

tls_client is called when sendmail acts as server, after a STARTTLS command has been issued, and from check_mail. The parameter is the value of {verify} and STARTTLS or MAIL, respectively.

Both rulesets behave the same. If no access map is in use, the connection will be accepted unless {verify} is SOFTWARE, in which case the connection is always aborted. Otherwise, {client_name}/{server_name} is looked up in the access map using the tag TLS_Srv (or TLS_Clt), which is done with the ruleset LookUpDomain. If no entry is found, {client_addr}/{server_addr} is looked up in the access map (same tag, ruleset LookUpAddr). If this doesn't result in an entry either, just the tag is looked up in the access map (included the trailing :). Values for those entries in the access map should be:

verification must have succeeded
verification must have succeeded and {cipher_bits} must be greater than or equal bits.
{cipher_bits} must be greater than or equal bits.

The RHS can optionally be prefixed by TEMP+ or PERM+ to select a temporary or permanent error. The default is a temporary error code (403 4.7.0) unless the macro TLS_PERM_ERR is set during generation of the .cf file.

If a certain level of encryption is required, then it might also be possible that this level is provided by TLS from a SASL algorithm, e.g., DIGEST-MD5. The number of bits used for the key of the symmetric cipher (here: 3DES) is stored in the macro {auth_ssf}.

Example: e-mail send to should only use an encrypted connection. e-mail received from hosts within the domain should only be accepted if they have been authenticated.	ENCR:112	PERM+VERIFY:112

sendmail 8.12 additionally offers TLS_Rcpt to enable per-recipient restrictions.

Received: Header

The Received: header reveals whether STARTTLS has been used. It contains an extra line:
(using ${tls_version} with cipher ${cipher} (${cipher_bits} bits) verified ${verify})

Notice: sendmail 8.12 has extended or slightly change some of this. For example, the STARTTLS related part of the Received: header looks now like this:

(version=${tls_version} cipher=${cipher} bit=${cipher_bits}  verify=${verify})


If you have setup all necessary certificates, you can test a STARTTLS connection by sending an e-mail to, see for details.

Advantages and Limits of TLS

STARTTLS has the following advantages: All of these advantages are provided transparently by MTAs without interaction of users. They do not need to have special software installed in their MUAs which additionally must be compatible with the software of the recipient. This is also the reason for several limits: Summary: to get end-to-end privacy, integrity and authentication, end-user software like PGP or S/MIME must be used. This requires at least some knowledge of such software and responsible use by end-users.



Sendmail Secure Switch/ sendmail 8.11 and later has been successfully tested against MTAs running postfix+STARTTLS patch, qmail+STARTTLS patch, PMDF V5.2-32, and zmailer.

Interoperability Problems

Sendmail Secure Switch/ sendmail 8.11 and later can't talk with some MTAs/MUAs running M$ Exchange can't send e-mail to Sendmail Secure Switch/ sendmail if STARTTLS is used. M$ knows about the problem and will provide a fix: ... and a QFE fix is now available from PSS. The fix will also be included in the next Windows 2000 service pack after SP2 and is already in Windows Xp 2002 beta.
Do not try STARTTLS with some servers

A configuration option to turn off STARTTLS in sendmail (as client) is implemented in the 8.12, it is not available in 8.11.

STARTTLS Debugging by Hand

A patch has been posted to the OpenSSL mailing list which contains a program similar to s_client to connect to MTAs for testing STARTTLS. Starting with OpenSSL 0.9.7, you can use
openssl s_client -starttls smtp -connect host:port ...


The implementation of STARTTLS in sendmail was influenced by the example programs of OpenSSL and the work of Lutz Jänicke.


An introduction to TLS by Stephen Thomas: "SSL and TLS Essentials", ISBN 0-471-38354-6. A basic introduction is Applied Cryptography. A lot of details are available in Menezes, van Oorschot and Vanstone: "Handbook of Applied Cryptography", ISBN: 0-8493-8523-7

If you prefer french instead of english, please read the instructions from Jean-Philippe Donnio.

[(links)] [Hints] [Avoiding UBE] [cf/README] [New]
Copyright © Claus Aßmann Please send comments to: <ca at>
Disclaimer: the information provided may be inaccurate or outdated or incomplete. Please contact me if you find an error.