Next:
Contents
Contents
Sendmail X:
Requirements,
Architecture,
Functional Specification,
Implementation,
and Performance
Claus Assmann
Contents
Sendmail X: Requirements
Requirements
Basic Requirements
Other Requirements
Explanation of Buzzwords
Robustness
Flexibility
Scalability
Extendability
Maintainability
Portability
Restrictions
Configuration
Configuration File
Configuration Options
Documentation
Documentation Format
Sendmail X: Architecture
General Architecture of sendmail X
Overview
Remarks about Performance
Remarks about Configuration
Simplicity and Flexibility
Option Grouping
Accessing and Changing the Configuration
Security Considerations
Control and Data Flow
Configuration
General
Syntax
Comments
Internationalization
Strings and other Types
Inclusion
Alternative Configuration Formats
Configurability
Consistency
Configuration File Structure
Option Names
Dynamic Configuration
Configuration: Conditionals
Configuration: Reference to Maps
Default Configuration
Displaying Current Configuration
Dynamically Changing Configuration
Configuration Example for PreAlpha Version
Anti-Spam Configuration for PreAlpha Version
Security Considerations
Supervisor
Security Considerations
Queue Manager
Queues
Current Selection of Queues and their Names
Queue Formats
Reference Counting
Data Flow: Moving Envelopes between Queues
Terminology
Data Flow: Queue oriented View
Detailled Data Flow: Transaction/Recipient oriented View
Updating Data in Queues after a Delivery Attempt
Reading Entries from Deferred Queue
Cut-Through Delivery
Scheduling
Two Level Scheduling
First Level Scheduling (Global)
Second Level Scheduling (Micro)
Minimizing Number of Transactions
Cleanup of Active Queue
Outgoing Connection Cache
Multiple Destinations
Data Structures to support Scheduling
Triggering Deliveries
DSNs
DSN Recipient Types
DSN: Return Body or Headers
DSN: Delayed
Load Control
Misc
Security Considerations
SMTP Server Daemon
Internet Server Application
SMTP Server Design Alternatives
Control Flow
Anti-Spam Checks
Valid User Checks
Address Rewriting
Envelope Address Rewriting
Header Address Rewriting
Security Considerations
Address Resolver
Address Resolver Operation
Generic Comment about Map Lookups
Mail Routing
Address Rewriting
Proposal for Routing and Rewriting
Valid Recipients
Valid Local Recipients
Valid Remote Recipients
Aliases
Forward
Other Approaches to Aliasing?
Expansion to Multiple Addresses
Virtual Hosting
Fallback
Security Considerations
Initial Mail Submission
Initial Mail Submission Alternatives
How to use queue directory?
Misc
Security Considerations
Mail Delivery Agents
Delivery Agent Modules
Specifying Delivery Agents
Local Delivery Agent
Security Considerations
SMTP Client
Control Flow
Security Considerations
Milter
Possible Enhancements
More Ideas for Enhancements
Security Considerations
Miscellaneous Programs
Access to the Queue Manager
Show Mail Queue
Force Queue Run
Mailstats
Performance Statistics
Security Considerations
Maps
Modules
Security Considerations
Security Hints
Privileged Access
Access to Files
Sockets
Running as a different user
Misc
Misc Misc
Configuration
Configuration Changes
Performance Measurements: Profiling
Logging
Logging Granularity
Parsing Logfiles
Extensible Logging
Debugging
Robust Programming
Schedule
Glossary
Sendmail X: Functional Specification
Functional Specification of sendmail X
Asynchronous APIs
Generic Description of Asynchronous Operation
Some Remarks about Identifiers
Configuration
Configuration Structure
Naming Conventions
Specification of Classes (Lists, Sets)
Supervisor
External Interfaces
Operation
Shutdown
Configuration
Internal Interfaces
Queue Manager
External Interfaces
Shutdown
Internal Interfaces
Indices for Accessing the EDBs
Interface to SMTP DAs
QMGR - Delivery Agents API
Transferring Data between EDBs
Detailled Data Flow for Cut-Through Delivery
Reading Entries from Deferred Queue
Reconstructing Data from IBDB
Cleaning up IBDB
Deferred Envelope Database: Recipient Addresses
Scheduler Algorithms
Slow Start and Connection Limit
Data Structures
Connection Cache Access
Connection Reuse Problem
Data for DSN
Incoming Queue (INCEDB)
Active Queue (ACTEDB, AQ)
Deferred Queue (DEFEDB)
Deferred Queue Cache (EDBC)
Connection Cache (incoming)
Connection Status (incoming)
Connection Cache (outgoing) Connections
Connection Status (outgoing)
DA Status Cache
Load Control Data Structures
Database and Cache APIs
QMGR - SMTPS API
QMGR - First Level Scheduler API
QMGR - Delivery Scheduler API
Interface to AR
Aliases
Updating Data in Queues after a Delivery Attempt
Preserving Order of Updates
Load Control Functionality
Load Control Functionality: Throttling
Load Control Functionality: Unthrottling
Handling out of Memory
Manual Interaction with QMGR
Getting Status Informations
Triggering Actions
SMTP Server Daemon
External Interfaces
Control Flow
Startup
States
Data Structures
Detailled Control Flow
Pipelining
Anti-Spam Checks
Anti-Spam Checks: Functionality
Anti-Spam Checks: Functionality for first Release
Anti-Spam Checks: API
Valid Recipient Checks
Valid Sender Checks
Map Lookups
Policy Milter Interface Behavior
Interaction between Anti-Spam Checks and Milter
More Generic Return and Reply Code Behavior
Internal Interfaces
Throttling
Removal of Entries from CDB
Address Resolver
External Interfaces
Valid Recipients
Valid Local Recipients
Internal Interfaces
Data Structures
Error Behavior
Caching of Map Lookups
Threading Model
Initial Mail Submission
External Interfaces
Internal Interfaces
Mail Delivery Agents
External Interfaces
Configuration
Configuring Multiple DAs
Selecting and Naming Delivery Agents
Internal Interfaces
Status Information from DA to QMGR
SMTP Client
External Interfaces
Internal Interfaces
Status Information from SMTP Client to QMGR
Milter
External Interfaces
Internal Interfaces
Miscellaneous Programs
Show Mail Queue
Force Queue Run
Mailstats
Security Hints
Owners and Permissions
Databases and Caches for Envelopes, Contents, and Connections
DBs with Multiple Access Keys
DBs with Non-unique Access Keys
Envelope Database Access Methods
Incoming Envelope Database API
Incoming Envelope Database API: RSC
Incoming Envelope Database: Misc
Incoming Envelope Database API: Disk Backup
Active Envelope Database API
Deferred Envelope Database API
Deferred Envelope Database Implementation
Content Database
Content Database API
CDB: Group Commits
Logging as CDB Implementation
CDB: Meta Data Operations
CDB: File Reuse
Connection Database (Cache)
Available Database Implementation
Restricted Size Caches
Restricted Size Caches With Varying Size
Atomic Updates
Atomic Disk Writes
Data Structures for Queues on Disk and for Communication
Size Estimates
Misc
Maps
Results of Map Lookups
Lookup Functionality
Placing the '@' sign
Replacing Patterns in RHS
Error Handling
Returning Errors
Error Classification
Converting Error Codes into Textual Descriptions
Misc
Libraries
Naming Conventions
Naming Conventions for Types
Naming Conventions for Functions
Naming Conventions for Variables
Naming Conventions for Structures
Naming Conventions for Macros
Include Files
I/O
Required I/O Functionality
I/O Buffer
Event Driven Loop
Resource Pools
Memory Handling
String Abstraction
Constant Strings
Timed Events
Shared Memory
Address Parsing
RFC 2821 parsing
RFC 2822 parsing
Token handling
Address rewrite engine
Internal Communication
Data Structures for Internal Communication
Marking End of Record
A Note about Record Types
Common API for Storage Libraries (Maps)
Data Structure for Storage Libraries
Abstraction Layer for Storage Libraries
Dynamic (Re)Open of Maps
Asynchronous Operation of Maps
Multi-Threading
Multiple Map Instances
DNS
Timeouts
Misc
Logging
ISC Logging
Modules
Building sendmail X
autotools
Operating System Calls
General Hints
Operating System Specific Hints
Worker Threads
SMTP Server and Worker Threads
Worker Thread Library
Thread Libraries
State Threads for Internet Applications
Dealing with Blocking Sections for Worker Threads
Worker Threads with Semaphores
Worker Threads with Dynamic Limits
Counting Active Threads
Summary: Dealing with Blocking Sections for Worker Threads
Event Thread Library
Overview
Sendmail X: Implementation
Introduction to the Sendmail X Source Code
Identifiers
Asynchronous Functions
Two Different Problems
Two Approaches
Asynchronous Functions: Callback Synchronization
Asynchronous Functions: Result Queue
Transaction Based Processing
Secure Programming
Schedule
Libraries
Queues, Lists, etc
Hash Tables
Classes
Trees
RSC
Typed RSC
DBs with Multiple Access Keys
DBs with Non-unique Access Keys
Event Thread Library
RCB Communication
DNS
DNS Data Structures
DNS Functions
Logging
Logfile Rotation
SMTP Server Daemon
First Prototype
Misc
Communication between SMTPS and QMGR
Implementation of Communication between SMTPS and QMGR
SMTPS - QMGR Protocol
SMTPS - SMAR Protocol
Mail Delivery Agents
Maintaining Delivery Classes and Agents
SMTP Client
First Prototype
SMTPC - QMGR Protocol
SMTP Client Implementation
SMTP Client Data Structures
SMTP Client Functions
Queue Manager
Queue Manager Implementation
Locking
Deadlock Avoidance
Data Structures
Data Flow
Functions
Initialization Functions
Communication Functions
Commit Task
Scheduler
QMGR to SMTPC Protocol
Load Control Implementation
Updating Recipient Status
Handling Bounces
DSN: Delayed
AR to QMGR
Address Resolver Implementation
Resolving Recipient Addresses
Alias Expansion
Aliases: Owner- Handling
Milter
Sendmail 8 Milter Protocol
Macros
Option Negotiation
SMTPS - Milter Protocol
Option Negotiation
Differences to Milter
Implementation of Databases and Caches for Envelopes, Contents, and Connections
Incoming Envelope Database
Incoming Envelope Database Disk Backup
Active Envelope Database
Deferred Envelope Database Implementation
Misc
IBDB Cleanup
Show Mail Queue
Testing
Problems encountered in the Implementation
Implementation Problems
Out of Memory
Transaction Based Processing
Behavior of the Implementation
Number of Open Outgoing Connections
Sendmail X: Performance Tests and Results
SMTP Server Daemon
SMTP Sink
SMTP Sink with CDB
SMTP Relaying Using a Sendmail X Prototype
Various Linux FS
Various FreeBSD Results
Various SunOS 5 Results
Various OpenBSD Results
Various AIX Results
Implementation of Queues and Caches
Filesystem Performance
Test Systems
Meta Data Operations
Writing a Logfile
Harddisk Performance
Performance of Berkeley DB
Miscellaneous about Performance
Performance of Various Programs
TCP/IP Performance
DB Lookup Performance
snprintf
Performance
Bibliography
Claus Assmann