BigAdmin Feature Article: Creating a Centralized Secure Log Server with syslog-ng and Stunnel

Skip to Content Java Solaris Communities About Sun How to Buy My Sun Connection Cart United States Worldwide

BigAdmin System Administration Portal

Feature Article

Print-friendly Version

Submit Content | Check BigAdmin Bucks

Creating a Centralized Secure Log Server with syslog-ng and Stunnel

Amy Rich, September, 2004

Contents:

Installing Stunnel
Creating Certificate Files for syslog-ng Over Stunnel
Configuring Stunnel for Use With syslog-ng
Installing syslog-ng
Configuring syslog-ng
Resources

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

The standard UNIX syslog daemon transports messages in the clear over UDP, meaning that anyone can sniff potentially sensitive data. The facility.level model is also fairly restrictive, and the default /etc/syslog.conf files supplied with most operating systems neglect to log many messages administrators might be interested in. Parsing out the logs in a way that makes sense for human readability or automated data mining is non-trivial. The UNIX syslog daemon doesn't easily allow one to split log files out by host or match log messages by regular expression, for example. Therefore, most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them.

As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar. One very popular syslog replacement is an open source program called syslog-ng. An organization can run syslog-ng on each UNIX host or just on the syslog server itself. If syslog-ng is run only on the log host, clients send data over UDP port 514 as usual, but better log organization and manipulation can be accomplished on the server.

The benefit of running syslog-ng on each UNIX host is the ability to encrypt the logging channel with IPSec or the utility Stunnel so that data is not readable by the casual sniffer. When combined with Stunnel as a transport mechanism, an organization has a secure way of centralizing log messages from all desired UNIX hosts for further processing. In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng.

Below I cover the installation and configuration of syslog-ng and Stunnel on machines running the Solaris 8 Operating System (SPARC Platform Edition), but the procedure also generally applies to older and newer versions of the Solaris OS on both SPARC and x86 platforms. Each of the reference machines discussed below comes installed with OpenSSL, tcp wrappers, the Solaris 8 /dev/urandom patch, and the GNU development environment (gcc, and so on) and several other freeware packages. The machine serving as the log server has also been thoroughly hardened since it will store sensitive and security-related information from all machines on the network. These reference machines all reside on the subnet 192.168.1, and the IP address of the log server is 192.168.1.10.

Installing Stunnel

The first step to implementing a secure log server is installing Stunnel on the server and each client. Stunnel could also be used with the stock syslog daemon (instead of replacing syslog with syslog-ng), but then we wouldn't have the flexibility we're after. In the instructions below, I configure and build stunnel to run with its own user and group and chrooted to its own private directory. To do this, first create the stunnel group and user (UID and GID picked at random):

/usr/sbin/groupadd -g 122 stunnel
/usr/sbin/useradd -c stunnel -d /nonexistent -m -g 122 -u 122 stunnel

Now grab the Stunnel source, unpack it, and configure it. On these particular hosts, the OpenSSL certificates are kept in /usr/local/etc/openssl/certs, and I wish to keep the doc directory in /usr/local along with other locally installed doc installations. I also set the localstatedir to /var/run/stunnel, since it doesn't need to persist through a reboot and I want it inside the chrooted directory.

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.05.tar.gz
tar zxf stunnel-4.05.tar.gz
cd stunnel-4.05

./configure --localstatedir=/var/run/stunnel \
 --with-pem-dir=/usr/local/etc/openssl/certs --datadir=/usr/local

make
make install

Creating Certificate Files for syslog-ng Over Stunnel

The Stunnel install process creates a self-signed certificate that you may opt to use. Since I run my own private certificate authority and will only run Stunnel for syslog-ng, I generate and sign my own syslog-ng-dedicated certificates. For detailed information on setting up your own CA and signing certificates, take a look at the SSL certificates HOWTO.

Assuming you're set up as your own CA or you will give the certificate requests to a publicly recognized CA, create a pem file for the server:

openssl req -new -days 3650 -nodes -config stunnel.cnf -out serverreq.pem \
 -keyout syslog-ng-server.pem

Also create a corresponding pem file for each client:

openssl req -new -days 3650 -nodes -config stunnel.cnf -out clientreq.pem \
 -keyout syslog-ng-client.pem

Sign each pem file with the local CA, or have them signed by a public CA. I use the sign.sh script that comes with the apache mod_ssl distribution:

sign.sh /tmp/serverreq.pem 
sign.sh /tmp/client1req.pem 
sign.sh /tmp/client2req.pem
sign.sh /tmp/client3req.pem

The resulting crt files include the certificates for each of the corresponding pem files. The server requires the server pem file, syslog-ng-server.pem, with the private key and certificate (copied over from the /tmp/serverreq.pem.crt file) of the server:

It also requires a client pem file, syslog-ng-client.pem, with only the certificates (from the crt files) from the signing CA and from every client (this example presumes three syslog-ng clients):

Each client needs its own certificate and private key in a pem file, syslog-ng-client.pem:

Each client would also need a pem file, syslog-ng-server.pem, containing only the certificates from the server and the signing CA:

On each machine, make sure that only root can read the certificate files for security reasons:

chmod 400 /usr/local/etc/openssl/certs/syslog-ng-*
chown root:other /usr/local/etc/openssl/certs/syslog-ng-*

Configuring Stunnel for Use With syslog-ng

On the server, create the syslog-ng-dedicated Stunnel configuration file, /usr/local/etc/stunnel/stunnel.conf, containing information similar to that below. This example file specifies the local cert/key and the server's certs, the stunnel user and group, and the chroot directory. The verify value of 3 ensures that stunnel will verify the peer with the locally installed certificate. Stunnel leaves verification off by default, so it's important to turn it on here. The final section of the configuration file specifies the port number for the SSL wrapped session and the IP:port where connections are accepted and redirected. Port 514 is the standard syslog port, and 5140 is an unused port picked at random. For additional information and configuration options, be sure to read the stunnel man page.

cert = /usr/local/etc/openssl/certs/syslog-ng-server.pem
CAfile = /usr/local/etc/openssl/certs/syslog-ng-client.pem
chroot = /var/run/stunnel
pid = /run/stunnel.pid
setuid = stunnel
setgid = stunnel
verify = 3
[5140]
   accept = 192.168.1.10:5140
   connect = 127.0.0.1:514

On each client, the syslog-ng-dedicated /usr/local/etc/stunnel/stunnel.conf file contains directives similar to the server's stunnel.conf file. The cert and CAfile values and the accept and connect values are swapped, and the client directive is added:

client = yes
cert = /usr/local/etc/openssl/certs/syslog-ng-client.pem
CAfile = /usr/local/etc/openssl/certs/syslog-ng-server.pem
chroot = /var/run/stunnel
pid = /run/stunnel.pid
setuid = stunnel
setgid = stunnel
verify = 3
[5140]
   accept = 127.0.0.1:514
   connect = 192.168.1.10:5140

Stunnel is now configured, and we're ready for the syslog-ng installation and configuration. If you wish to test Stunnel at this point, configure it to instead use another TCP port or service, such as IMAP or telnet, as described on the stunnel examples page.

Installing syslog-ng

The stable version of syslog-ng first requires the installation, or at least build, of the library libol. Download, unpack, and install the library as shown here:

wget http://www.balabit.com/downloads/libol/0.3/libol-0.3.14.tar.gz
tar zxf libol-0.3.14.tar.gz
cd libol-0.3.14

./configure
make
make install

Now retrieve the source for syslog-ng and unpack, configure, and install it. When configuring, I also add support for tcp wrappers since I have it installed and am actively using it for other daemons:

wget http://www.balabit.com/downloads/syslog-ng/1.6/src/syslog-ng-1.6.5.tar.gz
tar zxf syslog-ng-1.6.5.tar.gz
cd ../syslog-ng-1.6.5

./configure --enable-tcp-wrapper
make
make install

Be sure to open up the appropriate ports in any packet filters and/or tcp wrappers. The server needs to accept connections from the clients on TCP port 5140, and on UDP port 514 if the log host is also accepting unencrypted syslog messages. To support the extend syntax of tcp wrappers, add the following to /etc/hosts.deny on the server:

syslog-ng : LOCAL 127.0.0.1 192.168.1. : ALLOW

And on the client add the following to /etc/hosts.deny:

syslog-ng : LOCAL 127.0.0.1 : ALLOW

Now create an stunnel/syslog-ng startup script, /etc/init.d/syslog-ng, that will run at boot time on each machine. The script below is based on the syslog startup script for the Solaris 8 OS, and it also performs a savecore as well as starting stunnel and syslog-ng:

#!/sbin/sh
#

case "$1" in
'start')
	if [ -f /usr/local/etc/syslog-ng/syslog-ng.conf -a -x \
             /usr/local/sbin/syslog-ng ]; then
		#
		# Before syslogd starts, save any messages from previous
		# crash dumps so that messages appear in chronological order.
		#
		/usr/bin/savecore -m
		if [ -r /etc/dumpadm.conf ]; then
			. /etc/dumpadm.conf
			[ "x$DUMPADM_DEVICE" != xswap ] && \
			    /usr/bin/savecore -m -f $DUMPADM_DEVICE
		fi
		#
                # Start stunnel so logs are sent encrypted
                #
                if [ -f /usr/local/etc/stunnel/stunnel.conf \
                     -a -x /usr/local/sbin/stunnel ]; then
                   echo "Starting stunnel"
                   mkdir -p /var/run/stunnel/run
                   chown stunnel:stunnel /var/run/stunnel/run
		   /usr/local/sbin/stunnel 
                   echo "Starting syslog-ng"
                   /usr/local/sbin/syslog-ng 
                fi
	fi
	;;

'stop')
	if [ -f /var/run/syslog-ng.pid ]; then
		syspid=`/usr/bin/cat /var/run/syslog-ng.pid`
		[ "$syspid" -gt 0 ] && kill -15 $syspid && \
                echo "Killed syslog-ng"
	fi
        if [ -f /var/run/stunnel/run/stunnel.pid ]; then
                syspid=`/usr/bin/cat /var/run/stunnel/run/stunnel.pid`
                [ "$syspid" -gt 0 ] && kill -15 $syspid && \
                echo "Killed stunnel"
        fi

	;;

*)
	echo "Usage: $0 { start | stop }"
	exit 1
	;;
esac

Remove the links for the native Solaris syslog startup and shutdown scripts and replace them with links to the new syslog-ng scripts:

rm /etc/rc*.d/???syslog
ln -s /etc/init.d/syslog-ng /etc/rc0.d/K40syslog-ng
ln -s /etc/init.d/syslog-ng /etc/rc1.d/K40syslog-ng
ln -s /etc/init.d/syslog-ng /etc/rc2.d/S74syslog-ng
ln -s /etc/init.d/syslog-ng /etc/rcS.d/K40syslog-ng

Configuring syslog-ng

The flexibility of syslog-ng lies in its configuration file. The configuration directives important to log manipulation are source, filter, destination, and log. Source directives indicate the origination of the log messages both local and remote. Filter directives allow for the separation of log messages based on facility, level/priority, program name, host name, or regular expression matching. The destination can be files, pipes, streams and datagrams, UDP or TCP connections, ttys, or a program. The log directive is a collection of source, filter, and destination directives that define how a matching log message is processed. A discussion of all of the available directives can be found in the syslog-ng reference manual, and various examples are listed in the syslog-ng FAQ.

The following examples show log files being stored on each local host in /var/log as well as on the central log server in /var/log/clients/$YEAR/$MONTH/$HOST. The following /usr/local/etc/syslog-ng/syslog-ng.conf on the log host supports messages from the local host, stunnel encrypted hosts, and standard UDP hosts (like routers and switches that can't be made to use stunnel). The filters are based on facility and level, a program name match, and some combinations thereof.

# Options
options {
          use_fqdn(yes);
          sync(0);
          keep_hostname(yes);
          chain_hostnames(no);
	  create_dirs(yes);
        };

# Sources of syslog messages (both local and remote messages on the server)
source s_local   { 
                   sun-streams("/dev/log" door("/etc/.syslog_door")); 
                   internal();
                 };
source s_stunnel { 
                   tcp(ip("127.0.0.1")
                   port(514)
                   max-connections(1));
                 };

source s_udp     { udp(); };

# Level Filters
filter f_emerg   { level (emerg);            };
filter f_alert   { level (alert .. emerg);   };
filter f_crit    { level (crit .. emerg);    };
filter f_err     { level (err .. emerg);     };
filter f_warning { level (warning .. emerg); };
filter f_notice  { level (notice .. emerg);  };
filter f_info    { level (info .. emerg);    };
filter f_debug   { level (debug .. emerg);   };

# Facility Filters
filter f_kern   { facility (kern);   };
filter f_user   { facility (user);   };
filter f_mail   { facility (mail);   };
filter f_daemon { facility (daemon); };
filter f_auth   { facility (auth);   };
filter f_syslog { facility (syslog); };
filter f_lpr    { facility (lpr);    };
filter f_news   { facility (news);   };
filter f_uucp   { facility (uucp);   };
filter f_cron   { facility (cron);   };
filter f_local0 { facility (local0); };
filter f_local1 { facility (local1); };
filter f_local2 { facility (local2); };
filter f_local3 { facility (local3); };
filter f_local4 { facility (local4); };
filter f_local5 { facility (local5); };
filter f_local6 { facility (local6); };
filter f_local7 { facility (local7); };

# Custom Filters
filter f_user_none     { not facility (user);                     };
filter f_kern_debug    { filter (f_kern) and filter (f_debug);    };
filter f_daemon_notice { filter (f_daemon) and filter (f_notice); };
filter f_mail_crit     { filter (f_mail) and filter (f_crit);     };
filter f_mesg          { filter (f_kern_debug) or 
                         filter (f_daemon_notice) or 
                         filter (f_mail_crit);                    };
filter f_authinfo      { filter (f_auth) or program (sudo);       };

# Destinations: local files, the console, and the client files
destination l_authlog  { file ("/var/log/authlog");   };
destination l_messages { file ("/var/log/messages");  };
destination l_maillog  { file ("/var/log/maillog");   };
destination l_ipflog   { file ("/var/log/ipflog");    };
destination l_imaplog  { file ("/var/log/imaplog");   };
destination l_syslog   { file ("/var/log/syslog");    };

destination l_console  { file ("/dev/console");       };

destination r_authlog  { file
  ("/var/log/clients/$YEAR/$MONTH/$HOST/authlog");    }; 
destination r_messages { file 
  ("/var/log/clients/$YEAR/$MONTH/$HOST/messages");   }; 
destination r_maillog  { file 
  ("/var/log/clients/$YEAR/$MONTH/$HOST/maillog");    }; 
destination r_ipflog   { file 
  ("/var/log/clients/$YEAR/$MONTH/$HOST/ipflog");     }; 
destination r_imaplog  { file 
  ("/var/log/clients/$YEAR/$MONTH/$HOST/imaplog");    }; 
destination r_console  { file 
  ("/var/log/clients/$YEAR/$MONTH/$HOST/consolelog"); }; 
destination r_syslog   { file
  ("/var/log/clients/$YEAR/$MONTH/$HOST/syslog");     };
destination r_fallback { file
  ("/var/log/clients/$YEAR/$MONTH/$HOST/$FACILITY-$LEVEL"); };

# Log statements
# Local sources
log { source (s_local); filter (f_authinfo)  destination (l_authlog);  };
log { source (s_local); filter (f_mail);     destination (l_maillog);  };
log { source (s_local); filter (f_local0);   destination (l_ipflog);   };
log { source (s_local); filter (f_local1);   destination (l_imaplog);  };
log { source (s_local); filter (f_syslog);   destination (l_syslog);   };
log { source (s_local); filter (f_emerg); filter (f_user_none); 
                                             destination (l_console);  };
log { source (s_local); filter (f_mesg);  filter (f_user_none);
                                             destination (l_messages); }; 

# All sources, since we want to archive local and remote logs
log { source (s_local); source (s_stunnel); filter (f_authinfo);
     destination (r_authlog);   };
log { source (s_local); source (s_stunnel); filter (f_mail);
     destination (r_maillog);   };
log { source (s_local); source (s_stunnel); filter (f_local0);
   destination (r_ipflog);      };
log { source (s_local); source (s_stunnel); filter (f_local1);
   destination (r_imaplog);     };
log { source (s_local); source (s_stunnel); filter (f_syslog);
   destination (r_syslog);      };
log { source (s_local); source (s_stunnel); filter (f_emerg); 
      filter (f_user_none);
      destination (l_console);  };
log { source (s_local); source (s_stunnel); filter (f_mesg);
   filter (f_user_none);
      destination (l_messages); };

In this example client syslog-ng.conf, the filters remain the same, but most other parts of the configuration either change to reflect the client status or are removed:

# Options
options { 
          sync(0);
          use_fqdn(yes);
        };

# Sources of syslog messages (only local on clients)
source s_local { 
                 sun-streams("/dev/log" door("/etc/.syslog_door")); 
                 internal();
               };

# Destinations: local files, the console, and the remote syslog server
destination l_authlog  { file ("/var/log/authlog");   };
destination l_messages { file ("/var/log/messages");  };
destination l_maillog  { file ("/var/log/maillog");   };
destination l_ipflog   { file ("/var/log/ipflog");    };
destination l_imaplog  { file ("/var/log/imaplog");   };
destination l_console  { file ("/dev/console");       };
destination l_syslog   { file ("/var/log/syslog");    };
destination stunnel    { tcp ("127.0.0.1", port(514)); };

# Level Filters
filter f_emerg   { level (emerg);            };
filter f_alert   { level (alert .. emerg);   };
filter f_crit    { level (crit .. emerg);    };
filter f_err     { level (err .. emerg);     };
filter f_warning { level (warning .. emerg); };
filter f_notice  { level (notice .. emerg);  };
filter f_info    { level (info .. emerg);    };
filter f_debug   { level (debug .. emerg);   };

# Facility Filters
filter f_kern   { facility (kern);   };
filter f_user   { facility (user);   };
filter f_mail   { facility (mail);   };
filter f_daemon { facility (daemon); };
filter f_auth   { facility (auth);   };
filter f_syslog { facility (syslog); };
filter f_lpr    { facility (lpr);    };
filter f_news   { facility (news);   };
filter f_uucp   { facility (uucp);   };
filter f_cron   { facility (cron);   };
filter f_local0 { facility (local0); };
filter f_local1 { facility (local1); };
filter f_local2 { facility (local2); };
filter f_local3 { facility (local3); };
filter f_local4 { facility (local4); };
filter f_local5 { facility (local5); };
filter f_local6 { facility (local6); };
filter f_local7 { facility (local7); }; 

# Custom Filters
filter f_user_none     { not facility (user);                     };
filter f_kern_debug    { filter (f_kern) and filter (f_debug);    };
filter f_daemon_notice { filter (f_daemon) and filter (f_notice); };
filter f_mail_crit     { filter (f_mail) and filter (f_crit);     };
filter f_mesg          { filter (f_kern_debug) or 
                         filter (f_daemon_notice) or 
                         filter (f_mail_crit);                    };
filter f_authinfo      { filter (f_auth) or program (sudo);       };

# Log statements
# Log things locally
log { source (s_local); filter (f_authinfo); destination (l_authlog);  };
log { source (s_local); filter (f_mail);     destination (l_maillog);  };
log { source (s_local); filter (f_local0);   destination (l_ipflog);   };
log { source (s_local); filter (f_local1);   destination (l_imaplog);  };
log { source (s_local); filter (f_syslog);   destination (l_syslog);   };
log { source (s_local); filter (f_emerg); filter (f_user_none); 
                                             destination (l_console);  };
log { source (s_local); filter (f_mesg); filter (f_user_none);
                                             destination (l_messages); }; 

# Log everything remotely via stunnel
log { source (s_local);                      destination (stunnel);    };

More advanced uses of syslog-ng include sending log messages directly to data mining software, databases, email, or the printer, based on the importance of the message. Another useful tip is to send high-priority log messages to one file that can be watched by a real-time log analyzer like swatch, logsurfer, Log Tool, or Logwatch. The possibilities for automated data mining and monitoring are extensive since the log entries can be organized and handled in various ways.

Resources

Software packages on the reference systems:
- OpenSSL
- A HOW-TO for SSL certificates
- OpenSSL mailing lists
- tcp wrappers
- /dev/urandom patch for the Solaris 8 OS
- GNU gcc
- GNU wget
Stunnel resources:
- The latest Stunnel source code
- The Stunnel FAQ
- Some Stunnel examples
- The stunnel-users mail archive
syslog-ng resources:
- The syslog-ng home page
- The latest libol source code
- The latest syslog-ng source code
- The syslog-ng reference manual
- The syslog-ng FAQ
- Support packages for syslog-ng
- The syslog-ng mail archive
- The syslog-ng-announce mail archive
Real-time log parsing tools:
- swatch
- logsurfer
- Log Tool
- Logwatch

Discuss and comment on this resource in the BigAdmin Wiki

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.

Search BigAdmin

» Search the HCL

BigAdmin Wiki
Discuss and comment on this resource in the wiki.

BigAdmin Subscriptions
»	BigAdmin Newsletter
»	BigAdmin Blog
»	BigAdmin Wiki
»	RSS Feeds
BigAdmin Areas
»	All Topics/Sections
»	Articles and FAQs
»	Blogs and Wikis
»	Discussions
»	Documentation
»	Features
»	HCL
»	Multilingual
»	Patches
»	Products and Tech
»	Resources
»	Scripts
»	Software/Tools
»	Solaris 10 Apps
»	ShellCmds
»	Submit Content
»	Suggestions
»	SysAdmin Jobs
»	SysAdmin Topics
»	Tech Tips
BigAdmin Sun Center
»	Register Your System
»	Solaris Developer Center
»	Sun Support
»	Sun Doc Highlights
»	Sun Tech Days
»	Sun Advisory Panel for System Admins
»	Training and Certification
BigAdmin Topics
»	DST
»	DTrace
»	Interoperability
»	Networking
»	Predictive Self-Healing
»	ZFS
»	Zones

Contact About Sun News & Events Employment Site Map Privacy Terms of Use Trademarks Copyright Sun Microsystems, Inc.