UsMan's WoRkSpAce

Tuesday, June 12, 2007

Centralized log management with syslog-ng

*** Following are the key features of syslog-ng:
Centralized system logging
TCP based communication is inherently more reliable than UDP (can be TLSed)
Flexible filtering options for message content, category and origin
Diverse choices of sources and destinations including TCP, UDP, files, pipes, FIFO, datagram and stream unix sockets
Wide platform support (all major Unix and Linux)
Agent for windows to transfer logs to central server (only available in commercial edition)
Disk based message buffering, local storage of messages, if central server is unavailable (only available in commercial edition)
Flow control, where sylog-ng stops reading messages from sources, if destination cannot process the previously sent messages
Manipulate and re-structure log messages, such as adding hostnames and date to the beginning of every log message with templates. Can also be used for constructing SQL statements
Timezone support including interpreting timezone in message, associating source and destination with a particular time zone
Support large size messages with log_msg_size option
Private and configurable DNS cache allows storing IP to hostname mapping

*** Centralized logging is essential for large heterogeneous environments to serve the purpose of security, auditing and system health monitoring.

*** Limitations with default syslog include limited filtering based only on priority and facility, absence of content based filtering, lack of TCP support, lack of timezone information in log messages, max message size of 1024 bytes

*** syslog-ng can receive messages from applications, files, remote hosts and other sources. syslog-ng clients that send messages to syslog-ng server also run syslog-ng. Log-path defines what syslog-ng does with a message, connecting sources with destinations. Log-path in syslog-ng configuration is called log statement and can include filters. syslog-ng relay cannot write messages from the network sources to local files. As per syslog RFC, syslog message consists of PRI, HEADER and MSG. Total size of syslog message should be less than 1024 bytes. To replace the default syslogd, it is important to know how the native syslogd communicates on the particular platform. e.g /var/run/syslog SOCK_DGRAM on Mac OS X. Unix sockets can be of stream or datagram type. Source or destination driver is a communication method used for receiving or delivering log messages. A source should be defined only once, although it can be used in several log paths. Log paths are processed in their sequence of appearance. All log paths/statements are processed, so multiple actions can be taken for every message. flag options within log statement can override this behavior. 'final', 'fallback', 'catchall' and 'flow-control' are available flags. Filters consist of functions (which supports regular expressions) and boolean operators such as and, or, not. Templates can be used to create message formats or filenames and can include macros.

*** Configuration of syslog-ng begins with global object definition. Object can be source, destination, log, filter or template. Parameters within object definition can be compulsory or optional. Comments are pre-fixed with '#' character. Sample source statement is,
source {source-driver(params); source-driver(params); ...}; A sample destination statement is,
destination {destination-driver (params); destination-driver {params); ...}; A sample log statement is,
log {
source(name); source(name); ...
filter(name); filter(name); ...
destination(name); destination(name); ...
flags(flag1);
}; A sample filter statement is,
filter {expression; }; A sample template statement is,
template