This is the fifth part of my syslog-ng tutorial. Last time we had an overview of the syslog-ng configuration and had our first steps working with syslog-ng. Today we learn about syslog-ng source definitions and how to check the syslog-ng version and its enabled features.
You can watch the video or read the text below.
YouTube link: https://youtu.be/6P9PIrKYRKE
Source definition
A source definition is a collection of one or more source drivers. It consists of two parts. It starts with the word “source”, followed by a source identifier which you will use later to refer to the given source. After that, it lists the source drivers and their parameters. Here is an example source definition. Here is how its syntax looks like:
Using indentation and new lines is optional, but I highly recommend making your configuration easier to read.
Here is a simple file source:
And here is a more complex example with multiple source drivers:
The internal() driver collects the internal messages of syslog-ng, like starting up, network connection problems, and so on. You should have it in your configuration.
The two file() sources follow various log files.
The system() source collects system-specific local log messages. We will talk about this more later.
Depending on how you process and store logs later, you might have a single source definition with many drivers, multiple source definitions with just a single driver in each, or a mixture of these.
Source flags
Sources can have various parameters, like the file name in the previous example. You can also use flags to modify how sources work. This tutorial covers some of the most common flags – you can find more in the syslog-ng documentation.
By default, syslog-ng parses all incoming log messages as if they were formatted according to the RFC 3164 specification (which is also called legacy or BSD syslog). Incoming log messages might use different formatting, such as JSON or CSV. In this case, use the no-parse flag, which stores the whole log message into the MESSAGE field. You can use a parser later to extract information from the log message.
Sometimes, log messages arrive with formatting according to the RFC 5424 or “new” syslog specification. In that case, use the syslog-protocol flag.
Source drivers
In the various sample configurations, we have already seen some of the source drivers. Here is the list of the most common ones:
- internal(): Collects the internal messages of syslog-ng. You should collect these so that you can check if syslog-ng works properly or if there are any major problems.
- unix-stream(), unix-dgram(), pipe(): Collects log messages from Unix domain sockets and named pipes.
- file(): Reads log messages from files.
- network(): Reads legacy (RFC 3164) log sources.
- syslog(): Reads new (RFC 5424) syslog messages over the network.
- sun-stream(): Reads streams on Sun Solaris.
- program(): Runs a program and reads standard output.
- python(): Allows you to code your own source driver in Python.
For the complete list, see the syslog-ng documentation.
The system() source
The system() source can ease your life in multiple ways, as it lets you collect the system-specific local log messages of a host. Of course, this only works on operating systems and environments used by syslog-ng developers and contributors. So how can it ease your life, you might ask?
- You do not have to discover all possible system-specific log sources by yourself.
- You can use the same configuration on multiple systems.
- It offers easy transition from /dev/log to systemd-journal (so there is no need to edit the configuration).
- The system() source parses some of the incoming log messages automatically (such as sudo logs).
Here is an example of using it:
A common mistake
Duplicating sources can cause mysterious errors, like syslog-ng trying to bind twice on the same IP address and port. Make sure that you define a source only once, even if you need to use a source multiple times. Just refer to the same source name in any log path where you need it.
Checking the syslog-ng version
Before finishing this part of my syslog-ng tutorial, I want to show you a syslog-ng command line option. Using -V you can get the syslog-ng version, along with lots of other practical information. You can see the directory where syslog-ng modules are installed, the path for SCL, the list of available modules, and a number of build-time options, like Linux capabilities support.
Sources: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-101-part-5-sources
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
