Logging has evolved significantly from the traditional syslog days. Despite this, many syslog-ng users continue to rely on syslog protocols for log transport and flat files for log storage. While most SIEMs and log analytics tools can receive syslog messages or read them using their agents, you can often leverage the http() destination of syslog-ng to send logs. This method offers high performance and a simpler architecture to maintain.
Enhancing Log Transport with syslog-ng
Syslog-ng includes various drivers built on top of the http() destination. For example, the elasticsearch-http() destination sends logs to Elasticsearch or OpenSearch. Many services utilize the Elasticsearch Bulk API, including Sumo Logic and Splunk. However, some destinations where performance isn’t a primary concern (or could be a drawback) include instant messaging services like Telegram or Slack for alerting. Additionally, you can read the API documentation and write a new destination based on http() if needed.
Preparing Your Environment
For most modern operating systems, the bundled syslog-ng version is sufficient for http() features, requiring at least version 3.23. This version is available in the RHEL 8 EPEL repository, and other OSs usually have more recent versions. If your OS has an older version, you can check for third-party repositories with updated packages at syslog-ng’s third-party binaries page. These repositories also provide access to the latest syslog-ng features, such as type support or a fast MongoDB destination.
Simplifying Your Logging Architecture
While many SIEMs and log analysis tools have their log forwarding tools, using syslog-ng for both central log collection and forwarding to analytics tools can simplify your architecture. Typically, you want to save most incoming log messages for long-term archiving and send only a subset for further analysis. Each log analysis tool requires a different subset of messages.
Allowing various log forwarding tools to read logs saved for long-term archiving can waste resources, as you send all logs using multiple applications over the network. On the analytics side, processing more logs increases costs, especially for commercial applications licensed by log volume. Saving subsets of logs to files for reading also wastes disk space and requires extra applications.
By using syslog-ng to forward logs to different SIEMs and analysis tools, you ensure recipients only get the logs they need, without extra disk space or applications.
Boosting Performance
Elasticsearch, Sumo Logic, and Splunk offer HTTP-based APIs for log collection. Sending logs directly to these APIs simplifies your logging architecture and boosts performance. The http() destination supports multiple workers and load-balancing, allowing syslog-ng to utilize multiple CPU cores and network connections to send log messages. This feature enables syslog-ng to send logs to multiple ingest nodes in parallel, balancing the load.
These capabilities simplify your setup, as a single syslog-ng node can feed multiple Splunk or Elasticsearch nodes without needing a dedicated load-balancing application or appliance. While the tcp() destination offers similar possibilities, they are more limited. Cribl documentation also recommends using the elasticsearch-http() destination instead of the syslog protocol for these reasons.
Moving Forward
With HTTP becoming a common protocol for log transport, logging has surpassed the traditional syslog protocol. While some APIs work over HTTP but can’t be implemented using the http() destination of syslog-ng, many applications and services use the Elasticsearch Bulk API or the Splunk HEC API. Often, all you need is to determine the correct URL to use, making it unnecessary to read API docs and create an http()-based destination from scratch.
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
How we help
If you need to know more about syslog-ng, you’re in the right place, we’re here to help! DTA is One Identity’s distributor, especially in Singapore and Asia, our technicians have deep experience on the product and relevant technologies you can always trust, we provide this product’s turnkey solutions, including consultation, deployment, and maintenance service.
Click here and here and here to know more: https://dtasiagroup.com/one-identity/
