Choosing how to collect and forward log messages to Splunk using syslog-ng involves understanding the evolution of support for Splunk within syslog-ng, as well as weighing the pros and cons of different solutions, both open-source and commercial.
History of Splunk Support in syslog-ng
Traditionally, syslog-ng recommended a method involving central log collection using syslog-ng, storing logs locally, and using Splunk forwarders to send them to Splunk. While effective, this approach incurred overhead by requiring installation of multiple applications and duplicating data storage.
The introduction of Splunk HTTP Event Collector (HEC) simplified log forwarding to Splunk. Initially, using the http() destination in syslog-ng was straightforward but lacked encryption and scalability. To address these shortcomings, a Python script was developed for improved security and performance when called via the program() destination.
Over time, syslog-ng’s http() destination evolved with TLS support, multi-threading, and load balancing. Syslog-ng Premium Edition (PE) introduced splunk-hec(), streamlining Splunk configuration compared to direct http() usage.
Additionally, Splunk released Splunk Connect for Syslog (SC4S), a containerized solution based on syslog-ng. SC4S enhances syslog-ng with additional message parsers, though it offers limited configuration compared to syslog-ng.
Syslog-ng Store Box (SSB), built on syslog-ng PE, offers comprehensive log lifecycle management with a Splunk destination.
Recently, syslog-ng open-source edition (OSE) integrated a Splunk destination into its configuration library (SCL), eliminating the need for custom solutions starting from version 4.2.0.
Choosing the Right Solution
The choice between syslog-ng editions depends on various factors:
- syslog-ng PE and SSB: Ideal for organizations requiring robust support and exclusive features like compliance and cloud support. They offer commercial-grade reliability and advanced capabilities like LogStore for encrypted log storage.
- syslog-ng OSE: Suitable for long-time open-source users or small-scale deployments using the free version of Splunk. It now includes built-in Splunk destination, simplifying configuration without commercial support.
- SC4S: Suitable if Splunk is the sole destination and complex filtering isn’t required. It’s based on syslog-ng open-source with added parsers, but lacks extensive configuration flexibility.
Conclusion
Both syslog-ng PE and OSE provide high-performance log collection, parsing, filtering, and versatile destination options, including Splunk. Effective message parsing and filtering reduce license costs by forwarding only relevant logs to each service. Evaluate trial versions of commercial syslog-ng variants or explore the open-source edition to determine the best fit for your environment.
For more details or trials, visit syslog-ng trials page which also provides access to the syslog-ng GitHub page for the open-source edition.
Source: https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-splunk-using-syslog-ng
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
