There are many ways you can collect log messages using syslog-ng and forward them to Splunk. In this blog I collect the history of Splunk support in syslog-ng, and the advantages and disadvantages of various solutions, both open source and commercial.
History
For many years, the recommended method for sending syslog messages to Splunk was to collect logs centrally using syslog-ng, write them to disk, and use one of the Splunk forwarders to send the logs to Splunk. This method works reliably, but it also has much overhead. It requires installing two different applications, and often storing the same data twice: one set of logs for long term storage, another reduced set for the Splunk forwarder.
Splunk HEC made forwarding logs to Splunk a lot easier. I originally posted how to use the http() destination to forward log messages to Splunk using syslog-ng six years ago: https://www.syslog-ng.com/community/b/blog/posts/sending-logs-splunk-http. By that time the http() destination of syslog-ng was single threaded and did not support encryption. However, it was still suitable for sending log messages to Splunk running on the same server.
To work around the security and scalability issues of the http() destination, a Python script was created that could be called using the program() destination of syslog-ng. It added security and better performance.
Soon after my blog was published, the http() destination received TLS support and multi-threading. A few versions later even load-balancing was added.
Syslog-ng Premium Edition (PE), the commercial variant of syslog-ng gained a new destination: splunk-hec(). It was based on my blog and made configuring the Splunk destination a lot easier compared to using the http() destination directly.
Splunk also released a log collector based on syslog-ng, Splunk Connect for Syslog (SC4S): https://github.com/splunk/splunk-connect-for-syslog/. It is a containerized solution, which uses syslog-ng at its core. It has very limited configuration possibilities but provides additional message parsers compared to upstream syslog-ng.
Syslog-ng Store Box (SSB), an appliance built on top of syslog-ng PE providing full log life-cycle management, also added a Splunk destination.
In one of the recent releases syslog-ng open-source edition (OSE, which is simply called “syslog-ng” in my blogs) also added a Splunk destination to the syslog-ng configuration library (SCL). Those users, who implemented a Splunk destination based on my blogs or SC4S, do not require this workaround anymore with syslog-ng 4.2.0 or later.
How to choose?
What you should use depends on many factors. Splunk would love to see you using SC4S. One Identity prefers syslog-ng PE or SSB, however open-source syslog-ng might also be perfect for the job. One thing is for sure: there is no more need for the Python implementation, as native C has much better performance while using less resources.
syslog-ng PE and SSB
As a rule of thumb, if you pay for Splunk because of the large amount of log messages or paying only features, you should go with syslog-ng PE or SSB. This way you get commercial support for your complete syslog-ng PE/SSB infrastructure (including clients, relays and server) and access to exclusive features. Of course, the open-source version works just fine with the paying Splunk version, but if you invest into Splunk licenses, you might want to invest into the extra reliability features (like ALTP) of syslog-ng PE or SSB.
Syslog-ng PE includes most features of the open-source version (except some experimental features, like the Riemann destination), and adds some exclusive features for compliance (like LogStore – encrypted, time stamped log storage), cloud (like office365 source support) and Windows log collection. On the configuration side it retains the flexibility of syslog-ng open source.
SSB is an appliance providing full log life cycle management using a web-based GUI and a search interface. You have access to most syslog-ng PE features and a lot more using an easy-to-use web-based GUI. The compromise here is that some of the configuration flexibility compared to the command line version is lost. Syslog-ng PE clients and relays are included in the SSB license, which allows complex log processing before logs reach SSB.
syslog-ng
If you are a long-time syslog-ng open-source user, you do not have to switch to one of the commercial variants of syslog-ng to send log messages to Splunk. Especially, if you are a small user, using only the free version of Splunk. Users sent logs to Splunk based on my blog or SC4S sources for many years. Starting with syslog-ng version 4.2.0 you do not even have to create a Splunk destination yourself, it is now part of the syslog-ng configuration library (SCL). SC4S could also serve as inspiration for your own syslog-ng configuration.
The open-source version of syslog-ng does not come with commercial support. If you need commercial support and / or one of the syslog-ng PE exclusive features (various cloud and compliance features), you should go with syslog-ng PE or SSB.
SC4S
SC4S by Splunk is based on the open-source version of syslog-ng, which is running in a container to forward collected log messages to Splunk. Compared to upstream syslog-ng, SC4S contains several extra parsers. The original design of SC4S lacked filtering and log forwarding to other destinations. Both are available in recent versions, in a very limited way.
Use SC4S if Splunk is your only destination and you do not need complex filtering. However, if you have multiple destinations, need complex filtering, using syslog-ng directly is better for you. If you need commercial support and / or one of the syslog-ng PE exclusive features (various cloud and compliance features), you should go with syslog-ng PE or SSB.
What is next?
Using either syslog-ng editions you get access to high performance log collection, message parsing, filtering, and a large range of possible destinations. It enables you to collect log messages, store them for long term archival, and after some processing you can forward them to multiple destinations, including but not limited to Splunk.
Message parsing and filtering help you to forward only relevant logs to Splunk, and other services. Most of these charge you based on the amount of log data analyzed. Forwarding only relevant logs to each service means that you can optimize license costs using syslog-ng.
I hope that my blog helps you decide which solution suits your environment the best. If you are still in doubt, you can get access to trial versions of commercial syslog-ng variants at https://www.syslog-ng.com/trials/ which also has a link to the syslog-ng GitHub page for the open source edition.
Source: https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-splunk-using-syslog-ng
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
