The Syslog-ng Kafka source in Java — An Introduction

Kafka source is my Google summer of code project for the year 2016 with the Syslog-ng organization under the guidance of Viktor Juhász. Syslog-­ng can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to the destinations.Syslog­-ng has a Kafka destination, which is implemented in Java. Reading messages from kafka will make it as a queue between two Syslog-­ng instances.

Kafka is a distributed messaging system provides fast and highly scalable and redundant messaging through a pub­sub model. It allows a large amount of ad­hoc consumers and also it self heals from the errors and highly available in resilient. Therefore this feature in Syslog-ng, will improve the clustering performance of Syslog­-ng, Atomicity of data and communication between two Syslog­-ng instances.

Kafka source functional features

Kafka source is implemented in Java using Syslog-ng’s Java source. It is a high-level kafka consumer. Java source feature is still not integrated with the Syslog-ng product. Kafka source can,

  • Read messages from Kafka

Setting up the Syslog-ng

Since the latest version of the syslog-ng doesn’t have java source, you need to clone Java source featured version from here [1]. From now on we assume this directory will be $syslog-ng-home. Then install it by executing the following commands.

mkdir build
mkdir install
cd build/
../configure -enable-debug -prefix=$syslog-ng-home/install
make install

This will install the syslog-ng in $syslog-ng-home/install directory. You can find syslog-ng library in $syslog-ng-home/install/sbin directory

Setting up Kafka source

Now you need to get the kafka-source Jar. Currently, Kafka source is not integrated with the Syslog-ng. So you can clone it from here [2]. From now onwards we call this directory as $kafka-source-home.

Change the repository directory path to following in build.gradle


Then run Gradle build fatJar. Kafka source Jar can be found in $kafka-source-home/build/libs directory.

Kafka source requires, org.apache.kafka : version dependency, which can be downloaded from the maven central. (For this you require Internet connection)

The Syslog-ng Configuration

In order to start Kafka-source with syslog-ng you need to create a configuration file. Kafka source require 7 options to be filled by the user.

Example Syslog-ng Kafka source configuration

Here is a simple kafka source configuration with file destination.

@version: 3.8.0alpha0source kafka_source {
destination text_file {
log {

Save it in $kafka-source-home/kafka-source.conf

Setting up Kafka

After completing syslog-ng / kafka-source setups, you need to setup Kafka. Download kafka binary from here [3].

Call this directory as $kafka-home. Now you can start Zookeeper server, Kafka and Kafka producer accordingly,

$kafka-home/bin/ config/
$kafka-home/bin/ config/
$kafka-home/bin/ -broker-list localhost:9092 -topic testtopic

Playing with it

After successfully starting kafka producer, now you can start kafka consumer with syslog-ng.

$syslog-ng-home/install/sbin/syslog-ng -Fe -f $kafka-source-home/kafka-source.conf

If you are done with this, you can see kafka consumer is started with syslog-ng. Now you can produce messages from kafka producer and see them in kafka source console and from kakfka-output/output.txt (Destination we set in configuration file).

You can link this kafka source to another syslog-ng kafka destination or to tcp port or to any other destinations.

syslog-ng kafka source console

Stay tuned! 😉

Thank you for reading so far :). If you have any issues with kafka source or if you find any bugs in it feel free to comment on this thread 🙂 This kafka-source configurations/ functionalities can be changed with users’ suggestions. Please stay tuned for the updates.

Till next post,
Vithulan 😀
Happy Coding! 😉



Originally published at on August 14, 2016.

Computer Science Engineer

Computer Science Engineer