The Syslog-ng Kafka source in Java — An Introduction

Kafka source is my Google summer of code project for the year 2016 with the Syslog-ng organization under the guidance of Viktor Juhász. Syslog-­ng can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to the destinations.Syslog­-ng has a Kafka destination, which is implemented in Java. Reading messages from kafka will make it as a queue between two Syslog-­ng instances.

Kafka is a distributed messaging system provides fast and highly scalable and redundant messaging through a pub­sub model. It allows a large amount of ad­hoc consumers and also it self heals from the errors and highly available in resilient. Therefore this feature in Syslog-ng, will improve the clustering performance of Syslog­-ng, Atomicity of data and communication between two Syslog­-ng instances.

Kafka source functional features

Kafka source is implemented in Java using Syslog-ng’s Java source. It is a high-level kafka consumer. Java source feature is still not integrated with the Syslog-ng product. Kafka source can,

• Read messages from Kafka
• Send those read messages to syslog-ng
• Single topic consumption
• Can consume multiple topics with multiple syslog-ng sources
• Can consume messages from last read offset.

Setting up the Syslog-ng

Since the latest version of the syslog-ng doesn’t have java source, you need to clone Java source featured version from here [1]. From now on we assume this directory will be $syslog-ng-home. Then install it by executing the following commands.

./autogen.sh
 mkdir build
 mkdir install
 cd build/
 ../configure -enable-debug -prefix=$syslog-ng-home/install
 make
 make install

This will install the syslog-ng in $syslog-ng-home/install directory. You can find syslog-ng library in $syslog-ng-home/install/sbin directory

Setting up Kafka source

Now you need to get the kafka-source Jar. Currently, Kafka source is not integrated with the Syslog-ng. So you can clone it from here [2]. From now onwards we call this directory as $kafka-source-home.

Change the repository directory path to following in build.gradle

$syslog-ng-home/install/lib/syslog-ng/java-modules

Then run Gradle build fatJar. Kafka source Jar can be found in $kafka-source-home/build/libs directory.

Kafka source requires, org.apache.kafka : version 0.9.0.1 dependency, which can be downloaded from the maven central. (For this you require Internet connection)

The Syslog-ng Configuration

In order to start Kafka-source with syslog-ng you need to create a configuration file. Kafka source require 7 options to be filled by the user.

Example Syslog-ng Kafka source configuration

Here is a simple kafka source configuration with file destination.

@version: 3.8.0alpha0
source kafka_source {
 java(
 class_name(“org.syslog_ng.KafkaSourceHandler”)
 class_path(“$kafka-source-home/kafka-source-all-1.0-SNAPSHOT.jar”)
 option(“zookeeper_host”,”localhost:2181″)
 option(“group_id_name”,”groupid11″)
 option(“topic”,”testtopic”)
 option(“zookeeper_session_time_out”,”400″)
 option(“zookeeper_sync_time_out”,”200″)
 option(“commit_interval”,”1000″)
 option(“consumer_timeout_time”,”10000″)
 );
 };
destination text_file {
 file(“/home/user/kakfa_ouput/output.txt”);
 };
log {
 source(kafka_source);
 destination(text_file);
 };

Save it in $kafka-source-home/kafka-source.conf

Setting up Kafka

After completing syslog-ng / kafka-source setups, you need to setup Kafka. Download kafka binary from here [3].

Call this directory as $kafka-home. Now you can start Zookeeper server, Kafka and Kafka producer accordingly,

$kafka-home/bin/zookeeper-server-start.sh config/zookeeper.properties
$kafka-home/bin/kafka-server-start.sh config/server.properties
$kafka-home/bin/kafka-console-producer.sh -broker-list localhost:9092 -topic testtopic

Playing with it

After successfully starting kafka producer, now you can start kafka consumer with syslog-ng.

$syslog-ng-home/install/sbin/syslog-ng -Fe -f $kafka-source-home/kafka-source.conf

If you are done with this, you can see kafka consumer is started with syslog-ng. Now you can produce messages from kafka producer and see them in kafka source console and from kakfka-output/output.txt (Destination we set in configuration file).

You can link this kafka source to another syslog-ng kafka destination or to tcp port or to any other destinations.

syslog-ng kafka source console

Stay tuned! 😉

Thank you for reading so far :). If you have any issues with kafka source or if you find any bugs in it feel free to comment on this thread 🙂 This kafka-source configurations/ functionalities can be changed with users’ suggestions. Please stay tuned for the updates.

Till next post,
Vithulan 😀
Happy Coding! 😉

References

[1] https://github.com/juhaszviktor/syslog-ng/tree/f/JavaSource
[2] https://github.com/VIthulan/kafka-source
[3] http://kafka.apache.org/downloads.html

Originally published at http://vithulanmv.wordpress.com on August 14, 2016.