Securing an Apache Kafka broker - part I

Apache Kafka is a messaging system for the age of big data, with a strong focus on reliability, scalability and message throughput. This is the first part of a short series of posts on how to secure an Apache Kafka broker. In this post, we will focus on authenticating message producers and consumers using SSL. Future posts will look at how to authorize message producers and consumers.

1) Create SSL keys

As we will be securing the broker using SSL client authentication, the first step is to create some keys for testing purposes. Download the OpenSSL ca.config file used by the WSS4J project. Change the "certificate" value to "ca.pem", and the "private_key" value to "cakey.pem". You will also need to create a directory called "ca.db.certs", and make an empty file called "ca.db.index". Now create a new CA key and cert via:

• openssl req -x509 -newkey rsa:1024 -keyout cakey.pem -out ca.pem -config ca.config -days 3650

Just accept the default options. Now we need to convert the CA cert into jks format:

• openssl x509 -outform DER -in ca.pem -out ca.crt
• keytool -import -file ca.crt -alias ca -keystore truststore.jks -storepass security

Now we will create the client key, sign it with the CA key, and put the signed client cert and CA cert into a keystore:

• keytool -genkey -validity 3650 -alias myclientkey -keyalg RSA -keystore clientstore.jks -dname "CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE" -storepass cspass -keypass ckpass
• keytool -certreq -alias myclientkey -keystore clientstore.jks -file myclientkey.cer -storepass cspass -keypass ckpass
• echo 20 > ca.db.serial
• openssl ca -config ca.config -policy policy_anything -days 3650 -out myclientkey.pem -infiles myclientkey.cer
• openssl x509 -outform DER -in myclientkey.pem -out myclientkey.crt
• keytool -import -file ca.crt -alias ca -keystore clientstore.jks -storepass cspass
• keytool -import -file myclientkey.crt -alias myclientkey -keystore clientstore.jks -storepass cspass -keypass ckpass

Now follow the same template to create a "service" key in servicestore.jks, with store password "sspass" and key password "skpass". In addition, we will create a "broker" key in brokerstore.jks, with storepass "bspass" and key password "bkpass".  

2) Configure the broker

Download Apache Kafka and extract it (0.10.0.1 was used for the purposes of this tutorial). Copy the keys created in section "1" into $KAFKA_HOME/config. Start Zookeeper with:

• bin/zookeeper-server-start.sh config/zookeeper.properties

Now edit 'config/server.properties' and add the following:

• ssl.keystore.location=./config/brokerstore.jks
• ssl.keystore.password=bspass
• ssl.key.password=bkpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security
• security.inter.broker.protocol=SSL
• ssl.client.auth=required
• listeners=SSL://localhost:9092

and start the broker and then create a "test" topic with:

• bin/kafka-server-start.sh config/server.properties
• bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

3) Configure the producer

Now we will configure the message producer. Edit 'config/producer.properties' and add the following:

• security.protocol=SSL
• ssl.keystore.location=./config/servicestore.jks
• ssl.keystore.password=sspass
• ssl.key.password=skpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security

and start the producer with:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to the topic to make sure that everything is working ok.

4) Configure the consumer

Finally we will configure the message consumer. Edit 'config/consumer.properties' and add the following:

• security.protocol=SSL
• ssl.keystore.location=./config/clientstore.jks
• ssl.keystore.password=cspass
• ssl.key.password=ckpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security

and start the consumer with:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

The messages sent by the producer should appear in the console window of the consumer. So there it is, we have configured the Kafka broker to require SSL client authentication. In the next post we will look at adding simple authorization support to this scenario.