Friday, September 16, 2016

Securing an Apache Kafka broker - part I

Apache Kafka is a messaging system for the age of big data, with a strong focus on reliability, scalability and message throughput. This is the first part of a short series of posts on how to secure an Apache Kafka broker. In this post, we will focus on authenticating message producers and consumers using SSL. Future posts will look at how to authorize message producers and consumers.

1) Create SSL keys

As we will be securing the broker using SSL client authentication, the first step is to create some keys for testing purposes. Download the OpenSSL ca.config file used by the WSS4J project. Change the "certificate" value to "ca.pem", and the "private_key" value to "cakey.pem". You will also need to create a directory called "ca.db.certs", and make an empty file called "ca.db.index". Now create a new CA key and cert via:
  • openssl req -x509 -newkey rsa:1024 -keyout cakey.pem -out ca.pem -config ca.config -days 3650
Just accept the default options. Now we need to convert the CA cert into jks format:
  • openssl x509 -outform DER -in ca.pem -out ca.crt
  • keytool -import -file ca.crt -alias ca -keystore truststore.jks -storepass security
Now we will create the client key, sign it with the CA key, and put the signed client cert and CA cert into a keystore:
  • keytool -genkey -validity 3650 -alias myclientkey -keyalg RSA -keystore clientstore.jks -dname "CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE" -storepass cspass -keypass ckpass
  • keytool -certreq -alias myclientkey -keystore clientstore.jks -file myclientkey.cer -storepass cspass -keypass ckpass
  • echo 20 > ca.db.serial
  • openssl ca -config ca.config -policy policy_anything -days 3650 -out myclientkey.pem -infiles myclientkey.cer
  • openssl x509 -outform DER -in myclientkey.pem -out myclientkey.crt
  • keytool -import -file ca.crt -alias ca -keystore clientstore.jks -storepass cspass
  • keytool -import -file myclientkey.crt -alias myclientkey -keystore clientstore.jks -storepass cspass -keypass ckpass
Now follow the same template to create a "service" key in servicestore.jks, with store password "sspass" and key password "skpass". In addition, we will create a "broker" key in brokerstore.jks, with storepass "bspass" and key password "bkpass".  

2) Configure the broker

Download Apache Kafka and extract it (0.10.0.1 was used for the purposes of this tutorial). Copy the keys created in section "1" into $KAFKA_HOME/config. Start Zookeeper with:
  • bin/zookeeper-server-start.sh config/zookeeper.properties
Now edit 'config/server.properties' and add the following:
  • ssl.keystore.location=./config/brokerstore.jks
  • ssl.keystore.password=bspass
  • ssl.key.password=bkpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
  • security.inter.broker.protocol=SSL
  • ssl.client.auth=required
  • listeners=SSL://localhost:9092
and start the broker and then create a "test" topic with:
  • bin/kafka-server-start.sh config/server.properties
  • bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
3) Configure the producer

Now we will configure the message producer. Edit 'config/producer.properties' and add the following:
  • security.protocol=SSL
  • ssl.keystore.location=./config/servicestore.jks
  • ssl.keystore.password=sspass
  • ssl.key.password=skpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
and start the producer with:
  • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
Send a few messages to the topic to make sure that everything is working ok.

4) Configure the consumer

Finally we will configure the message consumer. Edit 'config/consumer.properties' and add the following:
  • security.protocol=SSL
  • ssl.keystore.location=./config/clientstore.jks
  • ssl.keystore.password=cspass
  • ssl.key.password=ckpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
and start the consumer with:
  • bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer
The messages sent by the producer should appear in the console window of the consumer. So there it is, we have configured the Kafka broker to require SSL client authentication. In the next post we will look at adding simple authorization support to this scenario.

4 comments:

  1. Hi,

    I followed this blog and am trying to set up SSL with Kafka. However, when I edit the server.properties file with the properties you had mentioned:

    ssl.keystore.location=./config/brokerstore.jks
    ssl.keystore.password=bspass
    ssl.key.password=bkpass
    ssl.truststore.location=./config/truststore.jks
    ssl.truststore.password=security
    security.inter.broker.protocol=SSL
    ssl.client.auth=required
    listeners=SSL://localhost:9092

    and start the server I get the following error:

    java.lang.IllegalArgumentException: requirement failed: security.inter.broker.protocol must be a protocol in the configured set of advertised.listeners. The valid options based on currently configured protocols are Set(PLAINTEXT)
    at scala.Predef$.require(Predef.scala:219)
    at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:956)
    at kafka.server.KafkaConfig.(KafkaConfig.scala:934)
    at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:698)
    at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:695)
    at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
    at kafka.Kafka$.main(Kafka.scala:58)


    Any thoughts?
    Im using kafka 2.11-0.9.0.1

    ReplyDelete
    Replies
    1. Figured the problem. First it was an issue with the older version of Kafka I was using and even with the 0.10* version you had use one must also set the

      advertised.listeners=SSL://localhost:9092

      parameter in order for it to work without errors.

      Thanks

      Delete
  2. I gave the same configurations in my case also and kafka is starting properly but schema registry is not starting. I dont know what might be the problem . Can you please help me with it?
    The schema registry version is 3.2.2

    ReplyDelete