Securing an Apache Kafka broker - part III

This is the third in a series of blog posts about securing Apache Kafka. The first post looked at how to secure messages and authenticate clients using SSL. The second post built on the first post by showing how to perform authorization using some custom logic. However, this approach is not recommended for non-trivial deployments. In this post we will show at how we can create flexible authorization policies for Apache Kafka using the Apache Ranger admin UI. Then we will show how to enforce these policies at the broker.

1) Install the Apache Ranger Kafka plugin

The first step is to download Apache Ranger (0.6.1-incubating was used in this post). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting plugin to a location where you will configure and install it:

• tar zxvf apache-ranger-incubating-0.6.1.tar.gz
• cd apache-ranger-incubating-0.6.1
• mvn clean package assembly:assembly -DskipTests
• tar zxvf target/ranger-0.6.1-kafka-plugin.tar.gz
• mv ranger-0.6.1-kafka-plugin ${ranger.kafka.home}

Now go to ${ranger.kafka.home} and edit "install.properties". You need to specify the following properties:

• COMPONENT_INSTALL_DIR_NAME: The location of your Kafka installation
• POLICY_MGR_URL: Set this to "http://localhost:6080"
• REPOSITORY_NAME: Set this to "KafkaTest".

Save "install.properties" and install the plugin as root via "sudo ./enable-kafka-plugin.sh". The Apache Ranger Kafka plugin should now be successfully installed (although not yet configured properly) in the broker.

2) Configure authorization in the broker

Configure Apache Kafka as per the first tutorial. There are a number of steps we need to follow to configure the Ranger Kafka plugin before it is operational:

• Edit 'config/server.properties' and add the following: authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
• Add the Kafka "config" directory to the classpath, so that we can pick up the Ranger configuration files: export CLASSPATH=$KAFKA_HOME/config
• Copy the Apache Commons Logging jar into $KAFKA_HOME/libs. 
• The ranger plugin will try to store policies by default in "/etc/ranger/KafkaTest/policycache". As we installed the plugin as "root" make sure that this directory is accessible to the user that is running the broker.

Now we can start the broker as in the first tutorial:

• bin/kafka-server-start.sh config/server.properties

3) Configure authorization policies in the Apache Ranger Admin UI 

At this point we should have configured the broker so that the Apache Ranger plugin is used to communicate with the Apache Ranger admin service to download authorization policies. So we need to install and configure the Apache Ranger admin service. Please refer to this blog post for how to do this. Assuming the admin service is already installed, start it via "sudo ranger-admin start". Open a browser and log on to "localhost:6080" with the credentials "admin/admin".

First lets add some new users that match the SSL principals we have created in the first tutorial. Click on "Settings" and "Users/Groups". Add new users for the principals:

• CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE
• CN=Service,O=Apache,L=Dublin,ST=Leinster,C=IE
• CN=Broker,O=Apache,L=Dublin,ST=Leinster,C=IE

Now go back to the Service Manager screen and click on the "+" button next to "KAFKA". Create a new service called "KafkaTest". Click "Test Connection" to make sure it can communicate with the Apache Kafka broker. Then click "add" to save the new service. Click on the new service. There should be an "admin" policy already created. Edit the policy and give the "broker" principal above the rights to perform any operation and save the policy. Now create a new policy called "TestPolicy" for the topic "test". Give the service principal the rights to "Consume, Describe and Publish". Give the client principal the rights to "Consum and Describe" only.

4) Test authorization

Now lets test the authorization logic. Bear in mind that by default the Kafka plugin reloads policies from the admin service every 30 seconds, so you may need to wait that long or to restart the broker to download the newly created policies. Start the producer:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to check that the producer is authorized correctly. Now start the consumer:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

If everything is configured correctly then it should work as in the first tutorial.