Last year, I wrote a
series of articles on securing Apache
Kafka. In particular, the
third article looked at how to use Apache
Ranger to create authorization policies for Apache Kafka in the Ranger security admin UI, and how to install the Ranger plugin for Kafka so that it picks up and enforces the authorization policies. In this article, we will cover an alternative way of creating and enforcing authorization policies in Apache Ranger for Apache Kafka using Apache
Atlas.
The Apache Ranger security admin UI allows you to assign users or groups a particular permission associated with a given Kafka topic. This is what is called a "Resource Based Policy" in Apache Ranger. However an alternative is also available called a "Tag Based Policy". Instead of explicitly associating the user/group + permission with a resource (such as a Kafka topic), instead we can associate the user/group + permission with a "tag" (we can also create "deny" based policies associated with a "tag"). The "tag" itself contains the information about the resource that is being secured.
How does Apache Ranger obtain the relevant tags and associated entities? This is where Apache Atlas comes in. The
previous post described how to secure access to Apache Atlas using Apache Ranger. Apache Atlas allows you to associate "tags" with entities such as Kafka topics, Hive tables, etc. Apache Ranger provides a "tagsync" service which runs periodically and obtains the tags from Apache Atlas and uploads them to Apache Ranger. The Ranger authorization plugin for Kafka downloads the authorization policies, including tags, from the Ranger admin service and evaluates whether access is allowed or not based on the policy evaluation. Let's look at an example...
1) Start Apache Atlas and create entities/tags for Kafka
The first step is to start Apache Atlas as per the
previous tutorial. Note that we are not using the Apache Ranger authorization plugin for Atlas, so there is no need to follow step 2). Next we need to upload the Kafka entity of type "kafka_topic" that we are interested in securing. That can be done via the following command:
- curl -v -H 'Accept: application/json, text/plain, */*' -H 'Content-Type: application/json; charset=UTF-8' -u admin:admin -d @kafka-create.json http://localhost:21000/api/atlas/entities
where "kafka-create.json" is defined as:
Once this is done, log in to the admin console using credentials "admin/admin" at http://localhost:21000. Click on "Tags" and "Create Tag" called "KafkaTag". Next go to "Search" and search for the entity we have uploaded ("KafkaTest"). Click on the "+" button under "Tags" and associate the entity with the tag we have created.
2) Start Apache Ranger and create resource-based authorization policies for Kafka
Next we will follow the
first tutorial to install Apache Kafka and to get a simple test-case working with SSL authentication, but no authorization (there is no need to start Zookeeper as we already have Apache Atlas running, which starts a Zookeeper instance). Next follow the
third tutorial to install the Apache Ranger admin service, as well as the Ranger plugin for Kafka. Create ("resource-based") authorization policies for the Kafka "test" topic in Apache Ranger. There is just one thing we need to change, call the Ranger service "cl1_kafka" instead of "KafkaTest" (this change needs to happen in Ranger, and in the "install.properties" when installing the Ranger plugin to Kafka).
Now verify that the producer has permission to publish to the topic, and the consumer has permission to consume from the topic. Once this is working, then remove the resource-based policy for the consumer, and verify that the consumer no longer has permission to consume from the topic.
3) Use the Apache Ranger TagSync service to import tags from Atlas into Ranger
To create tag based policies in Apache Ranger, we have to import the entity + tag we have created in Apache Atlas into Ranger via the Ranger TagSync service. After building Apache Ranger then extract the file called "target/ranger-<version>-tagsync.tar.gz". There are three alternatives available where the Ranger TagSync service can obtain tag information. From Apache Atlas via a Kafka topic, from Apache Atlas via the REST API and from a file. We will use the REST API of Atlas here. Edit 'install.properties' as follows:
- Set TAG_SOURCE_ATLAS_ENABLED to "false"
- Set TAG_SOURCE_ATLASREST_ENABLED to "true"
- Set TAG_SOURCE_ATLASREST_DOWNLOAD_INTERVAL_IN_MILLIS to "60000" (just for testing purposes)
- Specify "admin" for both TAG_SOURCE_ATLASREST_USERNAME and TAG_SOURCE_ATLASREST_PASSWORD
Save 'install.properties' and install the tagsync service via "sudo ./setup.sh". It can now be started via "sudo ranger-tagsync-services.sh start".
4) Create Tag-based authorization policies in Apache Ranger
Now we can create tag-based authorization policies in Apache Ranger. Earlier we used the name "cl1_kafka" for the service name instead of "KafkaTest" as in the previous tutorial. The reason for this is that the service name must match the qualified name attribute of the Kafka entity that we are syncing into Ranger.
In Ranger, click on "Access Manager" and "Tag Based Policies". Create a new "TAG" service called "KafkaTagService". When this is done go into the new service and click on "Add New Policy". Hit (upper-case) "K" in the "TAG" field and "KafkaTag" should pop up automatically (hence the import of tags from Atlas was successful). Add an "allow condition" for the client user with permissions to "consume" and "describe" for "kafka" as shown in the following picture:
Finally, edit the "cl1_kafka" service we created and for "Select Tag Service" select "KafkaTagService" and save. Finally, wait some time for the Ranger plugin to download the new policies and tags and try the consumer again. This time it should work! So we have shown how Ranger can create authorization policies based on tags as well as resources.
