Securing Apache Hadoop Distributed File System (HDFS) - part IV

This is the fourth in a series of blog posts on securing HDFS. The first post described how to install Apache Hadoop, and how to use POSIX permissions and ACLs to restrict access to data stored in HDFS. The second post looked at how to use Apache Ranger to authorize access to data stored in HDFS. The third post looked at how Apache Ranger can create "tag" based authorization policies for HDFS using Apache Atlas. In this post I will look at how you can implement transparent encryption in HDFS using the Apache Ranger Key Management Service (KMS).

1) Install and Configure the Apache Ranger KMS

If you have not done so already, then follow the instructions in this tutorial to install the Apache Ranger admin service, and then start it via "sudo ranger-admin start". Open a browser and go to "http://localhost:6080/". Log on with "admin/admin" and click on "Settings". Create a new user corresponding to the name of the user which starts HDFS.

The next step is to install the Apache Ranger KMS. Please follow step (2) in a blog post I wrote last year about this. When installation is complete, then start the KMS service with "sudo ranger-kms start". Log out of the Admin UI and then log back in again with the credentials "keyadmin/keyadmin". Click on the "+" button on the "KMS" tab to create a new KMS Service. Specify the following values:

• Service Name: kmsdev
• KMS URL: kms://http@localhost:9292/kms
• Username: keyadmin
• Password: keyadmin

When the "kmsdev" service has been created then click on it and edit the default policy that has been created. Edit the existing "allow condition" for "hdfs" adding in the user that will be starting HDFS (if not the "hdfs" user itself). Also grant the "CREATE" permission to that user so that we can create keys from the command line, and the "DECRYPT EEK" permission, so that the user can decrypt the data encryption key:

2) Create an encryption zone in HDFS

In your Hadoop distribution (after first following the steps in the first post), edit 'etc/hadoop/core-site.xml' and add the following property:

• hadoop.security.key.provider.path - kms://http@localhost:9292/kms

Similarly, edit 'etc/hadoop/hdfs-site.xml' and add the following property:

• dfs.encryption.key.provider.uri - kms://http@localhost:9292/kms

Start HDFS via 'sbin/start-dfs.sh'. Let's create a new encryption key called "enckey" as follows:

• bin/hadoop key create enckey

If you go back to the Ranger Admin UI and click on "Encryption / Key Manager" and select the "kmsdev" service, you should be able to see the new key that was created. Now let's create a new encryption zone in HDFS as follows:

• bin/hadoop fs -mkdir /zone
• bin/hdfs crypto -createZone -keyName enckey -path /zone
• bin/hdfs crypto -listZones

That's it! We can put data into the '/zone' directory and it will be encrypted by a key which in turn is encrypted by the key we have created and stored in the Ranger KMS.