Securing an Apache Kafka broker - part IV

This is the fourth in a series of articles on securing an Apache Kafka broker. The first post looked at how to secure messages and authenticate clients using SSL. The second post built on the first post by showing how to perform authorization using some custom logic. The third post showed how Apache Ranger could be used instead to create and enforce authorization policies for Apache Kafka. In this post we will look at an alternative authorization solution called Apache Sentry.

1) Build the Apache Sentry distribution

First we will build and install the Apache Sentry distribution. Download Apache Sentry (1.7.0 was used for the purposes of this tutorial). Verify that the signature is valid and that the message digests match. Now extract and build the source and copy the distribution to a location where you wish to install it:

• tar zxvf apache-sentry-1.7.0-src.tar.gz
• cd apache-sentry-1.7.0-src
• mvn clean install -DskipTests
• cp -r sentry-dist/target/apache-sentry-1.7.0-bin ${sentry.home}

Apache Sentry has an authorization plugin for Apache Kafka, amongst other big data projects. In addition it comes with an RPC service which stores authorization privileges in a database. For the purposes of this tutorial we will just configure the authorization privileges in a configuration file locally to the broker. Therefore we don't need to do any further configuration to the distribution at this point.

2) Configure authorization in the broker

Configure Apache Kafka as per the first tutorial. To enable authorization using Apache Sentry we also need to follow these steps. First edit 'config/server.properties' and add:

• authorizer.class.name=org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer
• sentry.kafka.site.url=file:./config/sentry-site.xml

Next copy the jars from the "lib" directory of the Sentry distribution to the Kafka "libs" directory. Then create a new file in the config directory called "sentry-site.xml" with the following content:

	<?xml version="1.0"?>
	<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
	<configuration>
	<property>
	<name>sentry.service.security.mode</name>
	<value>none</value>
	</property>
	<property>
	<name>sentry.kafka.provider</name>
	<value>org.apache.sentry.provider.file.LocalGroupResourceAuthorizationProvider</value>
	</property>
	<property>
	<name>sentry.kafka.provider.backend</name>
	<value>org.apache.sentry.provider.file.SimpleFileProviderBackend</value>
	</property>
	<property>
	<name>sentry.kafka.provider.resource</name>
	<value>file:/home/colm/dist/apache/kafka_2.11-0.10.0.1/config/sentry.ini</value>
	</property>
	</configuration>

view raw sentry-site.xml hosted with ❤ by GitHub

This is the configuration file for the Sentry plugin for Kafka. It essentially says that the authorization privileges are stored in a local file, and that the groups for authenticated users should be retrieved from this file. Finally, we need to specify the authorization privileges. Create a new file in the config directory called "sentry.ini" with the following content:

	[groups]
	admin = admin_role
	producer = describe_role, read_role, write_role
	consumer = describe_role, read_role, describe_consumer_group_role, read_consumer_group_role
	[roles]
	admin_role = Host=*->Cluster=kafka-cluster->action=ALL
	describe_role = Host=*->Topic=test->action=describe
	read_role = Host=*->Topic=test->action=read
	write_role = Host=*->Topic=test->action=write
	describe_consumer_group_role = Host=*->ConsumerGroup=test-consumer-group->action=describe
	read_consumer_group_role = Host=*->ConsumerGroup=test-consumer-group->action=read
	[users]
	Broker = admin
	Service = producer
	Client = consumer

view raw sentry.ini hosted with ❤ by GitHub

This configuration file contains three separate sections. The "[users]" section maps the authenticated principals to local groups. The "[groups]" section maps the groups to roles, and the "[roles]" section lists the actual privileges. Now we can start the broker as in the first tutorial:

• bin/kafka-server-start.sh config/server.properties 

3) Test authorization

Now lets test the authorization logic. Start the producer:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to check that the producer is authorized correctly. Now start the consumer:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

If everything is configured correctly then it should work as in the first tutorial. 

Securing an Apache Kafka broker - part III

This is the third in a series of blog posts about securing Apache Kafka. The first post looked at how to secure messages and authenticate clients using SSL. The second post built on the first post by showing how to perform authorization using some custom logic. However, this approach is not recommended for non-trivial deployments. In this post we will show at how we can create flexible authorization policies for Apache Kafka using the Apache Ranger admin UI. Then we will show how to enforce these policies at the broker.

1) Install the Apache Ranger Kafka plugin

The first step is to download Apache Ranger (0.6.1-incubating was used in this post). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting plugin to a location where you will configure and install it:

• tar zxvf apache-ranger-incubating-0.6.1.tar.gz
• cd apache-ranger-incubating-0.6.1
• mvn clean package assembly:assembly -DskipTests
• tar zxvf target/ranger-0.6.1-kafka-plugin.tar.gz
• mv ranger-0.6.1-kafka-plugin ${ranger.kafka.home}

Now go to ${ranger.kafka.home} and edit "install.properties". You need to specify the following properties:

• COMPONENT_INSTALL_DIR_NAME: The location of your Kafka installation
• POLICY_MGR_URL: Set this to "http://localhost:6080"
• REPOSITORY_NAME: Set this to "KafkaTest".

Save "install.properties" and install the plugin as root via "sudo ./enable-kafka-plugin.sh". The Apache Ranger Kafka plugin should now be successfully installed (although not yet configured properly) in the broker.

2) Configure authorization in the broker

Configure Apache Kafka as per the first tutorial. There are a number of steps we need to follow to configure the Ranger Kafka plugin before it is operational:

• Edit 'config/server.properties' and add the following: authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
• Add the Kafka "config" directory to the classpath, so that we can pick up the Ranger configuration files: export CLASSPATH=$KAFKA_HOME/config
• Copy the Apache Commons Logging jar into $KAFKA_HOME/libs. 
• The ranger plugin will try to store policies by default in "/etc/ranger/KafkaTest/policycache". As we installed the plugin as "root" make sure that this directory is accessible to the user that is running the broker.

Now we can start the broker as in the first tutorial:

• bin/kafka-server-start.sh config/server.properties

3) Configure authorization policies in the Apache Ranger Admin UI 

At this point we should have configured the broker so that the Apache Ranger plugin is used to communicate with the Apache Ranger admin service to download authorization policies. So we need to install and configure the Apache Ranger admin service. Please refer to this blog post for how to do this. Assuming the admin service is already installed, start it via "sudo ranger-admin start". Open a browser and log on to "localhost:6080" with the credentials "admin/admin".

First lets add some new users that match the SSL principals we have created in the first tutorial. Click on "Settings" and "Users/Groups". Add new users for the principals:

• CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE
• CN=Service,O=Apache,L=Dublin,ST=Leinster,C=IE
• CN=Broker,O=Apache,L=Dublin,ST=Leinster,C=IE

Now go back to the Service Manager screen and click on the "+" button next to "KAFKA". Create a new service called "KafkaTest". Click "Test Connection" to make sure it can communicate with the Apache Kafka broker. Then click "add" to save the new service. Click on the new service. There should be an "admin" policy already created. Edit the policy and give the "broker" principal above the rights to perform any operation and save the policy. Now create a new policy called "TestPolicy" for the topic "test". Give the service principal the rights to "Consume, Describe and Publish". Give the client principal the rights to "Consum and Describe" only.

4) Test authorization

Now lets test the authorization logic. Bear in mind that by default the Kafka plugin reloads policies from the admin service every 30 seconds, so you may need to wait that long or to restart the broker to download the newly created policies. Start the producer:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to check that the producer is authorized correctly. Now start the consumer:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

If everything is configured correctly then it should work as in the first tutorial.

Integrating Apache Camel with Apache Syncope - part III

This is the third in a series of blog posts about integrating Apache Camel with Apache Syncope. The first post introduced the new Apache Camel provisioning manager that is available in Apache Syncope 2.0.0, and gave an example of how we can modify the default behaviour to send an email to an administrator when a user was created. The second post showed how an administrator can keep track of user password changes for auditing purposes. In this post we will show how to integrate Syncope with Apache ActiveMQ using Camel.

1) The use-case

The use-case is that Apache Syncope is used for Identity Management in a large organisation. When users are created we would like to be able to gather certain information about the new users and process it dynamically in some way. In particular, we are interested in the age of the new users and the country in which they are based. Perhaps at the reception desk of the company HQ we display a map with the number of employees in each country highlighted. To decouple whatever applications are processing the data from Syncope itself, we will use a messaging solution, namely Apache ActiveMQ. When new users are created, we will modify the default Camel route to send a message to two topics corresponding to the age and location of the user.

2) Download and configure Apache ActiveMQ

The first step is to download Apache ActiveMQ (currently 5.14.0). Unzip it and start it via:

• bin/activemq start 

Now go to the web interface of ActiveMQ - 'http://localhost:8161/admin/', logging in with credentials 'admin/admin'. Click on the "Queues" tab and create two new queues called 'age' and 'country'.

3) Download and configure Apache Syncope

Download and extract the standalone version of Apache Syncope 2.0.0. Before we start it we will copy the jars we need to get Camel working with ActiveMQ in Syncope. In the "webapps/syncope/WEB-INF/lib" directory of the Apache Tomcat instance bundled with Syncope, copy the following jars:

• From $ACTIVEMQ_HOME/lib: activemq-client-5.14.0.jar + activemq-spring-5.14.0.jar + hawtbuf-1.11.jar + geronimo-j2ee-management_1.1_spec-1.0.1.jar
• From $ACTIVEMQ_HOME/lib/camel: activemq-camel-5.14.0.jar + camel-jms-2.16.3.jar
• From $ACTIVEMQ_HOME/lib/optional: activemq-pool-5.14.0.jar + activemq-jms-pool-5.14.0.jar + spring-jms-4.1.9.RELEASE.jar

Next we need to create a Camel spring configuration file containing a bean with the address of the broker. Add the following file to the Tomcat lib directory (called "camelRoutesContext.xml"):

	<beans
	xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
	http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
	<bean id="activemq"
	class="org.apache.activemq.camel.component.ActiveMQComponent">
	<property name="brokerURL" value="tcp://localhost:61616"/>
	</bean>
	</beans>

view raw camelRoutesContext.xml hosted with ❤ by GitHub

Now we can start the embedded Apache Tomcat instance. Open a browser and navigate to 'http://localhost:9080/syncope-console' logging in with 'admin/password'. The first thing we need to do is to configure user attributes for "age" and "country". Go to "Configuration/Types" in the left-hand menu, and click on the "Schemas" tab. Create two plain (mandatory) schema types: "age" of type "String" and "country" of type "Long". Now click on the "AnyTypeClasses" tab and create a new AnyTypeClass selecting the two plain schema types we just created. Finally, click on the "AnyType" tab and edit the "USER". Add the new AnyTypeClass you created and hit "save".

Now we will modify the Camel route invoked when a user is created. Click on "Extensions/Camel Routes" in the left-hand configuration menu. Edit the "createUser" route and add the following above the "bean method" part:

• <setBody><simple>${body.plainAttrMap[age].values[0]}</simple></setBody>
• <to uri="activemq:age"/>
• <setBody><simple>${exchangeProperty.actual.plainAttrMap[country].values[0]}</simple></setBody>
• <to uri="activemq:country"/>

This should be fairly straightforward to follow. We are setting the message body to be the age of the newly created User, and dispatching that message to the "age" queue. We then follow the same process for the "country". We also need to change "body" in the "bean method" line to "exchangeProperty.actual", this is because we have redefined what the body is for each of the Camel routes above.

Now let's create some new users. Click on the "Realms" menu and select the "USER" tab. Create new users "alice" in country "usa" of age "25" and "bob" in country "canada" of age "27". Now let's look at the ActiveMQ console again. We should see two new messages in each of the queues as follows, ready to be consumed:

Using SHA-512 with Apache CXF SOAP web services

XML Signature is used extensively in SOAP web services to guarantee message integrity, non-repudiation, as well as client authentication via PKI. A digest algorithm crops up in XML Signature both as part of the Signature Method (rsa-sha1 for example), as well as in the digests of the data that are signed. As recent weaknesses have emerged with the use of SHA-1, it makes sense to use the SHA-2 digest algorithm instead. In this post we will look how to configure Apache CXF to use SHA-512 (i.e. SHA-2 with 512 bits) as the digest algorithm.

1) Configuring the STS to use SHA-512

Apache CXF ships with a SecurityTokenService (STS) that is widely deployed. The principal function of the STS is to issue signed SAML tokens, although it supports a wide range of other functionalities and token types. The STS (for more recent versions of CXF) uses RSA-SHA256 for the signature method when signing SAML tokens, and uses SHA-256 for the digest algorithm. In this section we'll look at how to configure the STS to use SHA-512 instead.

You can specify signature and digest algorithms via the SignatureProperties class in the STS. To specify SHA-512 for signature and digest algorithms for generated tokens in the STS add the following bean to your spring configuration:

	<bean id="sigProps" class="org.apache.cxf.sts.SignatureProperties">
	<property name="signatureAlgorithm"
	value="http://www.w3.org/2001/04/xmldsig-more#rsa-sha512" />
	<property name="digestAlgorithm"
	value="http://www.w3.org/2001/04/xmlenc#sha512" />
	</bean>

view raw gistfile1.txt hosted with ❤ by GitHub

Next you need to reference this bean in the StaticSTSProperties bean for your STS:

• <property name="signatureProperties" ref="sigProps" />

2) Configuring WS-SecurityPolicy to use SHA-512

Service requests are typically secured at a message level using WS-SecurityPolicy. It is possibly to specify the algorithms used to secure the request, as well as the key sizes, by configuring an AlgorithmSuite policy. Unfortunately the last WS-SecurityPolicy spec is quite dated at this point, and lacks support for more modern algorithms as part of the default AlgorithmSuite policies that are defined in the spec. The spec only supports using RSA-SHA1 for signature, and only SHA-1 and SHA-256 for digest algorithms.

Luckily, Apache CXF users can avail of a few different ways to use stronger algorithms with web service requests. In CXF there is a JAX-WS property called 'ws-security.asymmetric.signature.algorithm' for AsymmetricBinding policies (similarly 'ws-security.symmetric.signature.algorithm' for SymmetricBinding policies). This overrides the default signature algorithm of the policy. So for example, to switch to use RSA-SHA512 instead of RSA-SHA1 simply set the following property on your client/endpoint:

• <entry key="ws-security.asymmetric.signature.algorithm" value="http://www.w3.org/2001/04/xmldsig-more#rsa-sha512"/>

There is no corresponding property to explicitly configure the digest algorithm, as the default AlgorithmSuite policies already support SHA-256 (although one could be added if there was enough demand). If you really need to support SHA-512 here, an option is to use a custom AlgorithmSuite (which will obviously not be portable), or to override one of the existing ones.

It's pretty straightforward to do this. First you need to create an AlgorithmSuiteLoader implementation to handle the policy. Here is one used in the tests that creates a custom AlgorithmSuite policy called 'Basic128RsaSha512', which extends the 'Basic128' policy to use RSA-SHA512 for the signature method, and SHA-512 for the digest method. This AlgorithmSuiteLoader can be referenced in Spring via:

	<bean id="policyLoader" class="org.apache.cxf.systest.ws.x509.SHA512PolicyLoader" >
	<constructor-arg ref="cxf"/>
	</bean>

view raw gistfile1.txt hosted with ❤ by GitHub

The policy in question looks like:

• <cxf:Basic128RsaSha512 xmlns:cxf="http://cxf.apache.org/custom/security-policy"/>

Invoking on the Talend ESB STS using SoapUI

Talend ESB ships with a powerful SecurityTokenService (STS) based on the STS that ships with Apache CXF. The Talend Open Studio for ESB contains UI support for creating web service clients that use the STS to obtain SAML tokens for authentication (and also authorization via roles embedded in the tokens). However, it is sometimes useful to be able to obtain tokens with a third party client. In this post we will show how SoapUI can be used to obtain SAML Tokens from the Talend ESB STS.

1) Download and run Talend Open Studio for ESB

The first step is to download Talend Open Studio for ESB (the current version at the time of writing this post is 6.2.1). Unzip it and start the container via:

• Runtime_ESBSE/container/bin/trun

The next step is to start the STS itself:

• tesb:start-sts

2) Download and run SoapUI

Download SoapUI and run the installation script. Create a new SOAP Project called "STS" using the WSDL:

• http://localhost:8040/services/SecurityTokenService/UT?wsdl

The WSDL of the STS defines a number of different services. The one we are interested in is the "UT_Binding", which requires a WS-Security UsernameToken to authenticate the client. Click on "UT_Binding/Issue/Request 1" in the left-hand menu to see a sample request for the service. Now we need to do some editing of the request. Remove the 'Context="?"' attribute from RequestSecurityToken. Then paste the following into the Body of the RequestSecurityToken:

• <t:TokenType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0</t:TokenType>
• <t:KeyType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/ws-sx/ws-trust/200512/Bearer</t:KeyType>
• <t:RequestType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</t:RequestType>

Now we need to configure a username and password to use when authenticating the client request. In the "Request Properties" box in the lower left corner, add "tesb" for the "username" and "password" properties. Now right click in the request pane, and select "Add WSS Username Token" (Password Text). Now send the request and you should receive a SAML Token in response.

Bear in mind that if you wish to re-use the SAML Token retrieved from the STS in a subsequent request, you must copy it from the "Raw" tab and not the "XML" tab of the response. The latter adds in whitespace that breaks the signature on the token. Another thing to watch out for is that the STS maintains a cache of the Username Token nonce values, so you will need to recreate the UsernameToken each time you want to get a new token.

3) Requesting a "PublicKey" KeyType

The example above uses a "Bearer" KeyType. Another common use-case, as is the case with the security-enabled services developed using the Talend Studio, is when the token must have the PublicKey/Certificate of the client embedded in it. To request such a token from the STS, change the "Bearer" KeyType as above to "PublicKey". However, we also need to present a certificate to the STS to include in the token.

As we are just using the test credentials used by the Talend STS, go to the Runtime_ESBSE/container/etc/keystores and extract the client key with:

• keytool -exportcert -rfc -keystore clientstore.jks -alias myclientkey -file client.cer -storepass cspass

Edit client.cer + remove the first and end lines (that contain BEGIN/END CERTIFICATE). Now go back to SOAP-UI and add the following to the RequestSecurityToken Body:

• <t:UseKey xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>...</ds:X509Certificate></ds:X509Data></ds:KeyInfo></t:UseKey>

where the content of the X.509 Certificate is the content in client.cer. This time, the token issued by the STS will contain the public key of the client embedded in the SAML Subject.

Securing an Apache Kafka broker - part II

In the previous post, we looked at how to configure an Apache Kafka broker to require SSL client authentication. In this post we will add authorization to the example, making sure that only authorized producers can send messages to the broker. In addition, we will show how to enforce authorization rules per-topic for consumers.

1) Configure authorization in the broker

Configure Apache Kafka as per the previous tutorial. To enforce some custom authorization rules in Kafka, we will need to implement the Kafka Authorizer interface. This interface contains an "authorize" method, which supplies a Session Object, where you can obtain the current principal, as well as the Operation and Resource upon which to enforce an authorization decision.

In terms of the example detailed in the previous post, we created broker, service (producer) and client (consumer) principals. We want to enforce authorization decisions as follows:

• Let the broker principal do anything
• Let the producer principal read/write on all topics
• Let the consumer principal read/describe only on topics starting with "test".

There is a sample Authorizer implementation available in some Kafka unit test I wrote in github that can be used in this example - CustomAuthorizer:

	@Override
	public boolean authorize(Session arg0, Operation arg1, Resource arg2) {
	if (arg0.principal() == null) {
	return false;
	}
	String principal = arg0.principal().getName();
	if (principal.startsWith("CN=Client")
	&& ("Read".equals(arg1.name()) || "Describe".equals(arg1.name()))
	&& arg2.name().startsWith("test")) {
	return true;
	} else if (principal.startsWith("CN=Service") && ("Read".equals(arg1.name()) || "Describe".equals(arg1.name())
	|| "Write".equals(arg1.name()) || "Create".equals(arg1.name()))) {
	return true;
	} else if (principal.startsWith("CN=Broker")) {
	return true;
	}
	return false;
	}

view raw gistfile1.txt hosted with ❤ by GitHub

Next we need to package up the CustomAuthorizer in a jar so that it can be used in the broker. You can do this by checking out the testcases github repo, and invoking "mvn clean package jar:test-jar -DskipTests" in the "apache/bigdata/kafka" directory. Now copy the resulting test jar in "target" to the "libs" directory in your Kafka installation. Finally, edit the "config/server.properties" file and add the following configuration item:

• authorizer.class.name=org.apache.coheigea.bigdata.kafka.CustomAuthorizer

2) Test authorization

Now lets test the authorization logic. Restart the broker and the producer:

• bin/kafka-server-start.sh config/server.properties
• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to check that the producer is authorized correctly. Now start the consumer:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

If everything is configured correctly then it should work as in the first tutorial. Now we will create a new topic called "messages":

• bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic messages

Restart the producer to send messages to "messages" instead of "test". This should work correctly. Now try to consume from "messages" instead of "test". This should result in an authorization failure, as the "client" principal can only consume from the "test" topic according to the authorization rules.

Securing an Apache Kafka broker - part I

Apache Kafka is a messaging system for the age of big data, with a strong focus on reliability, scalability and message throughput. This is the first part of a short series of posts on how to secure an Apache Kafka broker. In this post, we will focus on authenticating message producers and consumers using SSL. Future posts will look at how to authorize message producers and consumers.

1) Create SSL keys

As we will be securing the broker using SSL client authentication, the first step is to create some keys for testing purposes. Download the OpenSSL ca.config file used by the WSS4J project. Change the "certificate" value to "ca.pem", and the "private_key" value to "cakey.pem". You will also need to create a directory called "ca.db.certs", and make an empty file called "ca.db.index". Now create a new CA key and cert via:

• openssl req -x509 -newkey rsa:1024 -keyout cakey.pem -out ca.pem -config ca.config -days 3650

Just accept the default options. Now we need to convert the CA cert into jks format:

• openssl x509 -outform DER -in ca.pem -out ca.crt
• keytool -import -file ca.crt -alias ca -keystore truststore.jks -storepass security

Now we will create the client key, sign it with the CA key, and put the signed client cert and CA cert into a keystore:

• keytool -genkey -validity 3650 -alias myclientkey -keyalg RSA -keystore clientstore.jks -dname "CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE" -storepass cspass -keypass ckpass
• keytool -certreq -alias myclientkey -keystore clientstore.jks -file myclientkey.cer -storepass cspass -keypass ckpass
• echo 20 > ca.db.serial
• openssl ca -config ca.config -policy policy_anything -days 3650 -out myclientkey.pem -infiles myclientkey.cer
• openssl x509 -outform DER -in myclientkey.pem -out myclientkey.crt
• keytool -import -file ca.crt -alias ca -keystore clientstore.jks -storepass cspass
• keytool -import -file myclientkey.crt -alias myclientkey -keystore clientstore.jks -storepass cspass -keypass ckpass

Now follow the same template to create a "service" key in servicestore.jks, with store password "sspass" and key password "skpass". In addition, we will create a "broker" key in brokerstore.jks, with storepass "bspass" and key password "bkpass".  

2) Configure the broker

Download Apache Kafka and extract it (0.10.0.1 was used for the purposes of this tutorial). Copy the keys created in section "1" into $KAFKA_HOME/config. Start Zookeeper with:

• bin/zookeeper-server-start.sh config/zookeeper.properties

Now edit 'config/server.properties' and add the following:

• ssl.keystore.location=./config/brokerstore.jks
• ssl.keystore.password=bspass
• ssl.key.password=bkpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security
• security.inter.broker.protocol=SSL
• ssl.client.auth=required
• listeners=SSL://localhost:9092

and start the broker and then create a "test" topic with:

• bin/kafka-server-start.sh config/server.properties
• bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

3) Configure the producer

Now we will configure the message producer. Edit 'config/producer.properties' and add the following:

• security.protocol=SSL
• ssl.keystore.location=./config/servicestore.jks
• ssl.keystore.password=sspass
• ssl.key.password=skpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security

and start the producer with:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to the topic to make sure that everything is working ok.

4) Configure the consumer

Finally we will configure the message consumer. Edit 'config/consumer.properties' and add the following:

• security.protocol=SSL
• ssl.keystore.location=./config/clientstore.jks
• ssl.keystore.password=cspass
• ssl.key.password=ckpass
• ssl.truststore.location=./config/truststore.jks
• ssl.truststore.password=security

and start the consumer with:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

The messages sent by the producer should appear in the console window of the consumer. So there it is, we have configured the Kafka broker to require SSL client authentication. In the next post we will look at adding simple authorization support to this scenario.

Integrating Apache Camel with Apache Syncope - part II

A recent blog post introduced the new Apache Camel provisioning manager that is available in Apache Syncope 2.0.0. It also covered a simple use-case for the new functionality, where the "createUser" Camel route is modified to send an email to an administrator when a User is created, with some details about the created User in the email. In this post, we will look at a different use-case, where the Camel provisioning manager is used to extend the functionality offered by Syncope.

1) The use-case

Apache Syncope stores users in internal storage in a table called "SyncopeUser". This table contains information such as the User Id, name, password, creation date, last password change date, etc. In addition, if there is an applicable password policy associated with the User realm, a list of the previous passwords associated with the User is stored in a table called "SyncopeUser_passwordHistory":

As can be seen from this screenshot, the table stores a list of Syncope User Ids with a corresponding password value. The main function of this table is to enforce the password policy. So for example, if the password policy states that a user can't change back to a password for at least 10 subsequent password changes, this table provides the raw information that is needed to enforce the policy.

Now, what if the administrator wants a stronger audit trail for User password changes other than what is provided by default in the Syncope internal storage? In particular, the administrator would like a record of when the User changes a password. The "SyncopeUser" table only stores the last password change date. There is no way of seeing when each password stored in the "SyncopeUser_passwordHistory" table was changed. Enter the Camel provisioning manager...

2) Configure Apache Syncope

Download and install  Apache Syncope (I used the "standalone" download for the purposes of this demo). Start Apache Syncope and log on to the admin console. First we will create a password policy. Click on "Configuration" and then "Policies". Click on the "Password" tab and then select the "+" button to create a new policy. Give a description for the policy and then select a history length of "10".

Now let's create a new user in the default realm. Click on "realms" and hit the "edit" button. Select the password policy you have just created and click "Finish". This means that all users created in this realm will have the applicable policy applied to them. Now click on the "User" tab and the "+" button to add a new user called "alice". Click on "Password Management" and enter a password for "alice". Now we have a new user created, we want to be able to see when she updates her password from now on.

Click on "Extensions" and "Camel Routes". Find the "updateUser" route (might not be on the first page) and edit it. Create a new Camel "filter" (as per the screenshot below) just above the "bean method=" line with the following content:

• <simple>${body.password} != null</simple>
• <setHeader headerName="CamelFileName"><simple>${body.key}.passwords</simple></setHeader>
• <setBody><simple>New password '${body.password.value}' changed at time '${date:now:yyyy-MM-dd'T'HH:mm:ss.SSSZ}'\n</simple></setBody>
• <to uri="file:./?fileExist=Append"/>

So what we are doing here is to audit the password changes for a particular user, by writing out the password + Timestamp to a file associated with that user. Let's examine what each of these statements do in turn. The first statement is the filter condition. It states that we should execute the following statements if the password is not null. The password will only be non null if it is being changed. So for example, if the user just changes a given attribute and not the password, the filter will not be invoked.

The second statement sets the Camel Header "CamelFileName" to the format of "<user.id>.passwords". This header is used by the Camel File component as the file name to write out to. The third statement sets the exchange Body (the file content) to show the password value along with a Timestamp. Finally, the fourth statement is an invocation of the Camel File component, which appends the exchange Body to the given file name. As we have overridden the message Body in the second statement above, we need to change the ${body} in the create call to ${exchangeProperty.actual}, which is the saved Body. Click on "save" to save the modified route.

Now let's update the user "alice" and set a new password a couple of times. There should be a new file in the directory where you launched Tomcat containing the audit log for password changes for that particular user. With the power of Apache Camel, we could audit to a database, to Apache Kafka, etc etc.

Apache CXF Fediz 1.2.3 and 1.3.1 released

Apache CXF Fediz 1.2.3 and 1.3.1 have been released. The 1.3.1 release contains the following significant features/fixes:

• An update to use Apache CXF 3.1.7 
• Support for Facebook Login as a Trusted IdP.
• A fix for SAML SSO redirection on ForceAuthn or token expiry.
• A bug fix to support multiple realms in the IdP.
• A fix to enforce that mandatory claims are present in the received token.

In addition, both 1.2.3 and 1.3.1 contain a fix for a new security advisory - CVE-2016-4464:

Apache CXF Fediz is a subproject of Apache CXF which implements the WS-Federation Passive Requestor Profile for SSO specification. It provides a number of container based plugins to enable SSO for Relying Party applications. It is possible to configure a list of audience URIs for the plugins, against which the AudienceRestriction values of the received SAML tokens are supposed to be matched. However, this matching does not actually take place.

This means that a token could be accepted by the application plugin (assuming that the signature is trusted) that is targeted for another service, something that could potentially be exploited by an attacker.