SAML SSO support for the Apache CXF Fediz plugins

Apache CXF Fediz originated as a way of securing web applications using Single Sign-On via the WS-Federation Passive Requestor Profile. Plugins were written to support the most popular web application containers, such as Apache Tomcat, Jetty, Spring, Websphere, etc. Fediz then shipped an IdP which could be used to perform authentication using the container plugins. For the 1.3.0 release, support was added to the IdP to also support the SAML SSO protocol. From the next 1.4.4 release, the Tomcat 8 plugin can authenticate to a SAML SSO IdP, instead of using WS-Federation, with a few simple configuration changes. This makes it very easy to upgrade your Fediz-secured containers to use SAML SSO instead of WS-Federation.

In this article, we will secure the 'fedizhelloworld' application example that ships with Fediz, which is deployed in Apache Tomcat, using Keycloak as the SAML SSO IdP. We will show how to deploy and secure the application both manually and also by a docker image that I have created for quick deployment.

1) Download and configure Keycloak

Download and install the latest Keycloak distribution (tested with 3.4.3).

1.1) Create users in Keycloak

Start keycloak in standalone mode by running 'sh bin/standalone.sh'. First we need to create an admin user by navigating to the following URL, and entering a password:

• http://localhost:8080/auth/

Click on the "Administration Console" link, logging on using the admin user credentials. You will see the configuration details of the "Master" realm. For the purposes of this demo, we will create a new realm. Hover the mouse pointer over "Master" in the top left-hand corner, and click on "Add realm". Create a new realm called "fediz-samlsso". Now we will create a new user in this realm. Click on "Users" and select "Add User", specifying "alice" as the username. Click "save" and then go to the "Credentials" tab for "alice", and specify a password, unselecting the "Temporary" checkbox, and reset the password.

1.2) Create a new client application in Keycloak

Now we will create a new client application for 'fedizhelloworld' in Keycloak. Select "Clients" in the left-hand menu, and click on "Create". Specify the following values:

• Client ID: urn:org:apache:cxf:fediz:fedizhelloworld
• Client protocol: saml
• Client SAML Endpoint: https://localhost:9443/fedizhelloworld/secure

Once the client is created you will see more configuration options:

• Select "Sign Assertions"
• Select "Force Name ID Format".
• Valid Redirect URIs: https://localhost:9443/*

Click 'Save'. Now go to the "SAML Keys" tab of the newly created client. Here we will have to import the certificate of the Fediz RP so that Keycloak can validate the signed SAML requests. Click "Import" and specify:

• Archive Format: JKS
• Key Alias: mytomrpkey
• Store password: tompass
• Import file: rp-ssl-key.jks

1.3) Export the Keycloak signing certificate

Finally, we need to export the Keycloak signing certificate so that the Fediz plugin can validate the signed SAML Response from Keycloak. Select "Realm Settings" (for "fediz-samlsso") and click on the "Keys" tab. Copy and save the value specified in the "Certificate" textfield. 

2) Manually configure 'fedizhelloworld' application in Apache Tomcat

In this section, we'll look at manually configuring the 'fedizhelloworld' application in Apache Tomcat. To use a docker image skip to the next section. Download and extract Apache Tomcat 8 (tested with 8.5.32) to ${catalina.home}. Download Fediz 1.4.4 and build the source with "mvn clean install -DskipTests". Copy 'apache-fediz/target/apache-fediz-1.4.4' to a new directory (${fediz.home}).

2.1) Secure the Apache Tomcat container with the Fediz plugin

First we will secure the Apache Tomcat container with the Fediz plugin:

• Create a new directory: ${catalina.home}/lib/fediz
• Edit ${catalina.home}/conf/catalina.properties and append ',${catalina.home}/lib/fediz/*.jar' to the 'common.loader' property.
• Copy ${fediz.home}/plugins/tomcat8/lib/* to ${catalina.home}/lib/fediz
• Edit the TLS Connector in ${catalina.home}/conf/server.xml', and change the ports to avoid conflict with Keycloak, i.e. switch 8080 to 9080, 8443 to 9443, etc.
• In the same file, add configuration for the TLS port: <Connector port="9443" protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="150" SSLEnabled="true" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" keystoreFile="rp-ssl-key.jks" keystorePass="tompass" />

2.2) Deploy 'fedizhelloworld' to Tomcat

• Do a "mvn clean install" in ${fediz.home}/examples/simpleWebapp
• Copy ${fediz.home}/examples/simpleWebapp/target/fedizhelloworld.war to ${catalina.home}/webapps.
• Copy ${fediz.home}/examples/samplekeys/rp-ssl-key.jks to ${catalina.home}.
• Copy ${fediz.home}/examples/simpleWebapp/src/main/config/fediz_config.xml to ${catalina.home}/conf/

2.3) Configure 'fediz_config.xml'
 
Now we need to configure '${catalina.home}/conf/fediz_config.xml' that we copied from the Fediz example in the previous section:

• Under 'contextConfig', specify the key we are using to sign the SAML Request: <signingKey keyAlias="mytomrpkey" keyPassword="tompass">
              <keyStore file="rp-ssl-key.jks" password="tompass" type="JKS" />
          </signingKey>
• Change the trustManager keystore to: <keyStore file="keycloak.cert" type="PEM" />
• In the 'Protocol' section, change 'federationProtocolType' to 'samlProtocolType'.
• Change the 'Issuer' value to: http://localhost:8080/auth/realms/fediz-samlsso/protocol/saml
• Under 'Protocol' add: <signRequest>true</signRequest and <disableDeflateEncoding>true</disableDeflateEncoding>

Finally, create a file called 'keycloak.cert' in ${catalina.home}. In between "-----BEGIN CERTIFICATE----- / -----END CERTIFICATE-----" tags, paste the Keycloak signing certificate as retrieved in step "1.3" above. 

3) Using a docker image to deploy 'fedizhelloworld'

I have created a simple docker project which can be used to package and deploy the 'fedizhelloworld.war' into Apache Tomcat, which is secured with Fediz. The project is available on github here. Clone the project and build and run with the following steps:

• Edit 'keycloak.cert' and paste in the signing certificate as retrieved in step 1.3 above.
• Build with: docker build -t coheigea/fediz-samlsso-rp .
• Run with: docker run -p 9443:8443 coheigea/fediz-samlsso-rp

At the time of writing, the docker file references a SNAPSHOT build of Fediz, as 1.4.4 is not yet released.

4) Testing the service

To test the service navigate to:

• https://localhost:9443/fedizhelloworld/secure/fedservlet

You should be redirected to the Keycloak authentication page. Enter the user credentials you have created, and you will be redirected back to the 'fedizhelloworld' application successfully.

Securing web services using Talend's Open Studio for ESB - part I

This is the first part in a series of posts on securing web services using Talend's Open Studio for ESB. Talend's Open Studio for ESB is a freely available tool that comprises an eclipse-based studio to design and test web services, as well as a runtime based on Apache Karaf which can be used to deploy the web services built in the studio. In this post we will show how a simple SOAP web service can be designed and tested in the Studio.

1) Download and start the Studio and design a simple SOAP web service

Firstly download and extract Talend's Open Studio for ESB. Version 7.0.1 was used for the purpose of this blog post. The "Runtime_ESBSE" directory contains the runtime container, and the "Studio" directory the Open Studio for ESB. Launch the Studio and create a new project.

First we will create a simple SOAP "double-it" web service using the Studio. Right click on "Services" in the left-hand menu and then "Create Service". Call the service "DoubleIt" and click "Next" to create a new WSDL. The Studio then displays the design of the service, and the WSDL can be seen if you click on the "Source" tab. We will make a few changes to the default service.

Click on "DoubleItPortType" and change the operation name from "DoubleItOperation" to "DoubleIt". Next click on the right arrow next to "DoubleItRequest" and change the request type from "string" to "int". Do the same for the Response type. Now save the service and we are ready to move on to the next step.

2) Implement the "DoubleIt" service we have designed

After having designed the service above, now we need to implement the service by assigning a job to it. Right click on the "DoubleIt" service we have created ("Services/DoubleIt 0.1/DoubleItPortType 0.1/DoubleIt 0.1" in the left-hand menu) and select "Assign Job", and click "Next" to create a new job for this service. Now under "Job Designs" in the left-hand menu we see a new job has been created and the main window has been updated with tESBProviderRequest and tESBProviderResponse components.

Drag the tESBProviderResponse component over to the right hand side of the window. Now we need to think about how to handle the service logic. In our service, we want to take an input number, double it, and assign it to an output number. A Talend component available in the Studio that allows us to map XML is the "tXMLMap" component.

Find the "tXMLMap" component in the palette on the right hand side (under "XML"), and drag it onto the main window in between the two existing components. Now right click on tESBProviderRequest and select "Row" and "Main", and map the arrow onto the tXMLMap component. Do the same from tXMLMap to tESBProviderResponse, giving a name "Response" for the output when prompted.

Next we need to configure the "tXMLMap" component to implement the mapping logic. Double click on "tXMLMap". The input request is available on the left hand side, with a "root" payload. We want to map the request payload to the response payload. Hold the left mouse button down on the left-hand side over the request "root" payload, and move the mouse over to the right hand side and release on the response "root" payload, selecting "Add linker to target node".

Before we try to implement the "doubling" logic, we need to change the payload type from "String" to "int". Click on the "Tree Schema Editor" tab at the bottom of the screen and change both the request and response payload types to "int". Back up on the Response tab, click on "[row1.payload:/root]" and edit it to be "2 * [row1.payload:/root]".

Finally, we need to change the request/response elements to conform to the WSDL. Right-click on "root" in the left-hand request column and rename it to "ns2:DoubleItRequest". When this is done, right click again on this element and "Set a Namespace", with the namespace "http://www.talend.org/service/" and prefix "ns2". Similarly, on the right hand side, rename "root" to "ns2:DoubleItResponse", and set the namespace in the same way as for the request. Now click "Apply" and save the job.

Now click on the "Run" tab at the bottom of the screen and run the service. The service should now be deployed at "http://localhost:8090/services/DoubleIt".

3) Implement a client for the "DoubleIt" service

As well as designing, creating and testing web services, Talend Open Studio for ESB can also be used to create clients for these web services. To do this we need a new job. Right click on "Job Designs" in the left-hand menu and create a new job. Drag the "tESBConsumer" component to the main screen, as well as two "tLogRow" components. Right click on "tESBConsumer" and select "Row/Response" and drag the arrow to the first tLogRow component. In addition, select "Row/Fault" and drag the arrow to the second tLogRow component. This way we are logging both the response + faults from the remote service.

Left-click on "tESBConsumer" and specify "http://localhost:8090/services/DoubleIt?wsdl" as the WSDL location (the WSDL of our service is available at this address due to WSDL publish). Then click the "reload" button on the right-hand side and click "Finish". Next we need to implement the client logic - namely, to supply a number to double. Drag the "tFixedFlowInput" and "tXMLMap" components to the screen. Map "Row/Main" from "tFixedFlowInput" to "tXMLMap", and "Row" from "tXMLMap" to "tESBConsumer" with a new output name of "Request".

Now click on "tFixedFlowInput" and "Edit Schema". Add a new column of type "int" called "numberToDouble". Back in the component screen for "tFixedFlowInput" select "Use inline table" and enter a number (e.g. 200). Now click on "tXMLMap" to configure our mapping. Drag "numberToDouble" on the left-hand side over to "root" on the right-hand side, selecting "Add linker to target node". Right click on "root" on the right-hand side, and rename it to "ns2:DoubleItRequest", and again "Set a Namespace" with namespace "http://www.talend.org/service/" and prefix "ns2".

Click OK and save the job, and run it via the "Run" tab. In the console window we should see the response from the service, informing us that 200 doubled is "400". The job is also updated so that you can see the flow along with the throughput:

In the next tutorial we'll look at how to deploy our service and client jobs in the runtime container.

Streaming WS-Security MTOM support in Apache CXF

Apache CXF 3.0.0 introduced the new streaming (StAX-based) WS-security implementation via new functionality available in the core libraries - Apache WSS4J 2.0.0 and Apache Santuario 2.0.0. The StAX-based approach is more limited than the older DOM-based alternative, and in general slightly slower. However it can come into its own when sending or processing very large documents due to the low memory footprint of the library.

In addition, support was added in Apache CXF 3.2.0 for the DOM library to send and process WS-Security messages using MTOM. Essentially what this means is that we can compress WS-Security secured SOAP messages, by storing binary content in message attachments, instead of inlining them in the message (via BASE-64 encoding). When MTOM is enabled, Apache CXF will automatically use this functionality for WS-Security. However up until now, this functionality has not been available for the streaming WS-Security library.

This is set to change in Apache CXF 3.2.5. Support has been added in Apache Santuario to process CipherValue message elements in the streaming XML Security code that contain a "xop:Include" reference to a message attachment. Some fixes in Apache WSS4J build on this support, also adding support for processing BinarySecurityToken Elements that include 'xop:Include' instead of the inlined bytes. Both of these sets of changes are supported in Apache CXF in the following JIRA.

What this means is that Apache CXF 3.2.5 onwards will be able to process WS-Security enabled SOAP messages over MTOM. Please note however that support is limited to processing messages. The streaming code still inlines message bytes on the outbound side, unlike the DOM implementation. This could perhaps be implemented in the future if there is sufficient demand.

Running the Apache Ranger Admin service 1.0.0 in Docker

Apache Ranger 1.0.0 has been recently released after a long development cycle, featuring a huge number of improvements and bug fixes. A previous blog post covered how to manually install the Apache Ranger admin service, by compiling the Apache Ranger source and using MySQL as the database. However this involves a large number of steps, as well as installing MySQL, Apache Maven, Java, etc. In this post we will show how Docker Compose can be used to easily set up the Apache Ranger 1.0.0 Admin Service.

1) Description

The project is available in my github testcases repository here. This project is provided as a quick and easy way to play around with the Apache Ranger admin service. It should not be deployed in production as it uses default security credentials, it is not secure with kerberos, auditing is not enabled, etc. It contains the configuration required to build two Docker images:

• ranger-postgres: Contains a Docker File to set up a Postgres database for Apache Ranger, creating the necessary users for the Ranger admin installation scripts to work.
• ranger-admin: Contains a Docker File to build, configure and install the Apache Ranger admin service. It downloads the Apache Ranger source code, and builds and extracts the Admin service. It configures it to use the postgres database and starts the Admin service when the docker image is started.

2) Building and running

First we need to build the docker images. This can be done via:

• (In ranger-postgres) docker build . -t coheigea/ranger-postgres
• (In ranger-admin) docker build . -t coheigea/ranger-admin

Note that the ranger-admin docker images takes a long time to build due to having to build the source code using Apache Maven - and hence it needs to download a large amount of dependencies.

There are two ways of running the project. The easiest is to install Docker compose and then simply start it with:

•  docker-compose up

The alternative is to create a network so that we can link containers, and then run the images separately using docker, i.e.:

• docker network create my-network
• docker run -p 5432:5432 --name postgres-server --network my-network coheigea/ranger-postgres
• docker run -p 6080:6080 -it --network my-network coheigea/ranger-admin

Once the Ranger admin server is started then open a browser and navigate to:

• http://localhost:6080 (credentials: admin/admin)

To see how to create authorization policies for various big data components using the UI please refer to the numerous blog posts I have previously written on this topic (for example: Kafka, HBase, HDFS).

The Apache Sentry security service - part IV

This is the fourth in a series of blog posts on the Apache Sentry security service. The first post looked at how to get started with the Apache Sentry security service, both from scratch and via a docker image. The second post looked at how to define the authorization privileges held in the Sentry security service. The third post looked at securing Apache Kafka withe Apache Sentry, where the privileges were defined in the Sentry security service. In this post, we will update an earlier tutorial I wrote on securing Apache Hive using Apache Sentry to also retrieve the privileges from the Sentry security service.

1) Configure authorization in Apache Hive

Please follow this tutorial to install and configure Apache Hadoop and Apache Hive, except use version 2.3.2 of Apache Hive, which is the version supported by Apache Sentry 2.0.0. After installation, follow the instructions to create a table in Hive and make sure that a query is successful. Now we will integrate Apache Sentry 2.0.0 with Apache Hive. First copy the jars from the "lib" directory of the Sentry distribution to the Hive "lib" directory. We need to add three new configuration files to the "conf" directory of Apache Hive.

Create a file called 'conf/hiveserver2-site.xml' with the content:

Here we are enabling authorization and adding the Sentry authorization plugin. Note that it differs a bit from the hiveserver2-site.xml given in the previous tutorial, namely that we are not using the "v2" Sentry Hive binding as before.

Next create a new file in the "conf" directory of Apache Hive called "sentry-site.xml" with the following content:


This is the configuration file for the Sentry plugin for Hive. It instructs Sentry to retrieve the authorization privileges from the Sentry security service, and to get the groups of authenticated users from the 'sentry.ini' configuration file. As we are not using Kerberos, the "testing.mode" configuration parameter must be set to "true". Finally, we need to define the groups associated with a given user in 'sentry.ini' in the conf directory:

Here we assign "alice" the group "user". Note that in the earlier tutorial this file also contained the authorization privileges, but they are not required in this scenario as we are using the Apache Sentry security service.

2) Configure the Apache Sentry security service

Follow the first tutorial to install the Apache Sentry security service. Now we need to create the authorization privileges for our Apache Hive test scenario as per the second tutorial. Start the 'sentryCli" in the Apache Sentry distribution, and assign a role to the "user" group (of which "alice" is a member) with the privilege to perform a "select" statement on the "words" table:

• cr select_role
• gp select_role "Server=server1->Db=default->Table=words->Column=*->action=select"
• gr select_role user

Now we can test authorization after restarting Apache Hive. The user 'alice' should now be able query the table according to our policy:

• bin/beeline -u jdbc:hive2://localhost:10000 -n alice
• select * from words where word == 'Dare'; (works)

The Apache Sentry security service - part III

This is the third in a series of blog posts on the Apache Sentry security service. The first post looked at how to get started with the Apache Sentry security service, both from scratch and via a docker image. The second post looked at how to define the authorization privileges held in the Sentry security service. In this post we will look at updating an earlier tutorial I wrote about securing Apache Kafka with Apache Sentry, this time using the security service instead of defining the privileges in a file local to the Kafka distribution.

1) Configure authorization in the broker

Firstly download and configure Apache Kafka using SSL as per this tutorial, except use Kafka 0.11.0.2. To enable authorization using Apache Sentry we also need to follow these steps. First edit 'config/server.properties' and add:

• authorizer.class.name=org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer
• sentry.kafka.site.url=file:./config/sentry-site.xml

Next copy the jars from the "lib" directory of the Sentry distribution to the Kafka "libs" directory. Then create a new file in the config directory called "sentry-site.xml" with the following content:

This is the configuration file for the Sentry plugin for Kafka. It instructs Sentry to retrieve the authorization privileges from the Sentry security service, and to get the groups of authenticated users from the 'sentry.ini' configuration file. Create a new file in the config directory called "sentry.ini" with the following content:
Note that in the earlier tutorial this file also contained the authorization privileges, but they are not required in this scenario as we are using the Apache Sentry security service.

2) Configure the Apache Sentry security service

Follow the first tutorial to install the Apache Sentry security service. Now we need to create the authorization privileges for our Apache Kafka test scenario as per the second tutorial. Start the 'sentryCli" in the Apache Sentry distribution.

Create the roles:

• t kafka
• cr admin_role
• cr describe_role
• cr read_role
• cr write_role
• cr describe_consumer_group_role 
• cr read_consumer_group_role

Add the privileges to the roles:

• gp admin_role "Host=*->Cluster=kafka-cluster->action=ALL"
• gp describe_role "Host=*->Topic=test->action=describe"
• gp read_role "Host=*->Topic=test->action=read"
• gp write_role "Host=*->Topic=test->action=write"
• gp describe_consumer_group_role "Host=*->ConsumerGroup=test-consumer-group->action=describe"
• gp read_consumer_group_role "Host=*->ConsumerGroup=test-consumer-group->action=read"

Associate the roles with groups (defined in 'sentry.ini' above):

• gr admin_role admin
• gr describe_role producer
• gr read_role producer
• gr write_role producer
• gr read_role consumer
• gr describe_role consumer
• gr describe_consumer_group_role consumer
• gr read_consumer_group_role consumer

3) Test authorization

Now start the broker (after starting Zookeeper):

• bin/kafka-server-start.sh config/server.properties

Start the producer:

• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties

Send a few messages to check that the producer is authorized correctly. Now start the consumer:

• bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer

Authorization should succeed and you should see the messages made by the producer appear in the consumer console window.

Enabling Apache CXF Fediz plugin logging in Apache Tomcat

The Apache CXF Fediz subproject provides an easy way to secure your web applications via the WS-Federation Passive Requestor Profile. An earlier tutorial I wrote covers how to deploy and secure a "simpleWebapp" project that ships with Fediz in Apache Tomcat. One of the questions that came up recently on that article was how to enable logging for the Fediz plugin itself (as opposed to the IdP/STS). My colleague Jan Bernhardt has covered this topic using Apache Log4j. Here we will show a simple alternative way to enable logging using java.util.logging.

Please follow the earlier tutorial to set up and secure the "simpleWebapp" in Apache Tomcat. Note that after a successful test, the IdP logs appear in "logs/idp.log" and the STS logs appear in "logs/sts.log". However no logs exist for the plugin itself. To rectify this, copy the "slf4j-jdk14" jar into "lib/fediz" (for example from here). Then edit 'webapps/fedizhelloworld/WEB-INF/classes/logging.properties' with the following content:

This configuration logs "INFO" level messages to the Console (catalina.out) and logs "FINE" level messages to the log file "logs/rp.log" in XML Format. For example: