Friday, July 22, 2016

Syncing users and groups from LDAP into Apache Ranger

The previous post covered how to install the Apache Ranger Admin service. The Apache Ranger Admin UI supports creating authorization policies for various Big Data components, by giving users and/or groups permissions on resources. This means that we need to import users/groups into the Apache Ranger Admin service from some backend service in order to create meaningful authorization policies. Apache Ranger supports syncing users into the Admin service from both unix and ldap. In this post, we'll look at syncing in users and groups from an OpenDS LDAP backend.

1) The OpenDS backend

For the purposes of this tutorial, we will use OpenDS as the LDAP server. It contains a domain called "dc=example,dc=com", and 5 users (alice/bob/dave/oscar/victor) and 2 groups (employee/manager). Victor, Oscar and Bob are employees, Alice and Dave are managers. Here is a screenshot using Apache Directory Studio:



2) Build the Apache Ranger usersync module

Follow the steps in the previous tutorial to build Apache Ranger and to setup and start the Apache Ranger Admin service. Once this is done, go back to the Apache Ranger distribution that you have built and copy the usersync module:
  • tar zxvf target/ranger-0.6.0-usersync.tar.gz
  • mv ranger-0.6.0-usersync ${usersync.home}
3) Configure and build the Apache Ranger usersync service 

You will need to install the Apache Ranger Usersync service using "sudo". If the root user does not have a JAVA_HOME property defined, then edit ${usersync.home}/setup.sh + add in, e.g.:
  • export JAVA_HOME=/opt/jdk1.8.0_91
Next edit ${usersync.home}/install.properties and make the following changes:
  • POLICY_MGR_URL = http://localhost:6080
  • SYNC_SOURCE = ldap
  • SYNC_INTERVAL = 1 (just for testing purposes....)
  • SYNC_LDAP_URL = ldap://localhost:2389
  • SYNC_LDAP_BIND_DN = cn=Directory Manager,dc=example,dc=com
  • SYNC_LDAP_BIND_PASSWORD = test
  • SYNC_LDAP_SEARCH_BASE = dc=example,dc=com
  • SYNC_LDAP_USER_SEARCH_BASE = ou=users,dc=example,dc=com
  • SYNC_GROUP_SEARCH_BASE=ou=groups,dc=example,dc=com
Now you can run the setup script via "sudo ./setup.sh". 

4) Start the Usersync service

The Apache Ranger Usersync service can be started via "sudo ./ranger-usersync-services.sh start". After 1 minute (see SYNC_INTERVAL above), it should successfully copy the users/groups from the OpenDS backend into the Apache Ranger Admin. Open a browser and go to "http://localhost:6080", and click on "Settings" and then "Users/Groups". You should see the users and groups synced successfully from OpenDS.

Tuesday, July 19, 2016

Installing the Apache Ranger Admin UI

Apache Ranger 0.6 has been released, featuring new support for securing Apache Atlas and Nifi, as well as a huge amount of bug fixes. It's easiest to get started with Apache Ranger by downloading a big data sandbox with Ranger pre-installed. However, the most flexible way is to grab the Apache Ranger source and to build and deploy the artifacts yourself. In this tutorial, we will look into building Apache Ranger from source, setting up a database to store policies/users/groups/etc. as well as Ranger audit information, and deploying the Apache Ranger Admin UI.

1) Build the source code

The first step is to download the source code, as well as the signature file and associated message digests (all available on the download page). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting admin archive to a location where you wish to install the UI:
  • tar zxvf apache-ranger-incubating-0.6.0.tar.gz
  • cd apache-ranger-incubating-0.6.0
  • mvn clean package assembly:assembly 
  • tar zxvf target/ranger-0.6.0-admin.tar.gz
  • mv ranger-0.6.0-admin ${rangerhome}
2) Install MySQL

The Apache Ranger Admin UI requires a database to keep track of users/groups as well as policies for various big data projects that you are securing via Ranger. In addition, we will use the database for auditing as well. For the purposes of this tutorial, we will use MySQL. Install MySQL in $SQL_HOME and start MySQL via:
  • sudo $SQL_HOME/bin/mysqld_safe --user=mysql
Now you need to log on as the root user and create three users for Ranger. We need a root user with admin privileges (let's call this user "admin"), a user for the Ranger Schema (we'll call this user "ranger"), and finally a user to store the Ranger audit logs in the DB as well ("rangerlogger"):
  • CREATE USER 'admin'@'localhost' IDENTIFIED BY 'password';
  • GRANT ALL PRIVILEGES ON * . * TO 'admin'@'localhost' WITH GRANT OPTION;
  • CREATE USER 'ranger'@'localhost' IDENTIFIED BY 'password';
  • CREATE USER 'rangerlogger'@'localhost' IDENTIFIED BY 'password'; 
  • FLUSH PRIVILEGES;
Finally,  download the JDBC driver jar for MySQL and put it in ${rangerhome}.

3) Install the Apache Ranger Admin UI

You will need to install the Apache Ranger Admin UI using "sudo". If the root user does not have a JAVA_HOME property defined, then edit ${rangerhome}/setup.sh + add in, e.g.:
  • export JAVA_HOME=/opt/jdk1.8.0_91
Next edit ${rangerhome}/install.properties and make the following changes:
  • Change SQL_CONNECTOR_JAR to point to the MySQL JDBC driver jar that you downloaded above.
  • Set (db_root_user/db_root_password) to (admin/password)
  • Set (db_user/db_password) to (ranger/password)
  • Change "audit_store" from "solr" to "db"
  • Set "audit_db_name" to "ranger_audit"
  • Set (audit_db_user/audit_db_password) to (rangerlogger/password).
Now you can run the setup script via "sudo ./setup.sh".

4) Starting the Apache Ranger admin service

After a successful installation, we can start the Apache Ranger admin service with "sudo ranger-admin start". Now open a browser and go to "http://localhost:6080/". Log on with "admin/admin" and you should be able to create authorization policies for a desired big data component.


Friday, June 17, 2016

A new REST interface for the Apache CXF Security Token Service - part II

The previous blog entry introduced the new REST interface of the Apache CXF Security Token Service. It covered issuing, renewing and validating tokens via HTTP GET and POST with a focus on SAML tokens. In this post, we'll cover the new support in the STS to issue and validate JWT tokens via the REST interface, as well as how we can transform SAML tokens into JWT tokens, and vice versa. For information on how to configure the STS to support JWT tokens (via WS-Trust), please look at a previous blog entry.

1) Issuing JWT Tokens

We can retrieve a  JWT Token from the REST interface of the STS simply by making a HTTP GET request to "/token/jwt". Specify the "appliesTo" query parameter to insert the service address as the audience of the generated token. You can also include various claims in the token via the "claim" query parameter.

If an "application/xml" accept type is specified, or if multiple accept types are specified (as in a browser), the JWT token is wrapped in a "TokenWrapper" XML Element by the STS:

If "application/json" is specified, the JWT token is wrapped in a simple JSON Object as follows:

If "text/plain" is specified, the raw JWT token is returned. We can also get a JWT token using HTTP POST by constructing a WS-Trust RequestSecurityToken XML fragment, specifying "urn:ietf:params:oauth:token-type:jwt" as the WS-Trust TokenType.

2) Validating JWT Tokens and token transformation

We can also validate JWT Tokens by POSTing a WS-Trust RequestSecurityToken to the STS. The raw (String) JWT Token must be wrapped in a "TokenWrapper" XML fragment, which in turn is specified as the value of "ValidateTarget". Also, an "action" query parameter must be added with value "validate".

A powerful feature of the STS is the ability to transform tokens from one type to another. This is done by making a validate request for a given token, and specifying a value for "TokenType" that corresponds to a token type that is desired.  In this way we can validate a SAML token and issue a JWT Token, and vice versa.

To see some examples of how to do token validation as well as token transformation, please take a look at the following tests, which use the CXF WebClient to invoke on the REST interface of the STS.

Thursday, June 16, 2016

A new REST interface for the Apache CXF Security Token Service - part I

Apache CXF ships a Security Token Service (STS) that can issue/validate/renew/cancel tokens via the (SOAP based) WS-Trust interface. The principal focus of the STS is to deal with SAML tokens, although other token types are supported as well. JAX-RS clients can easily obtain tokens using helper classes in CXF (see Andrei Shakirin's blog entry for more information on this).

However, wouldn't it be cool if a JAX-RS client could dispense with SOAP altogether and obtain tokens via a REST interface? Starting from the 3.1.6 release, the Apache CXF STS now has a powerful and flexible REST API which I'll describe in this post. The next post will cover how the STS can now issue and validate JWT tokens, and how it can transform JWT tokens into SAML tokens and vice versa.

One caveat - this REST interface is obviously not a standard, as compared to the WS-Trust interface, and so is specific to CXF.

1) Configuring the JAX-RS endpoint of the CXF STS

Firstly, let's look at how to set up the JAX-RS endpoint needed to support the REST interface of the STS. The STS is configured in exactly the same way as for the standard WS-Trust based interface, the only difference being that we are setting up a JAX-RS endpoint instead of a JAX-WS endpoint. Example configuration can be seen here in a system test.


2) Issuing Tokens

The new JAX-RS interface supports a number of different methods to obtain tokens. In this post, we will just focus on the methods used to obtain XML based tokens (such as SAML).

2.1) Issue tokens via HTTP GET

The easiest way to get a token is by doing a HTTP GET on "/token/{tokenType}". Here is a simple example using a web browser:

 The supported tokenType values are:
  • saml
  • saml2.0
  • saml1.1
  • jwt
  • sct
A number of optional query parameters are supported:
  • keyType - The standard WS-Trust based URIs to indicate whether a Bearer, HolderOfKey, etc. token is required. The default is to issue a Bearer token.
  • claim - A list of requested claims to include in the token. See below for the list of acceptable values.
  • appliesTo - The AppliesTo value to use
  • wstrustResponse - A boolean parameter to indicate whether to return a WS-Trust Response or just the desired token. The default is false. This parameter only applies to the XML case.
Note that for the "Holder of Key" keytype, the STS will try to get the client key via the TLS client certificate, if it is available. The (default) supported claim values are:
  • emailaddress
  • role
  • surname
  • givenname
  • name
  • upn
  • nameidentifier
The format of the returned token depends on the HTTP application type:
  • application/xml - Returns a token in XML format
  • application/json - JSON format for JWT tokens only.
  • text/plain - The "plain" token is returned. For JWT tokens it is just the raw token. For XML-based tokens, the token is BASE-64 encoded and returned.
The default is to return XML if multiple types are specified (e.g. in a browser).

2.2) Issue tokens via HTTP POST

While the GET method above is very convenient to use, it may be that you want to pass other parameters to the STS in order to issue a token. In this case, you can construct a WS-Trust RequestSecurityToken XML fragment and POST it to the STS. The response will be the standard WS-Trust Response, and not the raw token as you have the option of receiving via the GET method.

3) Validating/Renewing/Cancelling tokens

It is only possible to renew or validate or cancel a token using the POST method detailed in section 2.2 above. You construct the RequestSecurityToken XML fragment in the same way as for Issue, except including the token in question under "ValidateTarget", "RenewTarget", etc. For the non-issue use case, you must specify a "action" query parameter, which can be "issue" (default), "validate", "renew", "cancel".

Friday, May 27, 2016

SAML SSO support in the Fediz 1.3.0 IdP

The Apache CXF Fediz Identity Provider (IdP) has had the ability to talk to third party IdPs using SAML SSO since the 1.2.0 release. However, one of the new features of the 1.3.0 release is the ability to configure the Fediz IdP to use the SAML SSO protocol directly, instead of WS-Federation. This means that Fediz can be used as a fully functioning SAML SSO IdP.

I added a new test-case to github to show how this works:
  • cxf-fediz-saml-sso: This project shows how to use the SAML SSO interceptors of Apache CXF to authenticate and authorize clients of a JAX-RS service. 
The test-case consists of two modules. The first is a web application which contains a simple JAX-RS service, which has a single GET method to return a doubled number. The method is secured with a @RolesAllowed annotation, meaning that only a user in roles "User", "Admin", or "Manager" can access the service. The service is configured with the SamlRedirectBindingFilter, which redirects unauthenticated users to a SAML SSO IdP for authentication (in this case Fediz). The service configuration also defines an AssertionConsumerService which validates the response from the IdP, and sets up the session for the user + populates the CXF security context with the roles from the SAML Assertion.

The second module deploys the Fediz IdP and STS in Apache Tomcat, as well as the "double-it" war above. It uses Htmlunit to make an invocation on the service and check that access is granted to the service. Alternatively, you can comment the @Ignore annotation of the "testInBrowser" method, and copy the printed out URL into a browser to test the service directly (user credentials: "alice/ecila").

The IdP configuration is defined in entities-realma.xml. Note that under "supportedProtocols" for the "idp-realmA" configuration is the value "urn:oasis:names:tc:SAML:2.0:profiles:SSO:browser". In addition, the default authentication URI is "saml/up". These are the only changes that are required to switch the IdP for "realm A" from WS-Federation to SAML SSO.

Friday, May 6, 2016

An interop demo between Apache CXF Fediz and Facebook

The previous post showed how to configure the Fediz IdP to interoperate with the Google OpenID Connect provider. In addition to supporting WS-Federation, SAML SSO and OpenID Connect, from the forthcoming 1.3.1 release the Fediz IdP will also support authenticating users via Facebook Connect. Facebook Connect is based on OAuth 2.0 and hence the implementation in Fediz shares quite a lot of common code with the OpenID Connect implementation. In this post, we will show how to configure the Fediz IdP to interoperate with Facebook.

1) Create a new app in the Facebook developer console

The first step is to create a new application in the Facebook developer console. Create a Facebook developer account. Login and register as a Facebook developer. Click on "My Apps" and "Add a New App". Select the "website" platform and create a new app with the name "Fediz IdP". Enter your email, select a Category and click on "Create App ID". Enter "https://localhost:8443/fediz-idp/federation" for the Site URL. Now go to the dashboard for the application and copy the App ID + App Secret and save for later.

2) Testing the newly created Client

To test that everything is working correctly, open a web browser and navigate to, substituting the client id saved in step "1" above:
  • https://www.facebook.com/dialog/oauth?response_type=code&client_id=<app-id>&redirect_uri=https://localhost:8443/fediz-idp/federation&scope=email
Login using your Facebook credentials and grant permission on the OAuth consent screen. The browser will then attempt a redirect to the given redirect_uri. Copy the URL + extract the "code" query String. Open a terminal and invoke the following command, substituting in the secret and code extracted above:
  • curl --data "client_id=<app-id>&client_secret=<app-secret>&grant_type=authorization_code&code=<code>&redirect_uri=https://localhost:8443/fediz-idp/federation" https://graph.facebook.com/v2.6/oauth/access_token
You should see a successful response containing the OAuth 2.0 Access Token.

3) Install and configure the Apache CXF Fediz IdP and sample Webapp

Follow a previous tutorial to deploy the latest Fediz IdP + STS to Apache Tomcat, as well as the "simpleWebapp". Note that you will need to use at least Fediz 1.3.1-SNAPSHOT here for Facebook support. Test that the "simpleWebapp" is working correctly by navigating to the following URL (selecting "realm A" at the IdP, and authenticating as "alice/ecila"):
  • https://localhost:8443/fedizhelloworld/secure/fedservlet 
3.1) Configure the Fediz IdP to communicate with the Facebook IdP

Now we will configure the Fediz IdP to authenticate the user in "realm B" by using the Facebook Connect protocol. Edit 'webapps/fediz-idp/WEB-INF/classes/entities-realma.xml'. In the 'idp-realmA' bean:
  • Change the port in "idpUrl" to "8443". 
In the 'trusted-idp-realmB' bean:
  • Change the "url" value to "https://www.facebook.com/dialog/oauth".
  • Change the "protocol" value to "facebook-connect".
  • Delete the "certificate" and "trustType" properties.
  • Add the following parameters Map, filling in a value for the client secret extracted above: <property name="parameters">
                <util:map>
                    <entry key="client.id" value="<app-id>" />
                    <entry key="client.secret" value="<app-secret>" />
                </util:map>
            </property>
It is possible to specify custom scopes via a "scope" parameter. The default value for this parameter is "email". The token endpoint can be configured via a "token.endpoint" parameter (defaults to "https://graph.facebook.com/v2.6/oauth/access_token"). Similarly, the API endpoint used to retrieve the user's email address can be configured via "api.endpoint", defaulting to "https://graph.facebook.com/v2.6". The Subject claim to be used to insert into a SAML Assertion is defined via "subject.claim", which defaults to "email".

3.2) Update the TLS configuration

By default, the Fediz IdP is configured with a truststore required to access the Fediz STS. However, this would mean that the requests to the Facebook IdP over TLS will not be trusted. to change this, edit 'webapps/fediz-idp/WEB-INF/classes/cxf-tls.xml', and change the HTTP conduit name from "*.http-conduit" to "https://localhost.*". This means that this configuration will only get picked up when communicating with the STS (deployed on "localhost"), and the default JDK truststore will get used when communicating with the Facebook IdP.

3.3) Update Fediz STS claim mapping

The STS will receive a SAML Token created by the IdP representing the authenticated user. The Subject name will be the email address of the user as configured above. Therefore, we need to add a claims mapping in the STS to map the principal received to some claims. Edit 'webapps/fediz-idp-sts/WEB-INF/userClaims.xml' and just copy the entry for "alice" in "userClaimsREALMA", changing "alice" to your email address used to log into Facebook.

Finally, restart Fediz to pick up the changes (you may need to remove the persistent storage first).

4) Testing the service

To test the service navigate to:
  • https://localhost:8443/fedizhelloworld/secure/fedservlet
Select "realm B". You should be redirected to the Facebook authentication page. Enter the user credentials you have created. You will be redirected to Fediz, where it creates a SAML token representing the act of authentication via the trusted third party IdP,  and redirects to the web application.

Friday, April 29, 2016

An interop demo between Apache CXF Fediz and Google OpenID Connect

The previous post introduced some of the new features in Apache CXF Fediz 1.3.0. One of the new enhancements is that the Fediz IdP can now delegate WS-Federation (and SAML SSO) authentication requests to a third party IdP via OpenID Connect. An article published in February showed how it is possible to interoperate between Fediz and the Keycloak OpenID Connect provider. In this post, we will show how to configure the Fediz IdP to interoperate with the Google OpenID Connect provider.

1) Create a new client in the Google developer console

The first step is to create a new client in the Google developer console to represent the Apache CXF Fediz IdP. Login to the Google developer console and create a new project. Click on "Enable and Manage APIs" and then select "Credentials". Click on "Create Credentials" and select "OAuth client id". Configure the OAuth consent screen and select "web application" as the application type. Specify "https://localhost:8443/fediz-idp/federation" as the authorized redirect URI. After creating the client, a pop-up window will specify the newly created client id and secret. Save both values locally. The Google documentation is available here.

2) Testing the newly created Client

It's possible to see the Google OpenId Connect configuration by navigating to:
  • https://accounts.google.com/.well-known/openid-configuration
This tells us what the authorization and token endpoints are, both of which we will need to configure the Fediz IdP. To test that everything is working correctly, open a web browser and navigate to, substituting the client id saved in step "1" above:
  • https://accounts.google.com/o/oauth2/v2/auth?response_type=code&client_id=<client-id>&redirect_uri=https://localhost:8443/fediz-idp/federation&scope=openid
Login using your google credentials and grant permission on the OAuth consent screen. The browser will then attempt a redirect to the given redirect_uri. Copy the URL + extract the "code" query String. Open a terminal and invoke the following command, substituting in the secret and code extracted above:
  • curl -u <client-id>:<secret> --data "client_id=<client-id>&grant_type=authorization_code&code=<code>&redirect_uri=https://localhost:8443/fediz-idp/federation" https://www.googleapis.com/oauth2/v4/token
You should see a succesful response containing (amongst other things) the OAuth 2.0 Access Token and the OpenId Connect IdToken, containing the user identity.

3) Install and configure the Apache CXF Fediz IdP and sample Webapp

Follow a previous tutorial to deploy the latest Fediz IdP + STS to Apache Tomcat, as well as the "simpleWebapp". Note that you will need to use Fediz 1.3.0 here for OpenId Connect support. Test that the "simpleWebapp" is working correctly by navigating to the following URL (selecting "realm A" at the IdP, and authenticating as "alice/ecila"):
  • https://localhost:8443/fedizhelloworld/secure/fedservlet 
3.1) Configure the Fediz IdP to communicate with the Google IdP

Now we will configure the Fediz IdP to authenticate the user in "realm B" by using the OpenID Connect protocol. Edit 'webapps/fediz-idp/WEB-INF/classes/entities-realma.xml'. In the 'idp-realmA' bean:
  • Change the port in "idpUrl" to "8443". 
In the 'trusted-idp-realmB' bean:
  • Change the "url" value to "https://accounts.google.com/o/oauth2/v2/auth".
  • Change the "protocol" value to "openid-connect-1.0".
  • Delete the "certificate" and "trustType" properties.
  • Add the following parameters Map, filling in a value for the client secret extracted above: <property name="parameters">
                <util:map>
                    <entry key="client.id" value="<client-id>" />
                    <entry key="client.secret" value="<secret>" />
                    <entry key="token.endpoint" value="https://accounts.google.com/o/oauth2/token" />
                    <entry key="scope" value="openid profile email"/>
                    <entry key="jwks.uri" value="https://www.googleapis.com/oauth2/v3/certs" />
                    <entry key="subject.claim" value="email"/>
                </util:map>
            </property>
There are a few additional properties configured here compared to the previous tutorial. It is possible to specify custom scopes via the "scope" parameter. In this case we are requesting the "profile" and "email" scopes. The default value for this parameter is "openid". In addition, rather than validating the signed IdToken using a local certificate, here we are specifying a value for "jwks.uri", which is the location of the signing key. The "subject.claim" property specifies the claim name from which to obtain the Subject name, which is inserted into a SAML Token that is sent to the STS.

3.2) Update the TLS configuration

By default, the Fediz IdP is configured with a truststore required to access the Fediz STS. However, this would mean that the requests to the Google IdP over TLS will not be trusted. to change this, edit 'webapps/fediz-idp/WEB-INF/classes/cxf-tls.xml', and change the HTTP conduit name from "*.http-conduit" to "https://localhost.*". This means that this configuration will only get picked up when communicating with the STS (deployed on "localhost"), and the default JDK truststore will get used when communicating with the Google IdP.

3.3) Update Fediz STS claim mapping

The STS will receive a SAML Token created by the IdP representing the authenticated user. The Subject name will be the email address of the user as configured above. Therefore, we need to add a claims mapping in the STS to map the principal received to some claims. Edit 'webapps/fediz-idp-sts/WEB-INF/userClaims.xml' and just copy the entry for "alice" in "userClaimsREALMA", changing "alice" to your Google email address.

Finally, restart Fediz to pick up the changes (you may need to remove the persistent storage first).

4) Testing the service

To test the service navigate to:
  • https://localhost:8443/fedizhelloworld/secure/fedservlet
Select "realm B". You should be redirected to the Google authentication page. Enter the user credentials you have created. You will be redirected to Fediz, where it converts the received JWT token to a token in the realm of Fediz (realm A) and redirects to the web application.