Configuring Kerberos for Kafka in Talend Open Studio for ESB

A few years back I wrote a blog post about how to create a job in Talend Open Studio for Big Data to read data from an Apache Kafka topic using kerberos. This job made use of the "tKafkaConnection" and "tKafkaInput" components. In Talend Open Studio for ESB, there is a component based on Apache Camel called "cKafka" that can also be used for the same purpose, but configuring it with kerberos is slightly different. In this post, we will show how to use the cKafka component in Talend Open Studio for ESB to read from a Kafka topic using kerberos.

1) Kafka setup

Follow a previous tutorial to setup an Apache Kerby based KDC testcase and to configure Apache Kafka to require kerberos for authentication. Kafka 2.5.0 was used for the purpose of this tutorial. Create a "test" topic and write some data to it, and verify with the command-line consumer that the data can be read correctly.

2) Download Talend Open Studio for ESB and create a route

Now we will download Talend Open Studio for ESB (7.3.1 was used for the purposes of this tutorial). Unzip the file when it is downloaded and then start the Studio using one of the platform-specific scripts. It will prompt you to download some additional dependencies and to accept the licenses. Right click on "Routes" and select "Create Route", entering a name for the route.

In the search bar under "Palette" on the right hand side enter "kafka" and hit enter. Drag the "cKafka" component that should appear into the route designer. Next find the "cLog" component under "Miscellaneous" and drag this to the right of the "cKafka" component. Right click the "cKafka" component and select "Row / Route" and connect the resulting arrow with the "cLog" component.

 
3) Configure the components

Now let's configure the individual components. Double-click on the "cKafka" component and enter "test" for the topic. Next, select "Advanced Settings" and scroll down to the kerberos configuration. For "Kerberos Service Name" enter "kafka". Then for "Security Protocol" select "SASL over Plaintext":

Next click on the "Run" tab and go to "Advanced Settings". Under "JVM Settings" select the checkbox for "Use specific JVM arguments", and add new arguments as follows:

• -Djava.security.auth.login.config=<path.to.kafka>/config/client.jaas 
• -Djava.security.krb5.conf=<path.to.kerby.project>/target/krb5.conf

For the first argument, you need to enter the path of the "client.jaas" file as described in the tutorial to set up the Kafka test-case. For the second argument, you need to specify the path of the "krb5.conf" file supplied in the target directory of the Apache Kerby test-case:

Now we are ready to run the job. Click on the "Run" tab and then hit the "Run" button. Send some data via the producer to the "test" topic and you should see the data appear in the Run Window in the Studio.

SSH improvements in Apache Karaf

Last year I contributed a number of SSH improvements to Apache Karaf, which I never found the time to blog about. In this post I'll cover how to use SSH with Apache Karaf, and also what the improvements were.

1) Using SSH with Apache Karaf

Download and extract Apache Karaf (4.2.8 was used for the purposes of this post). Start it by running "bin/karaf". By default, Karaf starts an SSH service which is configured in 'etc/org.apache.karaf.shell.cfg'. Here you can see that the default port is 8101. Karaf uses JAAS to authenticate SSH credentials - the default realm is "karaf". Associated with this realm is a PropertiesLoginModule, which authenticates users against the credentials stored in 'etc/users.properties'. Also note that the user must have a group defined that matches the value for "sshRole" in 'etc/org.apache.karaf.shell.cfg'. So let's try to SSH into Karaf using the default admin credentials, and it should work:

• ssh karaf@localhost -p 8101

2) SSH algorithm update

The first improvement, which was merged for the 4.2.7 release, was to remove support by default for a number of outdated algorithms:

• SHA-1 algorithms were removed
• CBC ciphers were removed
• Old ciphers such as 3-DES, Blowfish, Arcfour were removed

These can all be configured in 'etc/org.apache.karaf.shell.cfg' if necessary. The configuration values + defaults are now as follows:

• ciphers = aes256-ctr,aes192-ctr,aes128-ctr
• macs = hmac-sha2-512,hmac-sha2-256
• kexAlgorithms = ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

So for example, the following now fails with Karaf 4.2.8 using the default configuration:

• ssh karaf@localhost -p 8101 -c 3des-cbc

3) Elliptic curve support for SSH server keys 

The second improvement, which was also merged for 4.2.7, was to add support to configure Karaf with an elliptic curve SSH key. Previously only RSA keys were supported. When you start Karaf, it will generate an SSH key if one does not already exist, according to the "algorithm" (RSA) and "keySize" (2048) defined in 'etc/org.apache.karaf.shell.cfg', and store it in the "hostKey" (etc/host.key) file. As part of the improvement, the public key is also written out to a new configuration property "hostKeyPub" (etc/host.key.pub).

To see this in action, delete 'etc/host.key.*' and edit 'etc/org.apache.karaf.shell.cfg' and change:

• keySize = 256
• algorithm = EC

Now restart Karaf + try to ssh in using the "-v" parameter. You will see something like: "debug1: Server host key: ecdsa-sha2-nistp256 SHA256:sDa1k...".

4) Support for elliptic keys in the PublicKeyLoginModule

As well as supporting authentication using a password via the PropertiesLoginModule, Karaf also supports authentication using a public key via the PublickeyLoginModule. The PublickeyLoginModule authenticates a public key for SSH by comparing it to keys stored in 'etc/keys.properties'. I added support for Karaf 4.2.7 to be able to authenticate using elliptic keys stored in 'etc/key.properties', before only RSA public keys were supported.

To see how this works, generate a new elliptic curve key with an empty password:

• ssh-keygen -t ecdsa -f karaf.id_ec

Now edit 'etc/keys.properties' and copy the public key that was written in "karaf.id_ec.pub". For example:

• colm=AAAAE2VjZHNhLXNoY...0=,_g_:sshgroup
• _g_\:sshgroup = group,ssh

 Now we can SSH into Karaf without a password prompt via:

• ssh colm@localhost -p 8101 -i karaf.id_ec

5) Support for encrypted key password for SSH

Finally, I added support for encrypted key passwords for SSH. This change necessitated moving from using not-yet-commons-ssl to BouncyCastle for parsing SSH keys, as the former does not support encrypted keys or newer security algorithms in general. As a result, encrypted key passwords for SSH are not available in Karaf 4.2.x, but will be in the next major release (4.3.0). Note as well that encrypted key passwords only work for when Karaf is reading an externally generated encrypted private key.

To test this out, grab Karaf 4.3.x and generate a new RSA encrypted private key as follows (specifying a password of "security"):

• openssl genpkey -out rsa.pem -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -aes256

Edit 'etc/org.apache.karaf.shell.cfg' and change it as follows:

• hostKey = ${karaf.etc}/rsa.pem
• hostKeyPassword = security

Before starting Karaf, it's also necessary to register BouncyCastle as a security provider. Edit 'etc/config.properties' and add:

• org.apache.karaf.security.providers = org.bouncycastle.jce.provider.BouncyCastleProvider

Now copy the BouncyCastle provider jar (e.g. bcprov-jdk15on-1.65.jar) to lib/ext and restart Karaf. It should be possible then to SSH into Karaf.

Recent deserialization CVEs in Apache Camel

Three security advisories have been published recently for Apache Camel that were found partly or entirely by me. CVE-2020-11971 relates to a JMX rebind flaw that'll be the subject of a future blog post. In this post I'll take a quick look at the other two issues, which both relate to Java object deserialization.

1) CVE-2020-11973

CVE-2020-11973 was caused by the fact that the Camel Netty component enabled Java Object serialization by default, without any whitelisting of acceptable packages associated with classes that are being deserialized. This is problematic as a malicious user could try to exploit a "gadget chain" to perform a remote code execution, or else create a denial of service type attack by creating a recursive object graph.

The fix was to remove object deserialization by default for the Netty component in 2.25.1 + 3.2.0. Users who still wish to avail of object serialization can explicitly enable object encoders/decoders on the component.

2) CVE-2020-11972

CVE-2020-11972 is for the exact same issue as for CVE-2020-11973 above, but in the Camel RabbitMQ component. From Camel 2.25.1 and 3.2.0, a new configuration option called "allowMessageBodySerialization" is introduced, which defaults to false.

Users who wish to avail of object serialization/deserialization can set this configuration option to true, (bearing in mind that this is not secure!) which enables the following behaviour:

• The outbound message will be serialized on the producer side using Java serialization, if no type converter is available to handle the message body.
• On the consumer side, the message body will be deserialized using Java deserialization if the message contains a "CamelSerialize" header.

If you are using either the Netty or RabbitMQ components with Camel, then please make sure to update to the latest releases ASAP. 
 

Improving XML Decryption performance

Apache Santuario - XML Security for Java is a library that provides the ability to sign and encrypt XML. In this post we'll focus solely on encryption. Apache Santuario has two APIs for XML Encryption, an in-memory (DOM) based approach and a streaming (StAX based) API. The streaming API offers very good memory performance as the size of the XML being encrypted scales up. However there is not much difference in speed, particularly for decryption. The streaming API is also less flexible than the DOM API. In this post we will look at different ways to speed up the DOM based API using Serializers.

1) Serializers in Apache Santuario

The XMLCipher class in Santuario is the main entry point for encryption and decryption. The XMLCipher class makes use of the Serializer interface to serialize DOM elements to byte arrays for encryption, and to deserialize byte arrays to DOM elements for decryption. Santuario ships with two different implementations. The default is TransformSerializer, which makes use of the "javax.xml.transform.Transformer" API and requires Apache Xalan to work properly. The other alternative is the DocumentSerializer, which uses the standard DOM API. These implementations perform very similarly.

Apache CXF is a web services framework that heavily leverages Apache Santuario to perform XML encryption and decryption of web service messages. CXF is fully streaming based and so it made sense to look to re-use some of this functionality with a custom Serializer implementation. The result is the StaxSerializer. This largely re-uses the DOM implementation for encryption, but uses a streaming based approach for deserialization. StaxSerializer is used by default in Apache CXF when working with XML encryption/decryption.

2) Benchmarking

To benchmark the StaxSerializer, I adapted the Santuario benchmarking test-suite just to compare encryption performance using the different Serializers, and put it into github here:

• santuario-serializer-benchmark: This project contains two Junit tests used for benchmarking XML Encryption. In particular, they measure memory and timing performance for both encryption and decryption, ranging from small to very large XML files.

a) Heap memory consumption during decryption

Here is the result for the (default) TransformSerializer:

 In comparison, here is the result for the StaxSerializer:

The most obvious conclusion is that the streaming API is far more efficient in terms of memory consumption, especially when we scale to larger documents. However one would expect also to see less memory being consumed using the StaxSerializer compared to the TransformSerializer. Indeed, one can see that the TransformSerializer almost consumes 600MB for the largest case, wheras the StaxSerializer comes in under 450MB.

b) Time needed for decryption

Turning now to execution time, here is the result for the (default) TransformSerializer:

As stated above, the performance is fairly identical to the streaming layer. This is because the streaming implementation needs to make several passes of the XML for consistency reasons. Here is the result for the StaxSerializer:

Here we can see that the StaxSerializer actually offers superior performance to the streaming API. So if XML decryption performance is an issue for you, it might be worth considering using CXF's StaxSerializer.

Two final 2019 CVEs for Apache CXF

Apache CXF 3.3.5 and 3.2.12 have been released. These releases contain fixes for two new security advisories:

• CVE-2019-12423: Apache CXF OpenId Connect JWK Keys service returns private/secret credentials if configured with a jwk keystore. 

• CVE-2019-17573: Apache CXF Reflected XSS in the services listing page. Note that this attack exploits a feature which is not typically not
  present in modern browsers, who remove dot segments before sending the
  request. However, Mobile applications may be vulnerable.

Please see the CXF security advisories page for information on all of the CVEs issued for Apache CXF over the years.