My current project at Novell involves the development of a ReSTful web service for submission of audit records from security applications. The server is a Jersey servlet within an embedded Tomcat 6 container.
One of the primary reasons for using a ReSTful web service for this purpose is to alleviate the need to design and build a heavy-weight audit record submission client library. Such client libraries need to be orthogonally portable across both hardware platforms and languages in order to be useful to Novell’s customers. Just maintaining the portability of this client library in one language is difficult enough, without adding multiple languages to the matrix.
Regardless of our motivation, we still felt the need to provide a quality reference implementation of a typical audit client library to our customers. They may incorporate as much or as little of this code as they wish, but a good reference implementation is worth a thousand pages of documentation. (Don’t get me wrong, however–this is no excuse for not writing good documentation! The combination of quality concise documentation and a good reference implementation is really the best solution.)
The idea here is simple: Our customers won’t have to deal with difficulties that we stumble upon and then subsequently provide solutions for. Additionally, it’s just plain foolish to provide a server component for which you’ve never written a client. It’s like publishing a library API that you’ve never written to. You don’t know if the API will even work the way you originally intended until you’ve at least tried it out.
Since we’re already using Java in the server, we’ve decided that our initial client reference implementation should also be written in Java. Yesterday found my code throwing one exception after another while simply trying to establish the TLS connection to the server from the client. All of these problems ultimately came down to my lack of understanding of the Java key store and trust store concepts.
You see, the establishment of a TLS connection from within a Java client application depends heavily on the proper configuration of a client-side trust store. If you’re using mutual authentication, as we are, then you also need to properly configure a client-side key store for the client’s private key. The level at which we are consuming Java network interfaces also demands that we specify these stores in system properties. More on this later…
Using Curl as an Https Client
We based our initial assumptions about how the Java client needed to be configured on our use of the curl command line utility in order to test the web service. The curl command line looks something like this:
curl -k --cert client.cer --cert-type DER --key client-key.pem
--key-type PEM --header "Content-Type: application/audit+json"
-X POST --data @test-event.json https://10.0.0.1:9015/audit/log/test
The important aspects of this command-line include the use of the –cert, –cert-type, –key and –key-type parameters, as well as the fact that we specified a protocol scheme of “https” in the URL.
With one exception, the remaining options are related to which http method to use (-X), what data to send (–data), and which message properties to send (–header). The exception is the -k option, and therein lay most of our problems with this Java client.
The curl man-page indicates that the -k/–insecure option allows the TLS handshake to succeed without verifying the server certificate in the client’s CA (Certificate Authority) trust store. The reason this option was added was because several releases of the curl package shipped with a terribly out-dated trust store, and people were getting tired of having to manually add certificates to their trust stores everytime they hit a newer site.
Doing it in Java
But this really isn’t the safe way to access any secure public web service. Without server certificate verification, your client can’t really know that it’s not communicating with a server that just says it’s the right server. (“Trust me!”)
During the TLS handshake, the server’s certificate is passed to the client. The client should then verify the subject name of the certificate. But verify it against what? Well, let’s consider–what information does the client have access to, outside of the certificate itself? It has the fully qualified URL that it used to contact the server, which usually contains the DNS host name. And indeed, a client is supposed to compare the CN (Common Name) portion of the subject DN (Distinguished Name) in the server certificate to the DNS host name in the URL, according to section 3.1 “Server Identity” of RFC 2818 “HTTP over TLS”.
Java’s HttpsURLConnection class strictly enforces the advice given in RFC 2818 regarding peer verification. You can override these constraints, but you have to basically write your own version of HttpsURLConnection, or sub-class it and override the methods that verify peer identity.
Creating Java Key and Trust Stores
Before even attempting a client connection to our server, we had to create three key stores:
- A server key store.
- A client key store.
- A client trust store.
The server key store contains the server’s self-signed certificate and private key. This store is used by the server to sign messages and to return credentials to the client.
The client key store contains the client’s self-signed certificate and private key. This store is used by the client for the same purpose–to send client credentials to the server during the TLS mutual authentication handshake. It’s also used to sign client-side messages for the server during the TLS handshake. (Note that once authentication is established, encryption happens using a secret or symetric key encryption algorithm, rather than public/private or asymetric key encryption. Symetric key encryption is a LOT faster.)
The client trust store contains the server’s self-signed certificate. Client-side trust stores normally contain a set of CA root certificates. These root certificates come from various widely-known certificate vendors, such as Entrust and Verisign. Presumably, almost all publicly visible servers have a purchased certificate from one of these CA’s. Thus, when your web browser connects to such a public server over a secure HTTP connection, the server’s certificate can be verified as having come from one of these well-known certificate vendors.
I first generated my server key store, but this keystore contains the server’s private key also. I didn’t want the private key in my client’s trust store, so I extracted the certificate into a stand-alone certificate file. Then I imported that server certificate into a trust store. Finally, I generated the client key store:
$ keytool -genkey -alias server -keyalg RSA \
> -storepass changeit -keystore server-keystore.jks
What is your first and last name?
[Unknown]: audit-server
What is the name of your organizational unit?
[Unknown]: Eng
What is the name of your organization?
[Unknown]: Novell
What is the name of your City or Locality?
[Unknown]: Provo
What is the name of your State or Province?
[Unknown]: Utah
What is the two-letter country code for this unit?
[Unknown]: US
Is CN=audit-server, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US correct?
[no]: yes
Enter key password for <server>
(RETURN if same as keystore password):
$
$ keytool -exportcert -keystore server-keystore.jks \
> -file server.der -alias server -storepass changeit
Certificate stored in file <server.der>
$
$ keytool -importcert -trustcacerts -alias server \
> -keystore server-truststore.jks -storepass changeit \
> -file server.der
Owner: CN=audit-server, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US
Issuer: CN=audit-server, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US
Serial number: 491cad67
Valid from: Thu Nov 13 15:42:47 MST 2008 until: Wed Feb 11 15:42:47 MST 2009
Certificate fingerprints:
MD5: EE:FA:EE:78:A8:42:2B:F2:3A:04:50:37:D3:94:B3:C0
SHA1: 4E:BA:9B:2F:FC:84:10:5A:2E:62:D2:5B:B3:70:70:B5:2F:03:E1:CD
Signature algorithm name: SHA1withRSA
Version: 3
Trust this certificate? [no]: yes
Certificate was added to keystore
$
$ keytool -genkey -alias client -keyalg RSA -storepass changeit \
> -keystore client-keystore.jks
What is your first and last name?
[Unknown]: audit-client
What is the name of your organizational unit?
[Unknown]: Eng
What is the name of your organization?
[Unknown]: Novell
What is the name of your City or Locality?
[Unknown]: Provo
What is the name of your State or Province?
[Unknown]: Utah
What is the two-letter country code for this unit?
[Unknown]: US
Is CN=audit-client, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US correct?
[no]: yes
Enter key password for <client>
(RETURN if same as keystore password):
$
$ ls -1
client-keystore.jks
server.der
server-keystore.jks
server-truststore.jks
$
Telling the Client About Keys
There are various ways of telling the client about its key and trust stores. One method involves setting system properties on the command line. This is commonly used because it avoids the need to enter absolute paths directly into the source code, or to manage separate configuration files.
$ java -Djavax.net.ssl.keyStore=/tmp/keystore.jks ...
Another method is to set the same system properties inside the code itself, like this:
public class AuditRestClient
{
public AuditRestClient()
{
System.setProperty("javax.net.ssl.keyStore",
"/tmp/keystore.jks");
System.setProperty("javax.net.ssl.keyStorePassword",
"changeit");
System.setProperty("javax.net.ssl.trustStore",
"/tmp/truststore.jks");
System.setProperty("javax.net.ssl.trustStorePassword",
"changeit");
}
...
I chose the latter, as I’ll eventually extract the strings into property files loaded as needed by the client code. I don’t really care for the fact that Java makes me specify these stores in system properties. This is especially a problem for our embedded client code, because our customers may have other uses for these system properties in the applications in which they will embed our code. Here’s the rest of the simple client code:
...
public void send(JSONObject event)
{
byte[] bytes = event.toString().getBytes();
HttpURLConnection conn = null;
try
{
// establish connection parameters
URL url = new URL("https://10.0.0.1:9015/audit/log/test");
conn = (HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
conn.addRequestProperty("Content-Length", "" + bytes.length);
conn.addRequestProperty("Content-Type", "application/audit1+json");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.connect();
// send POST data
OutputStream out = (OutputStream)conn.getOutputStream();
out.write(bytes);
out.flush();
out.close();
// get response code and data
System.out.println(conn.getResponseCode());
BufferedReader read = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String query = null;
while((query = read.readLine()) != null)
System.out.println(query);
}
catch(MalformedURLException e) { e.printStackTrace(); }
catch(ProtocolException e) { e.printStackTrace(); }
catch(IOException e) { e.printStackTrace(); }
finally { conn.disconnect(); }
}
}
Getting it Wrong…
I also have a static test “main” function so I can send some content. But when I tried to execute this test, I got an exception indicating that the server certificate didn’t match the host name. I was using a hard-coded IP address (10.0.0.1), but my certificate contained the name “audit-server”.
It turns out that the HttpsURLConnection class uses an algorithm to determine if the server that sent the certificate really belongs to the server on the other end of the connection. If the URL contains an IP address, then it attempts to locate a matching IP address in the “Alternate Names” portion of the server certificate.
Did you notice a keytool prompt to enter alternate names when you generated your server certificate? I didn’t–and it turns out there isn’t one. The Java keytool utility doesn’t provide a way to enter alternate names–a standardized extension of the X509 certificate format. To enter an alternate name containing the requisite IP address, you’d have to generate your certificate using the openssl utility, or some other more functional certificate generation tool, and then find a way to import these foreign certificates into a Java key store.
…And then Doing it Right
On the other hand, if the URL contains a DNS name, then HttpsURLConnection attempts to match the CN portion of the Subject DN with the DNS name. This means that your server certificates have to contain the DNS name of the server as the CN portion of the subject. Returning to keytool, I regenerated my server certificate and stores using the following commands:
$ keytool -genkey -alias server -keyalg RSA \
> -storepass changeit -keystore server-keystore.jks
What is your first and last name?
[Unknown]: jmc-test.provo.novell.com
... (the rest is the same) ...
$ keytool -exportcert -keystore server-keystore.jks \
> -file server.der -alias server -storepass changeit
Certificate stored in file <server.der>
$
$ keytool -importcert -trustcacerts -alias server \
> -keystore server-truststore.jks -storepass changeit \
> -file server.der
Owner: CN=jmc-test.provo.novell.com, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US
Issuer: CN=jmc-test.provo.novell.com, OU=Eng, O=Novell, L=Provo, ST=Utah, C=US
Serial number: 491cad67
Valid from: Thu Nov 13 15:42:47 MST 2008 until: Wed Feb 11 15:42:47 MST 2009
Certificate fingerprints:
MD5: EE:FA:EE:78:A8:42:2B:F2:3A:04:50:37:D3:94:B3:C0
SHA1: 4E:BA:9B:2F:FC:84:10:5A:2E:62:D2:5B:B3:70:70:B5:2F:03:E1:CD
Signature algorithm name: SHA1withRSA
Version: 3
Trust this certificate? [no]: yes
Certificate was added to keystore
$
Of course, I also had to change the way I was specifying my URL in the client code:
...
URL url = new URL("https://jmc-test.provo.novell.com:9015/audit/log/test");
conn = (HttpURLConnection)url.openConnection();
...
At this point, I was finally able to connect to my server and send the message. Is this reasonable? Probably not for my case. Both my client and server are within a corporate firewall, and controlled by the same IT staff, so to force this sort of gyration is really unreasonable. Can we do anything about it? Well, one thing that you can do is to provide a custom host name verifier like this:
...
URL url = new URL("https://jmc-sentinel.dnsdhcp.provo.novell.com:9015/audit/AuditLog/test");
conn = (HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier(new HostnameVerifier()
{
public boolean verify(String hostname, SSLSession session)
{ return true; }
});
conn.setRequestMethod("POST");
...
When you do this, however, you should be aware that you give up the right to treat the connection as anything but an https connection. Note that we had to change the type of “conn” to HttpsURLConnection from its original type of HttpURLConnection. This means, sadly, that this code will now only work with secure http connections. I chose to use the DNS name in my URL, although a perfectly viable option would also have been the creation of a certificate containing the IP address as an “Alternate Name”.
Is This Okay?!
Ultimately, our client code will probably be embedded in some fairly robust and feature-rich security applications. Given this fact, we can’t really expect our customers to be okay with our sample code taking over the system properties for key and trust store management. No, we’ll have to rework this code to do the same sort of thing that Tomcat does–manage our own lower-level SSL connections, and thereby import certificates and CA chains ourselves. In a future article, I’ll show you how to do this.
Open Sourcery 