RESTful Authentication

My last post on RESTful transactions sure seemed to attract a lot of attention. There are a number of REST discussion topics that tend to get a lot of hand-waving by the REST community, but no real concrete answers seem to be forthcoming. I believe the most fundamental reasons for this include the fact that the existing answers are unpalatable – both to the web services world at large, and to REST purists. Once in a while when they do mention a possible solution to a tricky REST-based issue, the web services world responds violently – mostly because REST purists give answers like “just don’t do that” to questions like “How do I handle session management in a RESTful manner?”

I recently read an excellent treatise on the subject of melding RESTful web services concepts with enterprise web service needs. Benjamin Carlyle’s Sound Advice blog entry, entitled The REST Statelessness Constraint hits the mark dead center. Rather than try to persuade enterprise web service designers not to do non-RESTful things, Benjamin instead tries to convey the purposes behind REST constraints (in this case, specifically statelessness), allowing web service designers to make rational tradeoffs in REST purity for the sake of enterprise goals, functionality, and performance. Nice job Ben!

The fact is that the REST architectural style was designed with one primary goal in mind: to create web architectures that would scale well to the Internet. The Internet is large, representing literally billions of clients. To make a web service scale to a billion-client network, you have to make hard choices. For instance, http is connectionless. Connectionless protocols scale very well to large numbers of clients. Can you imagine a web server that had to manage 500,000 simultaneous long-term connections?

Server-side session data is a difficult concept to shoehorn into a RESTful architecture, and it’s the subject of this post. Lots of web services – I’d venture to say 99 percent of them – manage authentication using SSL/TLS and the HTTP “basic auth” authentication scheme. They use SSL/TLS to keep from exposing a user’s name and password over the wire, essentially in clear text. They use basic auth because it’s trivial. Even banking institutions use this mechanism because, for the most part, it’s secure. Those who try to go beyond SSL/TLS/basic auth often do so because they have special needs, such as identity federation of disparate services.

To use SSL/TLS effectively, however, these services try hard to use long-term TCP connections. HTTP 1.0 had no built-in mechanism for allowing long-term connections, but NetScape hacked in an add-on mechanism in the form of the “connection: keep-alive” header, and most web browsers support it, even today. HTTP 1.1 specifies that connections remain open by default. If an HTTP 1.1 client sends the “connection: close” header in a request then the server will close the connection after sending the response, but otherwise, the connection remains open.

This is a nice enhancement, because it allows underlying transport-level security mechanisms like SSL/TLS to optimize transport-level session management. Each new SSL/TLS connection has to be authenticated, and this process costs a few round-trips between client and server. By allowing multiple requests to occur over the same authenticated sesssion, the cost of transport-level session management is amortized over several requests.

In fact, by using SSL/TLS mutual authentication as the primary authentication mechanism, no application state need be maintained by the server at all for authentication purposes. For any given request, the server need only ask the connection layer who the client is. If the service requires SSL/TLS mutual auth, and the client has made a request, then the server knows that the client is authenticated. Authorization (resource access control) must still be handled by the service, but authorization data is not session data, it’s service data.

However, SSL/TLS mutual auth has an inherent deployment problem: key management. No matter how you slice it, authentication requires that the server know something about the client in order to authenticate that client. For SSL/TLS mutual auth, that something is a public key certificate. Somehow, each client must create a public key certificate and install it on the server. Thus, mutual auth is often reserved for the enterprise, where key management is done by IT departments for the entire company. Even then, IT departments cringe at the thought of key management issues.

User name and password schemes are simpler, because often web services will provide users a way of creating their account and setting their user name and password in the process. Credential management done. Key management can be handled in the same way, but it’s not as simple. Some web services allow users to upload their public key certificate, which is the SSL/TLS mutual-auth equivalent of setting a password. But a user has to create a public/private key pair, and then generate a public key certificate from this key pair. Java keytool makes this process as painless as possible, but it’s still far from simple. No – user name and password is by far the simpler solution.

As I mentioned above, the predominant solution today is a combination of CA-based transport-layer certificate validation for server authentication, and HTTP basic auth for client authentication. The web service obtains a public/private key pair that’s been generated by a well-known Certificate Authority (CA). This is done by generating a certificate signing request using either openssl or the Java keytool utility (or by using less mainstream tools provided by the CA). Because most popular web browsers today ship well-known CA certificates in their truststores, and because clients implicitly trust services that provide certificates signed by these well-known CA’s, people tend to feel warm and fuzzy because no warning messages pop up on the screen when they connect to one of these services. Should they fear? Given the service verification process used by CAs like Entrust and Verisign, they probably should, but that problem is very difficult to solve, so most people just live with this stop-gap solution.

On the server side, the web service needs to know the identity of the client in order to know what service resources that client should have access to. If a client requests a protected resource, the server must be able to validate that client’s right to the resource. If the client hasn’t authenticated yet, the server challenges the client for credentials using a response header and a “401 Unauthorized” response code. Using the basic auth scheme, the client base64-encodes his user name and password and returns this string in a response header. Now, base64 encoding is not encrytion, so the client is essentially passing his user name and password in what amounts to clear text. This is why SSL/TLS is used. By the time the server issues the challenge, the SSL/TLS encrypted channel is already established, so the user’s credentials are protected from even non-casual snoopers.

When the proper credentials arrive in the next attempt to request the protected resource, the server decodes the user name and password, verifies them against its user database, and either returns the requested resource, or fails the request with “401 Unauthorized” again, if the user doesn’t have the requisite rights to the requested resource.

If this was the extent of the matter, there would be nothing unRESTful about this protocol. Each subsequent request contains the user’s name and password in the Authorization header, so the server has the option of using this information on each request to ensure that only authorized users can access protected resources. No session state is managed by the server here. Session or application state is managed by the client, using a well-known protocol for passing client credentials on each request – basic auth.

But things don’t usually stop there. Web services want to provide a good session experience for the user – perhaps a shopping cart containing selected items. Servers typically implement shopping carts by keeping a session database, and associating collections of selected items with users in this database. How long should such session data be kept around? What if the user tires of shopping before she checks out, goes for coffee, and gets hit by a car? Most web services deal with such scenarios by timing out shopping carts after a fixed period – anywhere from an hour to a month. What if the session includes resource locks? For example, items in a shopping cart are sometimes made unavailable to others for selection – they’re locked. Companies like to offer good service to customers, but keeping items locked in your shopping cart for a month while you’re recovering in the hospital just isn’t good business.

REST principles dictate that keeping any sort of session data is not viable for Internet-scalable web services. One approach is to encode all session data in a cookie that’s passed back and forth between client and server. While this approach allows the server to be completely stateless with respect to the client, it has its flaws. First, even though the data is application state data, it’s still owned by the server, not the client. Most clients don’t even try to interpret this data. They just hand it back to the server on each successive request. But this data is application state data, so the client should manage it, not the server.

There’s no good answers to these questions yet. What it comes down to is that service design is a series of trade-offs. If you really need your web service to scale to billions of users, then you’d better find ways to make your architecture compliant with REST principles. If you’re only worried about servicing a few thousand users at a time, then perhaps you can relax the constraints a bit. The point is that you should understand the constraints, and then make informed design decisions.

RESTful Transactions

I was reading recently in RESTful Web Services (Leonard Richardson & Sam Ruby, O’Reilly, 2007) about how to implement transactional behavior in a RESTful web service. Most web services today do this with an overloaded POST operation, but the authors assert that this isn’t necessary.

Their example (in Chapter Eight) uses the classic bank account transaction scenario, where a customer wants to transfer 50 dollars from checking to savings. I’ll recap here for your benefit. Both accounts start with 200 dollars. So after a successful transaction, the checking account should contain 150 dollars and the savings account should contain 250 dollars. Let’s consider what happens when two clients operate on the same resources:

Client A -> Read account: 200 dollars
Client A -> Withdraw 50 dollars: 200 - 50 = 150 dollars
Client A -> Write account: 150 dollars

Client B -> Read account: 150 dollars
Client B -> Withdraw 50 dollars: 150 - 50 = 100 dollars
Client B -> Write account: 100 dollars

This is all well and good until you consider that the steps in these operations might not be atomic. Transactions protect against the following situation, wherein the separate steps of these two Clients’ operations are interleaved:

Client A -> Read account: 200 dollars
Client B -> Read account: 200 dollars
Client A -> Withdraw 50 dollars: 200 - 50 = 150 dollars
Client B -> Withdraw 50 dollars: 200 - 50 = 150 dollars
Client A -> Write account: 150 dollars
Client B -> Write account: 150 dollars

After both operations, the account should contain 100 dollars, but because no account locking was in effect during the two updates, the second withdrawal is lost. Thus 100 dollars was physically removed from the account, but the account balance reflects only a 50 dollar withdrawal. Transaction semantics would cause the following series of steps to occur:

Client A -> Begin transaction
Client A -> Read account: 200 dollars
Client B -> Begin Transaction (block)
Client A -> Withdraw 50 dollars: 200 - 50 = 150 dollars
Client A -> Write account: 150 dollars
Client A -> Commit transaction
Client B -> (unblock) Read account: 150 dollars
Client B -> Withdraw 50 dollars: 150 - 50 = 100 dollars
Client B -> Write account: 100 dollars
Client B -> Commit transaction

Web Transactions

The authors’ approach to RESTful web service transactions involves using POST against a “transaction factory” URL. In this case /transactions/account-transfer represents the transaction factory. The checking account is represented by /accounts/checking/11 and the savings account by /accounts/savings/55.

Now, if you recall from my October 2008 post, PUT or POST: The REST of the Story, POST is designed to be used to create new resources whose URL is not known in advance, whereas PUT is designed to update or create a resource at a specific URL. Thus, POSTing against a transaction factory should create a new transaction and return its URL in the Location response header.

A user might make the following series of web requests:

GET /transaction/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com
...
200 Ok

balance=200
---
GET /transaction/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com
...
200 Ok

balance=200

The fact that the client reads the account balances before beginning is implied by the text, rather than stated explicitly. At some later time (hopefully not much later) the transaction is started:

POST /transaction/account-transfer HTTP/1.1
Host: example.com
...
201 Created
Location: /transaction/account-transfer/11a5
---
PUT /transaction/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com

balance=150
...
200 Ok
---
PUT /transaction/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com

balance=250
...
200 Ok
---
PUT /transaction/account-transfer/11a5 HTTP/1.1
Host: example.com

committed=true
...
200 Ok

At first glance, this appears to be a nice design, until you begin to consider the way such a system might be implemented on the back end. The authors elaborate on one approach. They state that documents PUT to resources within the transaction might be serialized during building of the transaction. When the transaction is committed the entire set of serialized operations could then be executed by the server within a server-side database transaction. The result of committing the transaction is then returned to the client as the result of the client’s commit on the web transaction.

However, this can’t work properly, as the server would have to have the client’s view of the original account balances in order to ensure that no changes had slipped in after the client had read the accounts, but before the transaction was committed (or even begun!). As it stands, changes could be made by a third-party to the accounts before the new balances are written and there’s no way for the server to ensure that these other modifications are not overwritten by outdated state provided by the transaction log. It is, after all, the entire purpose of a transaction to protect a database against this very scenario.

Fixing the Problem

One way to make this work is to include account balance read (GET) operations within the transaction, like this:

POST /transaction/account-transfer HTTP/1.1
Host: example.com
...
201 Created
Location: /transaction/account-transfer/11a5
---
GET /transaction/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com
...
200 Ok

balance=200
---
PUT /transaction/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com

balance=150
...
200 Ok
---
GET /transaction/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com
...
200 Ok

balance=200
---
PUT /transaction/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com

balance=250
...
200 Ok
---
PUT /transaction/account-transfer/11a5 HTTP/1.1
Host: example.com

committed=true
...
200 Ok

The GET operations would, of course, return real data in real time. But the fact that the accounts were read within the transaction would give the server a reference point for later comparison during the execution of the back-end database transaction. If the values of either account balance are modified before the back-end transaction is begun, then the server would have to abort the transaction and the client would have to begin a new transaction.

This mechanism is similar in operation to lock-free data structure semantics. Lock-free data structures are found in low-level systems programming on symmetric multi-processing (SMP) hardware. A lock-free data structure allows multiple threads to make updates without the aid of concurrency locks such as mutexes and spinlocks. Essentially, the mechanism guarantees that an attempt to read, update and write a data value will either succeed or fail in a transactional manner. The implementation of such a system usually revolves around the concept of a machine-level test and set operation. The would-be modifier, reads the data element, updates the read copy, and then performs a conditional write, wherein the condition is that the value is the same as the originally read value. If the value is different, the operation is aborted and retried. Even under circumstances of high contention the update will likely eventually occur.

How this system applies to our web service transaction is simple: If the values of either account are modified outside of the web transaction before the back-end database transaction is begun (at the time the commit=true document is PUT), then the server must abort the transaction (by returning “500 Internal server error” or something). The client must then retry the entire transaction again. This pattern must continue until the client is lucky enough to make all of the modifications within the transaction that need to be made before anyone else touches any of the affected resources. This may sound nasty, but as we’ll see in a moment, the alternatives have less desirable effects.

Inline Transaction Processing

Another approach is to actually have the server begin a database transaction at the point where the transaction resource is created with the initial POST operation above. Again, the client must read the resources within the transaction. Now the server can guarantee atomicity — and data integrity.

As with the previous approach, this approach works whether the database uses global- or resource-level locking. All web transaction operations happen in real time within a database transaction, so reads return real data and writes happen during the write requests, but of course the writes aren’t visible to other readers until the transaction is committed.

A common problem with this approach is that the database transaction is now exposed as a “wire request”, which means that a transaction can be left outstanding by a client that dies in the middle of the operation. Such transactions have to be aborted when the server notices the client is gone. Since HTTP is a stateless, connectionless protocol, it’s difficult for a server to tell when a client has died. At the very least, database transactions begun by web clients should be timed out. Unfortunately, while timing out a database transaction, no one else can write to the locked resources, which can be a real problem if the database uses global locking. Additional writers are blocked until the transaction is either committed or aborted. Locking a highly contended resource over a series of network requests can significantly impact scalability, as the time frame for a given lock has just gone through the ceiling.

It’s clear that creating proper RESTful transaction semantics is a tricky problem.

Java HTTPS Client Issues

I’ve written in the last several months about creating a client for a RESTful web-based auditing service. In that client, I had to implement client-side authentication, which is much more involved (or it should be anyway) than writing a simple secure web client that accesses content from secure public web servers.

Such a simple secure web client has only a little more functionality than a non-secure (http) web client. Essentially, it must perform a check after each connection to the secure web server to ensure that the server certificate is valid and trustworthy. This involves basically two steps:

  1. Verifying the server’s certificate chain.
  2. Verifying the server’s host name against that certificate.

Verifying the Certificate

The purpose of step 1 is to ensure that the service you’re attempting to use is not trying
to pull something shady on you. That is, the owner of the service was willing to put his or her name on the line with a Certificate Authority (CA) like Entrust or VeriSign. When you purchase a CA-signed certificate, you have to follow various procedures that document who you are, and why you’re setting up the service. But don’t worry – the CA doesn’t get to determine if your service is worthy of public consumption. Rather, only that you are who you say you are. The CA verifies actual existence, names, addresses, phone numbers, etc. If there’s any question about the service later, a consumer may contact that CA to find out the details of the service provider. This is dangerous for scam artists because they can be tracked and subsequently prosecuted. Thus, they don’t want to deal with Certificate Authorities if they don’t have to.

The client’s verification process (step 1) usually involves following the certificates in the certificate chain presented by the server back to a CA-signed certificate installed in its own trust store. A normal Sun JRE comes with a standard JKS truststore in $JAVA_HOME/lib/security/cacerts. This file contains a list of several dozen world-renowned public Certificate Authority certificates. By default, the SSLContext object associated with a normal HTTPSURLConnection object refers to a TrustManager object that will compare the certificates in the certificate chain presented by servers with the list of public CA certificates in the cacerts trust store file.

If you have an older cacerts file that doesn’t happen to contain a certificate for a site to which you are connecting, or if you’ve set up the site yourself using a self-signed certificate, then you’ll encounter an exception when you attempt to connect:

javax.net.ssl.SSLHandshakeException: 
sun.security.validator.ValidatorException: 
PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target

Ouch! Does this mean you can’t connect to your test server while writing your client code? Can you only test against public servers? No, of course not, but unfortunately, it does mean a bit more work for you. You have basically two options. The first is to install your test server’s self-signed certificate into your default trust store. I first learned about this technique from a blog entry by Andreas Sterbenz in October of 2006. Nice article, Andreas. Thanks!

However, there is another way. You can write some temporary code in the form of your own sort of dumb trust manager that accepts any certificate from any server. Of course, you don’t want to ship your client with this code in place, but for testing and debugging, it’s nice not to have to mess up your default trust store with temporary certs that you may not want in there all the time. Writing DumbX509TrustManager is surprisingly simple. As with most well-considered Java interfaces, the number of required methods for the X509TrustManager interface is very small:

public class MyHttpsClient
{
  private Boolean isSecure;
  private String serverURL;

  private class DumbX509TrustManager 
      implements X509TrustManager 
  {
    public void checkClientTrusted(X509Certificate[] chain, 
        String authType) throws CertificateException {}

    public void checkServerTrusted(X509Certificate[] chain, 
        String authType) throws CertificateException {}

    public X509Certificate[] getAcceptedIssuers() 
        { return new X509Certificate[] {}; }
  }
  ...

To make use of this trust manager, simply obtain an SSLSocketFactory object in your client’s constructor that you can configure with your dumb trust manager. Then, as you establish https connections to your test server, install your preconfigured SSLSocketFactory object, like this:

  ...
  private SSLSocketFactory getSocketFactory()
  {
    SSLSocketFactory socketFactory = null;
    try
    {
      SSLContext context = SSLContext.getInstance("SSLv3");
      context.init(null, new X509TrustManager[] 
          { new DumbX509TrustManager() }, null);
      socketFactory = context.getSocketFactory();
    }
    catch (Exception e) { e.printstacktrace(); }
    return socketFactory;
  }

  public MyHttpsClient(String serverURL)
  {
    this.serverURL = serverURL;
    if (isSecure = serverURL.startsWith("https:"))
      sslSocketFactory = getSocketFactory();
  }

  public void process() 
  {
    try
    {
      HttpURLConnection conn = null;
      URL url = new URL(serverURL);
      try
      {
        conn = (HttpURLConnection)url.openConnection();
        if (isSecure)
        {
          HttpsURLConnection sconn = (HttpsURLConnection)conn;
              sconn.setSSLSocketFactory(sslSocketFactory);
        }
        conn.setRequestMethod(verb);
        conn.setDoOutput(false);
        conn.setDoInput(true);
        conn.connect();
        ...

That’s it. Warning: Don’t ship your client with DumbX509TrustManager in place. You don’t need it for public secure web servers anyway. If you know your client will only ever be used against properly configured public secure web servers, then you can rely on the default trust manager in the default socket factory associated with HttpsURLConnection.

If you think your client may be expected to work with non-public secure web servers with self-signed, or company-signed certificates, then you have more work to do. Here, you have two options. You can write some code similar to that found in browsers, wherein the client displays a dialog box upon connection, asking if you would like to connect to this “unknown” server just this once, or forever (where upon, the client then imports the server’s certificate into the default trust store). Or you can allow your customer to pre-configure the default trust store with certificates from non-public servers that he or she knows about in advance. But these are topics for another article.

Verifying the Server

Returning to the original two-step process, the purpose of step 2 (host name verification) is to ensure that the certificate you received from the service to which you connected was not stolen by a scammer.

When a CA-signed certificate is generated, the information sent to the Certificate Authority by the would-be service provider includes the fully qualified domain name of the server for which the new cert is intended. This FQDN is embedded in a field of the certificate, which the client uses to ensure that the server is really the owner of the certificate that it’s presenting.

As I mentioned in a previous article, Java’s keytool utility won’t let you generate self-signed certs containing the FQDN in the proper field, thus the default host name verification code will always fail with self-signed certs generated by keytool. Again, a simple dummy class comes to the rescue in the form of the DumbHostnameVerifier class. Just implement the HostnameVerifier interface, which has one required method, verify. Have it return true all the time, and you won’t see anymore Java exceptions like this:

HTTPS hostname wrong:  
should be <jmc-linux-64.provo.novell.com>

Here’s an example:

  ...
  private class DumbHostnameVerifier 
      implements HostnameVerifier
  {
    public boolean verify(String arg0, SSLSession arg1) 
        { return true; }
  }
  ...
  public void process() 
  {
        ...
        if (isSecure)
        {
          HttpsURLConnection sconn = (HttpsURLConnection)conn;
          sconn.setSSLSocketFactory(sslSocketFactory);
          sconn.setHostnameVerifier(new DumbHostnameVerifier());
        }
        ...

Scoping the Changes

A final decision you should make is the proper scope for setting the dummy trust manager and hostname verifier objects. The JSSE framework is extremely flexible. You can set these on a per-request basis, or as the class defaults, so that whenever a new HttpsURLConnection object is created, your objects are automatically assigned to them internally. For instance, you can use the following code to setup class default values:

public class MyHttpsClient
{
  private static class DumbX509TrustManager 
      implements X509TrustManager 
  {
    public void checkClientTrusted(X509Certificate[] chain, 
        String authType) throws CertificateException {}

    public void checkServerTrusted(X509Certificate[] chain, 
        String authType) throws CertificateException {}

    public X509Certificate[] getAcceptedIssuers() 
        { return new X509Certificate[] {}; }
  }

  private static class DumbHostnameVerifier 
      implements HostnameVerifier
  {
    public boolean verify(String arg0, SSLSession arg1) 
        { return true; }
  }

  private static SSLSocketFactory getSocketFactory()
  {
    SSLSocketFactory socketFactory = null;
    try
    {
      SSLContext context = SSLContext.getInstance("SSLv3");
      context.init(null, new X509TrustManager[] 
          { new DumbX509TrustManager() }, null);
      socketFactory = context.getSocketFactory();
    }
    catch (Exception e) { e.printstacktrace(); }
    return socketFactory;
  }

  static
  {
    HttpsURLConnection.setDefaultHostnameVerifier(
        new DumbHostnameVerifier());
    HttpsURLConnection.setDefaultSSLSocketFactory(
        getSocketFactory());
  }

  private String serverURL;
  
  public MyHttpsClient(String serverURL)
  {
    this.serverURL = serverURL;
  }
  ...

You can now remove the isSecure check in the process routine, because new instances of HttpsURLConnection will automatically be assigned objects of your new trust manager and hostname verifier classes – the default objects you stored in the classes with the HttpsClient class’s static initializer.

With that, you’re set to connect to any https server. Here’s a little insight for you: The difficult part – the real work – of writing https clients involves writing real code for these classes. I’ll write a future article that provides details on these processes. Again, I remind you: Don’t accidentally ship your clients with DumbHostnameVerifier in place! (Unless, of course, you want to. After all, it’s your code…)

Java Secure HTTP Keys, Part II

In my last article, I described the process of configuring client-side key and trust stores within a Java web client application. To keep it simple, I purposely used the built-in functionality of HttpsURLConnection to read certain System properties to obtain references to these credential stores, along with their passwords.

However, for an embedded client–as would be the case with library code–you’d not want to rely on any System properties, because these belong to your user and her application, not to your library. But, manually configuring the key and trust stores for a client-side https connection is a little more involved.

In this article, I’d like to show you how it’s done, and I’d like to begin by suggesting some required reading for a solid understanding of the way it all works. I’m referring to the Java Secure Socket Extensions (JSSE) Reference Guide. Since JSSE was introduced in Java 1.4, and hasn’t really changed much since then, this document is officially up to date–even in Java SE 6.

Getting Started…

Note that the process for setting up the key and trust stores hasn’t changed, so I’ll simply refer you to my previous article for this information.

To summarize, the goal here is to associate our key and trust stores with our client-side connections without specifying them in System properties. And it’s amazing the amount of extra work we have to go through in order to accomplish this seemingly simple task.

The first thing we’ll do is remove the calls to System.setProperty in our AuditRestClient constructor. We still need the values we wrote to those properties, so we’ll just convert them to constants in the AuditRestClient class. At some later point, these should undoubtedly be converted to properties that we read from our own configuration file, but for now, these constants will do:

  public class AuditRestClient
  {
    // URL components (should be configured variables)
    private static final String HTTP = "HTTP";
    private static final String HTTPS = "HTTPS";
    private static final String HOSTNAME = "10.0.0.1";
    private static final Integer PORT = 9015;

    // secure channel key material stores (should be configured)
    private static final String keystore = "/tmp/keystore.jks";
    private static final String truststore = "/tmp/truststore.jks";
    private static final String keypass = "changeit";
    private static final String trustpass = "changeit";

    // secure channel variables
    private Boolean isSecure = true;
    private SSLSocketFactory sslSocketFactory = null;

    public AuditRestClient()
    {
      setupSocketFactory();
    }
    ...

Building Your Own Socket Factory

The new version of the AuditRestClient constructor calls a private method called setupSocketFactory, which configures an SSLSocketFactory object for use later when we configure our HttpsURLConnection object. Here’s the code:

    ...
    private void setupSocketFactory()
    {
      try
      {
        String protocol = "TLS";
        String type = "JKS";
        String algorithm = KeyManagerFactory.getDefaultAlgorithm();
        String trustAlgorithm =
            TrustManagerFactory.getDefaultAlgorithm();

        // create and initialize an SSLContext object
        SSLContext sslContext = SSLContext.getInstance(protocol);
        sslContext.init(getKeyManagers(type, algorithm),
            getTrustManagers(type, trustAlgorithm),
            new SecureRandom());

        // obtain the SSLSocketFactory from the SSLContext
        sslSocketFactory = sslContext.getSocketFactory();
      }
      catch (Exception e) { e.printStackTrace(); }
    }
    ...

This private helper method calls two other private methods, getKeyManagers and getTrustManagers to configure the key and trust stores. Each of these two routines also call a routine named getStore to obtain the key and trust stores from the configured key and trust managers. Again, here’s the code for all three of these methods:

    ...
    private KeyStore getStore(String type,
        String filename, String pwd) throws Exception
    {
      KeyStore ks = KeyStore.getInstance(type);
      InputStream istream = null;

      try
      {
        File ksfile = new File(filename);
        istream = new FileInputStream(ksfile);
        ks.load(istream, pwd != null? pwd.toCharArray(): null);
      }
      finally { if (istream != null) istream.close(); }

      return ks;
    }

    private KeyManager[] getKeyManagers(String type,
        String algorithm) throws Exception
    {
      KeyStore ks = getStore(type, keyStore, keyPass);
      KeyManagerFactory kmf =
          KeyManagerFactory.getInstance(algorithm);

      kmf.init(ks, keypass.toCharArray());

      return kmf.getKeyManagers();
    }

    private TrustManager[] getTrustManagers(String type,
        String algorithm) throws Exception
    {
      KeyStore ts = getStore(type, trustStore, trustPass);
      TrustManagerFactory tmf =
          TrustManagerFactory.getInstance(algorithm);

      tmf.init(ts);

      return tmf.getTrustManagers();
    }
    ...

The getStore method calls KeyStore.getInstance to obtain an instance of the key store associated with the specified type–in this case, “JKS”. It should be noted that if you wish to specify your own provider, you may do so by calling the other version of KeyStore.getInstance, which accepts a string provider name, as well.

Using Your New Socket Factory

Now that you have your socket factory built (whew!), it’s time to look at how it’s used by the rest of the AuditRestClient code. Here’s the context for the use of the new object:

    public void send(JSONObject event)
    {
      byte[] bytes = event.toString().getBytes();
      HttpURLConnection conn = null;

      try
      {
        URL url = new URL(isSecure? HTTPS: HTTP,
            HOSTNAME, PORT, "/audit/log/test");
        conn = (HttpURLConnection)url.openConnection();
        if (isSecure)
        {
          HttpsURLConnection sconn = (HttpsURLConnection)conn;
          sconn.setSSLSocketFactory(sslSocketFactory);
        }
        conn.setRequestMethod("POST");
        conn.addRequestProperty("Content-Length", "" + bytes.length);
        conn.addRequestProperty("Content-Type", CTYPE);
        conn.addRequestProperty("Connection", "Keep-Alive");
        conn.setDoOutput(true);
        conn.setDoInput(true);
        conn.connect();
        ...

Now, this code is completely independent of application owned System properties. Additionally, it’s portable between secure and non-secure HTTP channels. This protocol portability requires a type cast of the connection from HttpURLConnection to HttpsURLConnection in one place (as highlighted in the example above in bold text).

You may have also noticed that I converted the previous version of send to use the other popular form of the URL constructor. This form accepts constituent parts of the URL as separate parameters, rather than as a single string. It’s a bit more efficient under the covers, as the constructor doesn’t need to parse these components from the URL string. It made more sense on my end, as well since I’m parameterizing several of these parts now anyway. Attributes like HOSTNAME and PORT will eventually be read from a library configuration file.

PUT or POST: The REST of the Story

Web service designers have tried for some time now to correlate CRUD (Create, Retrieve, Update and Delete) semantics with the Representational State Transfer (REST) verbs defined by the HTTP specification–GET, PUT, POST, DELETE, HEAD, etc.

So often, developers will try to correlate these two concepts–CRUD and REST–using a one-to-one mapping of verbs from the two spaces, like this:

  • Create = PUT
  • Retrieve = GET
  • Update = POST
  • Delete = DELETE

“How to Create a REST Protocol” is an example of a very well-written article about REST, but which makes this faulty assumption. (In fairness to the author, he may well have merely “simplified REST for the masses”, as his article doesn’t specifically state that this mapping is the ONLY valid mapping. And indeed, he makes the statement that the reader should not assume the mapping indicates a direct mapping to SQL operations.)

In the article, “I don’t get PUT versus POST” the author clearly understands the semantic differences between PUT and POST, but fails to understand the benefits (derived from the HTTP protocol) of the proper REST semantics. Ultimately, he promotes the simplified CRUD to REST mapping as layed out above.

But such a trivial mapping is inaccurate at best. The semantics of these two verb spaces have no direct correlation. This is not to say you can’t create a CRUD client that can talk to a REST service. Rather, you need to add some additional higher-level logic to the mapping to complete the transformation from one space to the other.

While Retrieve really does map to an HTTP GET request, and likewise Delete really does map to an HTTP DELETE operation, the same cannot be said of Create and PUT or Update and POST. In some cases, Create means PUT, but in other cases it means POST. Likewise, in some cases Update means POST, while in others it means PUT.

The crux of the issue comes down to a concept known as idempotency. An operation is idempotent if a sequence of two or more of the same operation results in the same resource state as would a single instance of that operation. According to the HTTP 1.1 specification, GET, HEAD, PUT and DELETE are idempotent, while POST is not. That is, a sequence of multiple attempts to PUT data to a URL will result in the same resource state as a single attempt to PUT data to that URL, but the same cannot be said of a POST request. This is why a browser always pops up a warning dialog when you back up over a POSTed form. “Are you sure you want to purchase that item again!?” (Would that the warning was always this clear!)

After that discussion, a more realistic mapping would seem to be:

  • Create = PUT iff you are sending the full content of the specified resource (URL).
  • Create = POST if you are sending a command to the server to create a subordinate of the specified resource, using some server-side algorithm.
  • Retrieve = GET.
  • Update = PUT iff you are updating the full content of the specified resource.
  • Update = POST if you are requesting the server to update one or more subordinates of the specified resource.
  • Delete = DELETE.

NOTE: “iff” means “if and only if”.

Analysis

Create can be implemented using an HTTP PUT, if (and only if) the payload of the request contains the full content of the exactly specified URL. For instance, assume a client issues the following Create OR Update request:

   HTTP/1.1 PUT /GrafPak/Pictures/1000.jpg
   ...

   <full content of 1000.jpg ... >

This command is idempotent because sending the same command once or five times in a row will have exactly the same effect; namely that the payload of the request will end up becoming the full content of the resource specified by the URL, “/GrafPak/Pictures/1000.jpg”.

On the other hand, the following request is NOT idempotent because the results of sending it either once or several times are different:

   HTTP/1.1 POST /GrafPak/Pictures
   ...

   <?xml version="1.0" encoding="UTF-8"?> 
   <GrafPak operation="add" type="jpeg">
     <[CDATA[ <full content of some picture ... > ]]>
   </GrafPak>

Specifically, sending this command twice will result in two “new” pictures being added to the Pictures container on the server. According to the HTTP 1.1 specification, the server’s response should be something like “201 Created” with Location headers for each response containing the resource (URL) references to the newly created resources–something like “/GrafPak/Pictures/1001.jpg” and “/GrafPak/Pictures/1002.jpg”.

The value of the Location response header allows the client application to directly address these new picture objects on the server in subsequent operations. In fact, the client application could even use PUT to directly update these new pictures in an idempotent fashion.

What it comes down to is that PUT must create or update a specified resource by sending the full content of that same resource. POST operations, on the other hand, tell a web service exactly how to modify the contents of a resource that may be considered a container of other resources. POST operations may or may not result in additional directly accessible resources.

References