LingPipe Blog

When using Java’s default serialization, any changes to the member variables or method signatures of the class breaks backward compatibility in the sense that anything serialized from the old class won’t deserialize in the new class. So do similar changes to superclasses or changing the superclass type.

After a bit of overview, I’ll show you the trick I’ve been using to maintain backward compatibility when upgrading classes with new meaningful parameters.

Serial Version ID

The problem stems from the fact that Java uses reflection to compute a serial version ID for each class. (You can calculate it yourself using the serialver command that ships with Sun/Oracle’s JDK.) When methods or member variables change, the serial version changes. To deserialize, the serialized class’s ID must match the current class’s ID.

As long as the (non-transient) member variables don’t change, this problem can be tackled by explicitly declaring the serial version ID. (See the Serializable interface javadoc for how to declare). With an explicit serial version ID declared, the serialization mechanism uses the declared value rather than computing it from the class signature via reflection.

If there are serializable objects out there you need to deserialize, use serialver to calculate what the version ID used to be, then declare it in the class. If you’re starting from scratch, the ID can be anything.

So always use an explicit serial version when you first create a class. It can’t hurt, and it’s likely to help with backward compatibility.

Taking Control with the Externalizer Interface

But what if (non-transient) member variables (non-static class variables) change? Version IDs won’t help, because the default read and write implemented through reflection will try to deserialize the new variables and find they’re not there and throw an exception.

You need to take control using the Externalizable interface, which extends Serializable. Now you control what gets read and written.

Serialization and deserialization will still work, as long as the new member variable is either not final or is defined in the nullary (no-arg) constructor.

The Simplest Case of The Problem

Suppose we have the class

import java.io.*;
public class Foo implements Externalizable {
    static final long serialVersionUID = 42L;
    public String mArg1;
    public Foo() { }
    public void setArg1(String arg1) { mArg1 = arg1; }
    public void readExternal(ObjectInput in) 
        throws IOException, ClassNotFoundException {
        mArg1 = (String) in.readObject();
    }
    public void writeExternal(ObjectOutput out) 
        throws IOException {
        out.writeObject(mArg1);
    }
}

So far, so good. If I add the following main(),

    public static void main(String[] args) 
        throws IOException, ClassNotFoundException {
        Foo foo = new Foo();
        foo.mArg1 = "abc";
        System.out.println("1: " + foo.mArg1);

        FileOutputStream out = new FileOutputStream("temp.Foo");
        ObjectOutput objOut = new ObjectOutputStream(out);
        objOut.writeObject(foo);
        objOut.close();

        FileInputStream in = new FileInputStream("temp.Foo");
        ObjectInput objIn = new ObjectInputStream(in);
        Foo foo2 = (Foo) objIn.readObject();
        System.out.println("2: " + foo2.mArg1);
        objIn.close();
    }

and compile and run, I get

c:\carp>javac Foo.java
c:\carp>javac Foo.java
c:\carp>java Foo
1: abc
2: abc
c:\carp>ls -l temp.Foo
----------+ 1 Bob Carpenter None 31 2010-05-04 15:14 temp.Foo

So far, so good.

But now suppose I add a second variable to Foo,

public class Foo ...
...
    public String mArg2;
...
    public void setArg2(String arg2) { mArg2 = arg2; }
...

And try to run just the deserialization with the existing file,

    public static void main(String[] args) 
        throws IOException, ClassNotFoundException {

        FileInputStream in = new FileInputStream("temp.Foo");
        ObjectInput objIn = new ObjectInputStream(in);
        Foo foo2 = (Foo) objIn.readObject();
        System.out.println("2: " + foo2.mArg1);
        objIn.close();
    }

Because I haven’t changed readExternal(), we still get the same answer.

But what if I want to serialize the second argument? And give it a default value of null for classes serialized before it was added? The obvious change to read and write don’t work,

    public void readExternal(ObjectInput in) 
        throws IOException, ClassNotFoundException {
	mArg1 = (String) in.readObject();
	mArg2 = (String) in.readObject();
    }
    public void writeExternal(ObjectOutput out) 
        throws IOException {
        out.writeObject(mArg1);
        out.writeObject(mArg2);
    }

It’ll throw an exception, as in

c:\carp>javac Foo.java
c:\carp>java Foo
Exception in thread "main" java.io.OptionalDataException
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at Foo.readExternal(Foo.java:12)
        at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at Foo.main(Foo.java:37)

Now what?

The Trick: Marker Objects

Here’s what I’ve been doing (well, actually, I’m doing this through a serialization proxy, but the idea’s the same),

    public void readExternal(ObjectInput in) 
        throws IOException, ClassNotFoundException {
        Object obj = in.readObject();
        if (Boolean.TRUE.equals(obj)) {
            mArg1 = (String) in.readObject();
            mArg2 = (String) in.readObject();
        } else {
            mArg1 = (String) obj;
        }
    }
    public void writeExternal(ObjectOutput out) 
        throws IOException {
        out.writeObject(Boolean.TRUE);
        out.writeObject(mArg1);
        out.writeObject(mArg2);
    }

The new implementation always writes an instance of Boolean.TRUE (any small object that’s easily identifiable would work), followed by values for the first and second argumet. (In practice, these might be null in this class, so we’d have to be a bit more careful.) The trick here is the external read method reads an object, then checks if it’s Boolean.TRUE or not. If it’s not, we have an instance serialized by the previous version, so we cast that object to a string, assign it to the first argument and return. If it is, we read two args and assign them.

Now the result of deserializing our previous instance works again:

c:\carp>javac Foo.java
c:\carp>java Foo
2: abc

And there you have it, backward compatibility with new non-transient member variables.

The Fine Print

The trick only works if there’s an object that’s being serialized (or a primitive with a known restricted range of possible values). It doesn’t have to be the first variable, but there has to be something you can test to tell which version of the class you have.

Continuing on from the previous blog entry, The Futility of Commenting Code, I’d like to address the dissenting comment of Sandman, who claims to have only seen the kind of caricature comments I made up in posts about commenting. Well, here goes reality.

A Real Example

Consider the following real example, which I chose because I knew the source for Apache projects was available online, and suspecting Apache Commons would be less heavily checked than more popular projects. I dove into the e-mail package, because it’s the first one I actually use. I found this example right away:

• source for org.apache.commons.mail.ByteArrayDataSource (revision 782475)

pieces of which I repeat here:

 /**
 * Create a datasource from a String.
 *
 * @param data A String.
 * @param aType A String.
 * @throws IOException IOException
 */
public ByteArrayDataSource(String data, String aType) throws IOException {

    this.type = aType;
    try {
        baos = new ByteArrayOutputStream();
      
        // Assumption that the string contains only ASCII
        // characters! Else just pass in a charset into this
        // constructor and use it in getBytes().
        baos.write(data.getBytes("iso-8859-1"));
        baos.flush();
        baos.close();
    } catch (UnsupportedEncodingException uex) {
        throw new IOException("The Character Encoding is not supported.");
    } finally {
        if (baos != null) {
            baos.close();
        }
    }
}

I wasn’t talking about API comments, but these display the same problem. This public constructor is documented with “Create a datasource from a String”, but in fact, there are two string parameters, both documented as “a String”. That’s what the delcaration says, so this is exactly the kind of redundant comment I was talking about.

Next, consider the one code comment. It starts on the wrong foot, with “Assumption that the string contains only ASCII characters!”. If you look at the method call, data.getBytes("iso-8859-1"), you’ll see that it’s actually more general than documented, in that it’ll work for any ISO-8859-1 characters (aka Latin1). The second part of the comment, “Else just pass in a charset into this constructor and use it in getBytes()” makes no sense, because the bytes are hard coded, and there is no char encoding argument to the constructor.

Furthermore, the catch block (with its throw) should just be removed. It’s catching an UnsupportedEncodingException, which extends IOException, then throwing a fresh IOException. It should just be removed, at which point an unsupported encoding throws an unsupported encoding exception; you don’t even need to change the signature, because unsupported encoding exceptions are kinds of I/O exceptions. There are two problems with the code as is. First, the the IOException reports an unhelpful message; the unsupported encoding exception has the info on what went wrong in its message. Second, it’s returning a more general type, making it harder for catchers to do the right thing. You might also consider the fact that the original stack trace is lost a problem.

Another instance of (almost) commenting the language is later in the same file:

try {
    int length = 0;
    byte[] buffer = new byte[ByteArrayDataSource.BUFFER_SIZE];
         
    bis = new BufferedInputStream(aIs);
    baos = new ByteArrayOutputStream();
    osWriter = new BufferedOutputStream(baos);
         
    //Write the InputData to OutputStream
    while ((length = bis.read(buffer)) != -1) {
        osWriter.write(buffer, 0, length);
    }
    osWriter.flush();
    osWriter.close();
} ... 

I’d argue comments like “Write the InputData to OutputStream” are useless, because this is the idiom for buffered writes.

Aside on Bad Code

Maybe you think you should always buffer streams. In this case, that’s wrong. Both bufferings simply cause twice as many assignments as necessary.

Buffering the input stream is useless because you’re already reading into the byte array buffer.

Buffering the output stream is useless, because you’re writing to a byte array output stream baos.

Furthermore, you don’t need to close or flush a byte array output stream, because both are no-ops (see the ByteArrayOutputStream javadoc).

Finally, the naming is wonky, because osWriter is not a writer, it’s an output stream.

This isn’t an isolated case. Other files in this package have similar problems with doc.

Another Example

While we’re picking on Apache Commons, I checked out another package I’ve used, fileUpload.

• org.apache.commons.fileupload/DiskFileUpload:

public class DiskFileUpload extends FileUploadBase {
    // ------------------------------------ Data members

    /**
    * The factory to use to create new form items.
    */
   private DefaultFileItemFactory fileItemFactory;
	
    // ------------------------------------ Constructors 

That’s the kind of pre-formatted useless stuff I was talking about. We know what a member variable and constructor look like in Java. There weren’t any other code comments in that file.

In that same package, the fileupload.ParameterParser class has some, though, and I’m guessing they’re of the kind that others mentioned as “good” comments, such as:

    ...
    // Trim leading white spaces
    while ((i1 < i2) && (Character.isWhitespace(chars[i1]))) {
        i1++;
    } 
    ...

I’d argue that perhaps a method called trimLeadingWhiteSpace() implemented the same way would be clearer. But if you’re not going to define new methods, I’d say this kind of comment helps. But always verify that the code does what it says it does; don’t take the comment’s word (or method name’s word) for it.

I couldn’t leave that file without commenting on their return-only-at-the-end-of-a-function style:

String result = null;
if (i2 > i1) {
    result = new String(chars, i1, i2 - i1);
}
return result; 

I have no idea why people do it, but as you can see in this case, it just makes the code hard to follow.

More Examples

I thought only a couple wouldn’t be convincing. So here’s some links and examples:

• collections.buffer.BoundedBuffer

public boolean isFull() {
    // size() is synchronized
    return (size() == maxSize());
} 

Documenting what’s clear from the code. (And nice section divider comments.)

• digester.Digester

 // Return the parser we already created (if any)
if (parser != null) {
    return (parser);
} 
...
 } catch (RuntimeException e) {
    // rethrow, after logging
    log.error(e.getMessage(), e);
    throw e;
} 

Yes, that’s what return, log, and throw do.

• logging.impl.LogFactoryImpl

public LogFactoryImpl() {
    super();
    initDiagnostics(); // method on this object 

Yes, it really says “method on this object”.

There’s lots more, so I’ll leave finding them as an exercise.

It’s Not All Bad

There were useful code comments in those packages that explained how the code corresponded to an algorithm in Knuth, or how complex invariants were being preserved in complex loops. I’m really not saying don’t comment code at all.

I often read about novice programmers’ aspirations for a greater density of code comments, because they think it’s “professional”. I have news for you. Professional coders don’t comment their own code much and never trust the comments of others they find in code. Instead, we try to learn to read code and write more readable code.

API Documentation vs. Code Comments

I’m a big believer in clear API documentation. Java makes this easy with javadoc. It even produces relatively spiffy results.

But if you look at the LingPipe source code, you’ll find very few code comments.

Comments Lie

The reason to be very suspicious of code comments is that they can lie. The code is what’s executed, so it can’t lie. Sure, it can be obfuscated, but that’s not the same as lying.

I don’t mean little white lies, I mean big lies that’ll mess up your code if you believe them. I mean comments like “verifies the integrity of the object before returning”, when it really doesn’t.

A typical reason for slippage between comments and code is patches to the code that are not accompanied by patches to the comments. (This can happen for API doc, too, if you fiddle with your APIs.)

Another common reason is that the code author didn’t actually understand what the code was doing, so wrote comments that were wrong. If you really mess up along these lines, your code’s likely to be called out on blogs like Coding Horror or Daily WTF.

Most Comments Considered Useless

The worst offenses in the useless category are things that simply repeat what the code says.

// Set x to 3
x = 3;

It’s even better when embedded in a bloated Fortran-like comment block:

/******************************************
 ************* add1 ***********************
 ******************************************
 *  A function to add 1 to integers       *
 *  arguments                             *
 *     + n, any old integer               *
 *****************************************/
public int add1(int n) { return n+1; }

Thanks. I didn’t know what int meant or what n+1 did. This is a classic rookie mistake, because rookies don’t know the language or its libraries, so often what they do seems mysterious to them. For instance, commenting n >>> 2 with “shift two bits to right and fill in on the left with zeroes” may seem less egregious, but your reader should know that >>> is the unsigned shift operator (or look it up).

There is a grey line in the sand (I love mixed metaphors) here. When you’re pulling out techniques like those in Bloch and Gafter’s Java Puzzlers, you might want to reconsider how you code something, or adding a wee comment.

Eliminate, don’t Comment Out, Dead Code

I remember being handed a 30-page Tcl/Tk script at Bell Labs back in the 1990s. It ran some kind of speech recognition pipeline, because that’s what I was up to at the time. I picked it up and found dozens of methods with the same name, and lots of commented-out code. This makes it really hard to follow what’s going on, especially if whole blocks get commented out with /* ... */.

Please don’t do this. You should lLearn any of thea version control systems instead, like SVN.

When do I Comment Code?

I add comments in code in two situations. The first is when I wrote something inefficiently, but I know a better way to do it. I’ll write something like “use dynamic programming to reduce quadratic to linear”. This is a bad practice, and I wish I could stop myself from doing it. I feel bad writing something inefficient when I know how to do it more efficiently, and I certainly don’t want people reading the code to think I’m clueless.

I know only one compelling reason to leave comments: when you write code that’s not idiomatic in the language it’s written in, or when you do something for efficiency that’s obscure. And even then, keep the comments telegraphic, like “C style strings for efficiency”, “unfolds the E step and M step of EM”, “first step unfolded for boundary checks” or something along those lines.

Update: 13 Dec 2012. I’ve thought about this issue a bit more and wanted to add another reason to comment code: to associate the code with an algorithm. If you’re implementing a relatively complex algorithm, then you’re going to have the algorithm design somewhere and it can be helpful to indicate which parts of the code correspond to which parts of the algorithm. Ideally, though, you’d just write functions with good names to do the stages of the algorithm if they’re clear. But often that’s not really an option because of the shape of the algorithm, mutability of arguments, etc.

Also, I want to be clear that I’m not talking about API comments in code, which I really like. For instance, we do that in LingPipe using Javadoc. I think these comments are really really important, but I think of them somehow more as specifications than as comments.

Write Readable Code Instead

What you should be doing is trying to write code that’s more readable.

I don’t actually mean in Knuth’s literate programming sense; Knuth wants programs to look more like natural language, which is a Very Bad Idea. For one, Knuth has a terrible track record, bringing us TeX, which is a great typesetting language, but impossible to read, and a three-volume set of great algorithms written in some of the most impenetrable, quirky pseudocode you’re ever likely to see.

Instead, I think we need to become literate in the idioms and standard libraries of the programming language we’re using and then write literately in that language.

The biggest shock for me in moving from academia to industry is just how much of other people’s code I have to read. It’s a skill, and I got much better at it with practice. Just like reading English.

Unfortunately, effective programming is as hard, if not harder, than effective essay writing. Writing understandable code that’s also efficient enough and general enough takes lots of revision, and ideally feedback from a good code reviewer.

Not everyone agrees about what’s most readable. Do I call out my member variables from local variables with a prefix? I do, but lots of perfectly reasonable programmers disagree, including the coders for Java itself. Do I use really verbose names for variables in loops? I don’t, but some programmers do. Do I use four-word-long class names, or abbreviations? The former, but it’s another judgment call.

However you code, try to do it consistently. Also, remember that coding standards and macro libraries are not good places to innovate. It takes a while to develop a consistent coding style, but it’s worth the investment.