0. I’m a bit more optimistic than Breck that it’s getting close to some of the kinds of questions people have. Part of the issue is whether the system can learn when it’s going to be wrong in a more heterogeneous question-answering environment. So far, NLP technologies are not very good at that (look, for instance, at the number of false positives from relatively easy problems like web query spell checking).

1. Going back to Deep Blue (IBM’s chess-playing program), you could say the same thing about it in comparison to a human. IBM didn’t replicate a human chess player, they built something different with different abilities. Most specifically, the ability to beat a human at a game that seemed, at least until the point of IBM’s entry, to require intelligence.

I’d say that in many ways Watson is better than a reference librarian. If you just allowed Watson to show you the source of its answer, it’d be more believable still.

In fact, plain old search engines like Bing or Google work better than generic reference librarians for the kinds of queries I make (e.g., [hamiltonian monte carlo convergence rate]).

2. The analogy to flight reminds me of Hynek Hermansky’s 1998 ICSLP talk. About that time, the funding agencies in the US and Europe were declaring speech a “solved problem”. Hynek put up images ranging from hot air baloons (perhaps a better analogy to where we’re at now), through to jets, and asked when the flight problem was solved, and if it was solved, why research into flight was such a large part of DARPA and other funding agency’s research budget.

3. Temporal questions are a mess if you can’t add a specific date.

I also noted that the Jeopardy’s rich keyword sets result in correct “answers” being highly ranked in search engines. Even less realistic are the Jeopardy “questions” which describe two of the correct answer’s senses. Those are nice problems but aren’t realistic to automatic Q/A (“This plain-weave, sheer fabric made with tightly twisted yard is also used to describe a pie or a cake”).

That being said I don’t want to diminish IBM’s fantastic accomplishment and applaud the progress that they made in this sort of “first flight”.