Scientific Innovator’s Dilemma


After some e-mail exchange with Mark Johnson about how to stimulate some far-out research that might be fun to read about, I was sitting at the dinner table with Alice Mitzi, ranting about the sociology of science.

My particular beef is low acceptance rates and the conservative nature of tenure committees, program committees, and grant review panels. It makes it hard to get off the ground with a new idea, while making it far too easy to provide a minor, often useless, improvement on something already well known. Part of the problem is that the known is just a lot easier to recognize and review. I don’t spend days on reviews like I did as a grad student — if the writer can’t explain the main idea in the abstract/intro, into the reject pile it goes without my trying to work through all the math.

Mitzi listened patiently and after I eventually tailed off, said “Isn’t that just like the innovator’s dilemma, only for science?”. Hmm, I thought, “hmm”, I mumbled, then my brain caught up and I finally let out an “a-ha”. Then I said, “I should blog about this!”.

I learned about the problem in the title of one of the best business books I’ve ever read, The Innovator’s Dilemma, by Clayton M. Christensen. It’s full of case studies about why players with the dominant positions in their industries fail. You can read the first chapter and disk drives case study online, or cut to the dryer Wikipedia presentation of disruptive technology.

The basic dilemma is that an existing business generates so much revenue and at such a high margin, that any new activity not directly related to this existing business can’t be justified. My favorite case study is of earth-movers. Back in the day (but not too far back) we had steam shovels that used cables to move their enormous shovels. They were big, and they moved lots of earth. If you needed to do strip mining, foundation digging for skyscrapers, or needed to lay out a city’s road system, these were just what you wanted. The more dirt they moved the better. So along comes the gasoline powered engine. The steam shovel companies looked at the new techology and quickly adopted it; swapping out steam for gasoline meant you could move more dirt with more or less the same set of cables. It’s what we in the software business call a “no brainer”.

A few years later, an enterprising inventor figured out how to replace the cable actuators with hydraulics. When first introduced, hydraulics were relatively weak compared to cables, so you couldn’t build a shovel big enough to compete with a cable-actuated gasoline-powered shovel. The big shovel companies looked at hydraulics, but couldn’t figure out how to make money with them. The first hydraulic shovels were tiny, being good only for jobs like digging the foundation for a house or digging a trench from a house to sewer mains. Even more importantly, there was no existing market for small earth movers compared to the much more lucrative market for big earth movers, and even if you could capture all the little stuff, it still wouldn’t affect the big company’s bottom line.

So new companies sprung up in a new market to sell hydraulic shovels that could fit on a small truck. As hydraulic technology continued to improve in strength, more and more markets opened up that took slightly more power. Even so, nothing that’d make a dent in the bottom line of the big cable-activated shovel companies.

Eventually, hydraulics got powerful enough that they could compete with cable-activated shovels. At this point, the cable-actuated shovel companies mainly went out of business. Up until just before the capabilities crossed, it still didn’t make sense in terms of the big company’s bottom line to move to hydraulics. There just wasn’t enough income in it. Until too late.

Christensen’s book is loaded with case studies, and it’s easy to think of more once you have the pattern down. The business types prefer generic, unscaled graphs like this one to illustrate what’s going on:


How disruptive technology gains a hold over time, by gradually moving into more lucrative markets (source: Wikipedia)

Smaller disks disrupted larger disks for the same reason; sure, they could fit into a minicomputer (or microcomputer), but they cost a lot per byte. At every stage of disk diameter downsizing, the dominant players mostly went bankrupt or left the business in the face of the up-and-coming smaller disk manufactures, who always managed to overtake their big-disk competitors in terms of capacity, price (and reliability, if I recall correctly). You’d think the big companies would have learned their lesson after the third iteration, but that just shows how strong a problem the innovator’s dilemma remains.

In computational linguistics and machine learning research, the big company on top is whatever technique has the best performance on some task. I thought I’d never see the end of minor variants on three-state acoustic HMMs for context-dependent triphones in speech recognition when we knew they could never sort ‘d’ from ‘t’ (here’s some disruptive “landmark-based” speech recognition). Disruptive technologies might not have state of the art performance or might not scale, but they should have some redeeming features. One could view statistical NLP as being disruptive itself; in the beginning, it only did sequences with frequency-based estimators. But remember, just because a technique performs below the best published results doesn’t make it disruptive.

The remaining dilemma is that none of the follow-on books by Christensen or others provide a good read, much less a solution to the innovator’s dilemma.

3 Responses to “Scientific Innovator’s Dilemma”

  1. Dan Says:

    If the dominant player in an industry has enough cash on hand, couldn’t they just acquire the company with the disruptive technology before it becomes a threat to their business? Of course that doesn’t really apply in the “academic” case.

  2. lingpipe Says:

    The crux of the dilemma is that when you can afford the other company, they don’t look like a threat. And there is typically a whole bunch of such companies. No one has enough money to just buy everything that might conceivably ever be a threat. And no one’s smart enough to predict where the threats come from. The disk drive case studies have a lot of examples of companies trying to anticipate the move from minicomputer disks to microcomputer disks, and mostly failing.

    A good thought exercise is this: what should Microsoft buy to prevent an erosion of its operating systems and office software businesses? What should Apple do to keep up its music business? Or if you think it’s too late for these dinosaurs, what should Google buy to prevent an erosion of its core advertising business? When should newspapers have been worried about losing classified ads to eBay and Craigslist and who should they have bought for how much?

    Or to finish with a relevant academic example, what should traditional academic publishers like Wiley or Elsevier do now in the face of self-hosted open-access journals like the Journal of Machine Learning Research? What they’re actually doing is twofold. One is moving to author-based payment, such as teh $2650 Oxford University Press would charge us to publish in one of their journals:

    The publishers are also lobbying congress to remove the law that says all content funded by NIH needs to be open source and block any such moves the government might make in the future:

    It’s a typical defend-your-existing-business move.

  3. Krishna Says:

    The Economist has an interesting article around scientific paper publishing and the “Winners Curse”.

    And they also bring up a point about scientific papers being biased towards “positive” results, this is probably true of authors as well, who probably dont want to publish “negative” results.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: