After some e-mail exchange with Mark Johnson about how to stimulate some far-out research that might be fun to read about, I was sitting at the dinner table with
Alice Mitzi, ranting about the sociology of science.
My particular beef is low acceptance rates and the conservative nature of tenure committees, program committees, and grant review panels. It makes it hard to get off the ground with a new idea, while making it far too easy to provide a minor, often useless, improvement on something already well known. Part of the problem is that the known is just a lot easier to recognize and review. I don’t spend days on reviews like I did as a grad student — if the writer can’t explain the main idea in the abstract/intro, into the reject pile it goes without my trying to work through all the math.
Mitzi listened patiently and after I eventually tailed off, said “Isn’t that just like the innovator’s dilemma, only for science?”. Hmm, I thought, “hmm”, I mumbled, then my brain caught up and I finally let out an “a-ha”. Then I said, “I should blog about this!”.
I learned about the problem in the title of one of the best business books I’ve ever read, The Innovator’s Dilemma, by Clayton M. Christensen. It’s full of case studies about why players with the dominant positions in their industries fail. You can read the first chapter and disk drives case study online, or cut to the dryer Wikipedia presentation of disruptive technology.
The basic dilemma is that an existing business generates so much revenue and at such a high margin, that any new activity not directly related to this existing business can’t be justified. My favorite case study is of earth-movers. Back in the day (but not too far back) we had steam shovels that used cables to move their enormous shovels. They were big, and they moved lots of earth. If you needed to do strip mining, foundation digging for skyscrapers, or needed to lay out a city’s road system, these were just what you wanted. The more dirt they moved the better. So along comes the gasoline powered engine. The steam shovel companies looked at the new techology and quickly adopted it; swapping out steam for gasoline meant you could move more dirt with more or less the same set of cables. It’s what we in the software business call a “no brainer”.
A few years later, an enterprising inventor figured out how to replace the cable actuators with hydraulics. When first introduced, hydraulics were relatively weak compared to cables, so you couldn’t build a shovel big enough to compete with a cable-actuated gasoline-powered shovel. The big shovel companies looked at hydraulics, but couldn’t figure out how to make money with them. The first hydraulic shovels were tiny, being good only for jobs like digging the foundation for a house or digging a trench from a house to sewer mains. Even more importantly, there was no existing market for small earth movers compared to the much more lucrative market for big earth movers, and even if you could capture all the little stuff, it still wouldn’t affect the big company’s bottom line.
So new companies sprung up in a new market to sell hydraulic shovels that could fit on a small truck. As hydraulic technology continued to improve in strength, more and more markets opened up that took slightly more power. Even so, nothing that’d make a dent in the bottom line of the big cable-activated shovel companies.
Eventually, hydraulics got powerful enough that they could compete with cable-activated shovels. At this point, the cable-actuated shovel companies mainly went out of business. Up until just before the capabilities crossed, it still didn’t make sense in terms of the big company’s bottom line to move to hydraulics. There just wasn’t enough income in it. Until too late.
Christensen’s book is loaded with case studies, and it’s easy to think of more once you have the pattern down. The business types prefer generic, unscaled graphs like this one to illustrate what’s going on:
Smaller disks disrupted larger disks for the same reason; sure, they could fit into a minicomputer (or microcomputer), but they cost a lot per byte. At every stage of disk diameter downsizing, the dominant players mostly went bankrupt or left the business in the face of the up-and-coming smaller disk manufactures, who always managed to overtake their big-disk competitors in terms of capacity, price (and reliability, if I recall correctly). You’d think the big companies would have learned their lesson after the third iteration, but that just shows how strong a problem the innovator’s dilemma remains.
In computational linguistics and machine learning research, the big company on top is whatever technique has the best performance on some task. I thought I’d never see the end of minor variants on three-state acoustic HMMs for context-dependent triphones in speech recognition when we knew they could never sort ‘d’ from ‘t’ (here’s some disruptive “landmark-based” speech recognition). Disruptive technologies might not have state of the art performance or might not scale, but they should have some redeeming features. One could view statistical NLP as being disruptive itself; in the beginning, it only did sequences with frequency-based estimators. But remember, just because a technique performs below the best published results doesn’t make it disruptive.
The remaining dilemma is that none of the follow-on books by Christensen or others provide a good read, much less a solution to the innovator’s dilemma.