correct me if i’m wrong

A topic that has been weighing heavily on my mind for the last several months was brought up in a thread yesterday on Hacker News: where is the innovation in software?

Innovation should get easier over time. New giants continue to provide more shoulders for us on which to stand. Yet, it feels that the implementation of revolutionary ideas are lacking. Can you cite a new company that is doing something technology interesting? Everything nowadays is a copycat. Today, it’s a social network for X and tomorrow it’s an iPhone app for Y, where X and Y are equally absurd.

I believe we have reached an inflection point in the development of software (and by extension, technology). Software platforms are converging to a point of stability, where a small number of competing players emerge, and it’s a land grab to be the leader on those platforms.

Ultimately, we are at the point where the cost of failure is so low that it is rational to pursue an irrational venture. Many companies in the Web 2.0 era have been hit-based. By “hit-based,” I mean that the differences between social network platforms, for example, are so marginal, that the best ones ultimately emerge non-deterministically. One can focus on producing a quality product, or one can try releasing a ‘hit,’ and the network effect will lead to a tumult of new users to your product.

The prime example of hit-based products are the Friendster/MySpace/Facebook social network evolution.  All of these sites experienced explosive growth in spite of not necessarily doing anything better than their competitors. MySpace is the ugliest site on the Internet. Facebook started as no more than a CRUD site built on LAMP. These ‘qualities’ are not difficult to achieve and their competitors were and are just as capable in achieving them.

Another good example is iPhone application development. When iFart is taking off, then developers have the incentive to produce as much attention-grabbing filth as possible. While iPhone apps don’t all directly take advantage of the network effect, they certainly see an exponential rise in popularity once they gain traction.

The common denominator among all these is that they’re cheap to build, can be deployed to the masses, and have the potential of phenomenal growth and return on investment. The only way in which this trend can be stopped is when the market becomes so overly saturated that every nook, cranny and variation thereof has been exploited. Until then, we will continue to see social networking sites for babies and dead people.

So, why aren’t developers intrinsically motivated to build something better? One of America’s hottest topics is the lack of science education and aptitude among students. Americans simply aren’t being trained to possess the skills to develop innovative technology.  Universities are importing talent and hemorraghing it after graduation.  This doesn’t pertain specifically to computer science and related disciplines, but to all of the hard sciences, all of which software touches.

Lack of education is also a sign of the times.  The United States emerged from the Great Depression with the highest education rate ever.  We are currently, along with the rest of the world, entering economic times of similar magnitude.  People are rushing back to school to become more competitive and to skirt the job market.  We have already seen reports of increasing education in computer science.

As people are becoming more educated, they will feed their new knowledge back into the ecosystem.  Companies–who are already forced to innovate in tough economic times–will benefit from highly trained workers.  So, the United States, as the incumbent world leader in science and technology, is struggling to maintain it’s lead. Other countries are gaining momentum and doing a good job in spite of the dismal economy. As the economy picks up (whenever that may be), it’ll bring software innovation along with it in full force, both in the United States and abroad.

Lastly, a discussion about technology innovation would not be complete without a mention of the huge decrease in IPOs in 2008. While there is much controversy with respect to the relative shortage of IPOs in the era succeeding the 2002 Sarbanes-Oxley Act,  the simple fact of the matter is that there are much fewer large fund-raising events than usual, hence reducing the capacity and flexibility of companies to invest in research and risk-taking pursuits. Moreover, M&A is becoming the standard for startup exit strategies, which is generally detrimental to growth, as acquired companies are more likely to get axed than a public company is likely to fail (anybody remember Dodgeball?).

What can we conclude? Do you feel software innovation has been sluggish? What are your examples and counterexamples?

My take is that we, as software engineers, must become more innovative. As Alan Kay once said, “the best way to predict the future is to invent it.” We must push the boundaries of computer science. We must bridge the gap between the techies and the rest of the population. However, we must be patient. The convergence of these disparite entities will likely lead to a spurt of technological growth unprecedented in history (follow-up post soon to come!).

  Please see further discussion at Hacker News.