Big Data. Super Bowl. Small Minds.

You may have heard of the “Super Bowl Stock Market Indicator” by now, right? It is well known enough to have its own entries in both Investopedia and Wikipedia, after all. The Super Bowl Stock Market Indicator holds that when a team from the old AFL (AFC division) wins, the market will decline in the coming year, while if a team from the old NFL (NFC division) wins, then the market will be up for the year.
Investopedia says that, historically speaking, the indicator has proved 80% accurate since the 1960’s. And this year, both teams were from the old NFL, so it should be a good year for the stock market. There was a small glitch with the line-up for Super Bowl XLVII, however. Technically the Ravens are an expansion team, having come into being after the AFL-NFL merger, so some (but not all) “experts” on these matters consider the 49ers to be the only authentic old-NFL team.
Since the Ravens won the Super Bowl, I have a prediction about the state of the Super Bowl Stock Market Indicator next year. If the market goes up this year, then we’ll be told that the Indicator still works. If the market goes down, then we’ll be told that the Indicator still works, because the 49ers were the only authentic old-NFL team, and they lost.
Either way, this is a classic example of a spurious correlation (which is the technical term for a "random coincidence"), and spurious correlations like this are going to be more and more common as computers make it possible to parse data in ever more ways. Before the election, for example, Republicans had their own Indicator, which was also a spurious correlation. Since 1845, when national elections were permanently changed to occur on the first Tuesday in November, no Republican presidential candidate had ever lost a November 6 election, while six had won, so this Indicator had proved 100% accurate. Until November 6, 2012, that is.
Big Data. Anybody sick of that term yet? No one should doubt the benefits that could flow from careful analysis of the many terabytes of unstructured data being generated every day, through social media commentary, YouTube videos, contact center conversations, and the like. McKinsey estimates that retailers alone could increase their margins by up to 60%, and European governments could reduce their operating expenses by more than €100 million, simply by making full use of the data now available.
But the downside to this surfeit of data is the shortage of expertise available to deal with it. The same McKinsey report suggests that within five years, the US alone could face a shortage of more than 150,000 professionals with the technical skills necessary to parse, analyze and make sense of Big Data.
If your kids will soon be on their way to college, and provided they have any quantitative talent at all, it would be hard to find a more promising career asset than a good background in statistics. And if you’re a business executive and you don’t want to be benched prematurely, you should probably start boning up a little bit on statistics yourself.
But I have another challenge, as well: As a free and democratic society, if we all don't all begin to show a bit more sophistication in our approach to numbers and statistics, then we will be condemned to continue reading the tea leaves of spurious correlations and false analyses fed to us by an equally innumerate press corps. Don't let technology make your brain obsolete!

Original Article : http://www.linkedin.com/today/post/article/20130204035821-17102372-big-data-super-bowl-small-minds