Digital Fortune-Telling

“It's tough to make predictions, especially about the future.” This Yogiism may or may not have been said by Yogi Berra, but it nonetheless identifies a problem that people have been trying to solve for a long time. The invention of computers has helped the field of prediction tremendously, and we are generally better at predicting things than ever before, but, as many statisticians have pointed out, we have a long way to go. In his book The Signal and The Noise, Nate Silver shows that while we have gotten better at predicting things like the weather but still cannot predict when earthquakes will happen any better than we could 30 years ago, with some people even suggesting that earthquake prediction is simply impossible.

However, in the world of big data, there is nothing that researchers and businesses will not try to predict no matter how impossible it is. The SAT predicts how well you will do in college, Amazon predicts what you will buy, Facebook predicts what ads you will click on, and Netflix even held a competition with a $1 million dollar prize to see who could come up with the algorithm that could best predict user ratings for movies. One particularly interesting attempt at prediction is being done by venture capitalist firm Bloomberg Data, who uses an algorithm to "try to find entrepreneurs before they even start a company." [1]

They tested their algorithm through a study of 1.5 million professionals in New York and Silicon Valley, using information that was publicly available online. The article does not go into too much detail about the algorithm, but it takes into consideration things like work history and education history. According to Roy Bahat, who owns the firm, work history alone is extremely useful because "for example, if somebody's ever worked at a startup that's backed by venture capitalist then they're much more likely to start a startup in the future because that's the world they've seen." Their algorithm identified 350 entrepreneurs as having a great chance of success, and Bloomberg Beta contacted them with hopes that they could potentially invest in them in the future.

One interesting thing about this is that it is impossible to test the validity of the prediction without waiting many years to compare the predictions with the actual real-world results. This also means that algorithms like this take a long time to improve, compared to prediction algorithms that can be tested and analyzed quickly and repeatedly like weather predictions. Surely though, Bloomberg Data is not the only firm trying to make predictions like this, and I expect we will see more stories like this in the future.

More generally, this story shows the trend in attempting to use big data to make predictions that are otherwise impossible to make effectively. It also shows that big data alone does not make predictions, rather how we interpret the data is what matters. Deciding what to look at and what to ignore, differentiating between the signal and the noise, is even more difficult with more data to look at. Still, big data will be used by nearly every industry in some way and will play a big role in shaping the future.

[1] http://www.npr.org/blogs/alltechconsidered/2014/10/05/351851015/fortune-tellers-step-aside-big-data-looks-for-future-entrepreneurs