Book Review: Super Crunchers, by Ian Ayres

January 2008 and written by Gordon Hertzman - We now live in a world with more data than ever before, and where the power to “crunch the numbers” growing exponentially. So, does more data at the click of a mouse make us any smarter? According to Ian Ayres, a Yale “econometrician” and author of the bestselling Super Crunchers, the ability to mine large datasets is the new source of competitive advantage in business. It is foundation of Google’s success, and Wal-mart won’t make a move without it. It has also made for smarter public policy and for thousands of lives saved in hospitals.

In case you were wondering, Super Crunchers has nothing to do with a cereal, but refers to the statistical analysis of large datasets or data mining for better decision making. In Super Crunchers, its Man vs. Machine. As our world becomes more complex, we just can’t see that forest for the trees the way those supercomputers can.

For those of you who have read Freakonomics, you will see this book as derivative. For those of you have not, read it first. In Freakonomics, Steven Levitt “explores the hidden side of everything”. He applies statistical on a wide range of familiar issues from parenting to the how much we cheat. In Super Crunchers, Ayres also takes us “behind the curtain”, this time to large, familiar organizations who are crunching numbers to get new insights into how to do things better.

So how does a book about data mining make the Wall Street Journal bestseller list? It’s an easy read with a minimum of jargon, with a strong focus on anecdotes we can all relate to. The book starts with the story of Orley Ashenfelter, a Princeton Economist who claimed that he could mathematically predict how good a wine vintage would be; no “swishing or spitting” required. Needless to say he was ridiculed by the wine establishment until he predicted that in 1989 (and again in 1990) that we would have the wine harvest “of the century” Today there is a lot more respect for the his notion that average temperature and rainfall is correlated to the quality of a wine harvest.

In the examples that follow we learn how companies we know as consumers have tailored their service offerings based on data mining. Under the hood, Google relies on massively large datasets for their search engine. Harrah’s Casino’s (big success story) track their customers at every touch point and swipe of their loyalty cards. They even calculate your “pain point” (how much money you are comfortable losing) and will usher you to the buffet if you get too close to it. “Happy Customers” gamble 24% more, and this adds up to millions for them. If your last flight was late Continental Airlines may buy you a drink on the next one. We all know about how Amazon recommends books. Netflix recommends other movies you would like, getting customers “off” the new releases. Blockbuster knows the probability that you will return a movie late. UPS can predict when you are about to switch to one of their competitors and will dispatch a salesperson accordingly. Monster.com refined their website using randomized testing. Best Buy and other retailers can predict how long you will work for them based on your job application. Ayres even talks about a credit card company that can calculate your probability of divorce based on a change in your spending habits. Better pay those “No Tell Motel” bills in cash…

It becomes clear that “super crunching” has paid off for many Fortune 1000 companies. This trend may in part explain the recent takeovers of the leading data miners. IBM just picked up Cognos for $5 billion and Oracle purchased Hyperion for $3.3 billion.

The world’s largest data warehouse (over 150 terabytes) is at Teradata, who is referred to in this book the way the Beatles would be in a history of rock and roll. Teradata does the “super crunching” for Wal-mart, AT&T, Best Buy, EBay, FedEx, Harrah’s, Verizon, PayPal and T-Mobile.

So who do you listen to, your “domain experts” or the analyst at Teradata? The books gives us many examples where the “truth” was only discovered on number crunching. “The numbers don’t lie”. “Google and scores of other businesses thrive in large part because they are masters of the algorithmic mindset”. This evangelistic tone the pervades the book; no balanced view here.

And what about the “dark side of data”?. Where is the mention that Harrah’s was at one time criticized for targeted marketing campaigns to compulsive gamblers, even those “on the wagon”. How about this privacy issues, or the increased intrusion of marketing efforts by companies who now know that much more about you. To his credit Ayers suggested that information should be shared so that all can benefit. Imagine getting a phone call from the IRS or Revenue Canada and hearing that if you reduce your inventory by 13%, your risk of bankruptcy will drop by 8%.

So, who wins in the end, Man or Machine? Even the BI experts will tell you it still requires human judgment to make sense of things, and there is a limit to “how much future the past can predict.” I prefer a more balanced view, and also remember that Mark Twain, quoting Disraeli, said "There are three types of lies - lies, damn lies, and statistics." And also that Gregory F. Treverton in his book Reshaping National Intelligence for an Age of Information, tells us that there is so much data out there that we can practically make a case for anything.