Order Neil Peterson's
Embracing the Edge
on Amazon today.

Or buy it at the Edge Foundation. All profits from the sale of the book go to support the mission of the Edge Foundation.

View Neil's latest traveling blogs





Neil Peterson is an affiliate member of the National Speakers Association and frequently speaks to organizations on a variety of topics. Learn more

You may contact Neil about speaking engagements at:
(206) 910-7515

By Neil Peterson | April 4, 2009

Statistical analysis and data mining are having their day in the sun. Over the past several years, what has been a staid and incomprehensible science has evolved into a (semi) pop culture darling as economists, government officials and business leaders around the world have started using statistics to show us surprising social truths, hidden behind the mass of numbers collected about us and stored in databases. These new analytical decision makers eschew taking action based solely on expertise or experience based intuition.  Instead they use sophisticated data mining techniques to learn more about us and what we have done, as well as predict what we are likely to do in the future.

Yale Professor, lawyer, and economist Ian Ayres explores this new numerical view of the world in his latest book, Super Crunchers: Why Thinking by Numbers is the New Way to be Smart. Ayres is asking for a departure from intuitive decision making to a data-based approach, using algorithms that are far more accurate than human predictions could ever be. And this new way of thinking could effect our lives on a profound level – from who we choose to marry or which government programs receive funding, to the type of care you will receive in a doctor’s office.

Interview with Ian Ayres, author of Super Crunchers

Customers of online retailers such as Amazon.com have relied on other user’s testimonials for years, but the game changed when Amazon realized that by recommending items, based on a person’s past purchases – in correlation with similar purchases made by others – they could drive their sales and build consumer loyalty. Netflix, the popular DVD-by-mail site, uses their data in a similar fashion, suggesting movies based on the user’s previous rentals – in tandem with the numerous ratings given by millions of other subscribers.

But this data-based prediction model is not just about sales. One of the many excellent examples in Ayres’ book is online dating service eHarmony. After studying over 5,000 married couples, eHarmony founder Neil Clark Warren developed a predictive model based on twenty-nine separate variables. Using a client questionnaire, the site seeks to pull personality characteristics that the responder may not even be consciously aware of. Using this ‘hidden’ data, and a database of past client information, eHarmony’s algorithms work to find a match that is more compatible across a greater number of aspects.

Ayres also sees great promise for the use of statistical predictions in the field of medical diagnostics. Creating a database of symptoms and diseases could expedite difficult diagnoses – potentially preventing thousands of deaths a year, as well as helping hospitals in developing new protocols to guard against post-op infections and complications.

As the technologies of data collection and data mining get more sophisticated, the amount of raw data we are collecting is overwhelming; the volume of data is so large it now has to be measured in petabytes (a “petabyte” is a quadrillion bytes).   The data is also astonishing in what it can tell us about ourselves and our probable future.  We don’t need to feel threatened by the emergence of analytics based decision making.  By asking the right questions (and Ayres insists, checking and re-checking) we can make this data work for us – and make our lives easier, safer, and more enjoyable.