The Mathematics of Planet Earth (a group of over 140 scientific societies, universities, research institutes, and foundations around the world) published an article in November 2013 by Andrea Tosin of the National Research Council’s Institute for Applied Mathematics about the predictability of human behavior. Tosin wrote, “Individual interaction rules can be interpreted in a deterministic way only up to a certain extent, due to the ultimate unpredictability of human reactions. It is the so-called bounded rationality, which makes two individuals react possibly not the same, even if they face the same conditions. In opinion formation problems this issue is of paramount importance, for the volatility of human behaviors can play a major role in causing extreme events with massive impact….”

This is exactly the problem confronting many big data and analytics efforts as they probe into the dynamic of customer behaviors and develop predictability models for when particular customers are most likely to purchase, and what they are most likely to purchase.

For years, companies have sought this 360-degree understanding of their customers with an ultimate desire of being able to understand each individual customer and to match company offers with customer needs. They have attempted to gain this 360-degree understanding through customer relationship management solutions, but those solutions are only as strong as the quality of data that is input into them by employees at call desks, sales, service, and other customer touch points.

Companies are also finding out that big data can only go so far.

Christian Madsbjerg and Mikkel Rasmussen of ReD Associates, a consultancy that assists companies in understanding human behavior, recently discussed how big data is not enough to reach big goals. They wrote in The Wall Street Journal that “In fact, companies rely too much on the numbers, graphs and factoids of big data risk, insulating themselves from the rich, qualitative reality of their customers’ everyday lives. They can lose the ability to imagine and intuit how the world — and their own businesses — might be evolving.”

This is where thick data enters the picture.

The notion behind thick data is that you can’t always depend on numerics and algorithms to summarize the 360-degree experience of a customer, or of any other human activity or relationship where unforeseeable factors can enter in.

Samsung adopted a thick data analytics approach to determine its next-generation television design concepts. Instead of solely relying on big data crunching, the company hired external help to conduct hours of interviews with customers, including the analysis of videos and conversations. Samsung wanted to understand how modern households viewed television sets as inputs to its engineering and marketing processes. What the Samsung research ultimately revealed is that people thought of TVs in their homes primarily as furniture and not as electronics.

Using non-statistical human behavior analysis for product design is nothing new. A number of years ago, I was visiting the CEO of a software report generator manufacturer, and we were talking about the importance of usability in designing reports. “You have technicians on your staff, and they know the product inside and out,” I said, “but how do you make sure that ordinary people who might be manufacturing analysts on a plant floor can easily define and run one of these reports unassisted?”

He answered, “We invite a number of our clients to our facility here on the east coast, and we ask them to work with the software on their own. While they are doing this, we videotape them. Afterwards, we analyze the videos to determine which portions of the report design process were easy and intuitive for them, and which they had trouble with. We then break down all of the trouble areas to determine what the issues were, and we find ways to ergonomically solve these.”

This is the gap that big data will be asked to close in approximating — and anticipating — the human experience.