Intuition and Creativity Process in Big Data Science
“We live in the era of information and the trends that are hidden in the streams of data points,” writes TechNewsWorld’s Anjul Bhambhri. “Those who ask the right questions and apply the right technologies and talent are certain to crack the curious case of big data.”
“Even if you have petabyes of data, you still need to know how to ask the right questions to apply it.” So writes Alistair Croll, a founding partner at start-up accelerator Year One Labs and an analyst at Bitcurrent.
Croll cites the story of a friend, one which represents the classic example of a firm not asking the right questions with regard to Big Data: “He’s a ridiculously heavy traveler, racking up hundreds of thousands of miles in the air each year. He’s the kind of flier airlines dream of: loyal, well-heeled, and prone to last-minute, business-class trips. He’s is exactly the kind of person an airline needs to court aggressively, one who represents a disproportionately large amount of revenues. He’s an outlier of the best kind. He’d been a top-ranked passenger with United Airlines for nearly a decade, using their Mileage Plus program for everything from hotels to car rentals. And then his company was acquired. The acquiring firm had a contractual relationship with American Airlines, a competitor of United with a completely separate loyalty program. My friend’s air travel on United and its partner airlines dropped to nearly nothing. He continued to book hotels in Shanghai, rent cars in Barcelona, and buy meals in Tahiti, and every one of those transactions was tied to his loyalty program with United. So the airline knew he was traveling – just not with them. Astonishingly, nobody ever called him to inquire about why he’d stopped flying with them. As a result, he’s far less loyal than he was. But more importantly, United has lost a huge opportunity to try to win over a large company’s business, with a passionate and motivated inside advocate.”
Croll continues: “Ultimately, this is what my friend’s airline example underscores. It takes an employee, deciding that the loss of high-value customers is important, to run a query of all their data and find him, and then turn that into a business advantage. Without the right questions, there really is no such thing as big data.”
Per another commentator: “Apparently, business schools [are beginning to teach a skill generally] called ‘data-based decision-making,’ suggesting that the skill is reducible to pedagogical form. But ‘asking the right question’ remains more of an art than a science. It requires practice, patience, and time.”
“Data analytics were once considered the purview of math, science and information-technology specialists,” notes the Wall Street Journal. “Now barraged with data from the Web and other sources, companies want employees who can both sift through the information and help solve business problems or strategize. For example, luxury fashion company Elie Tahari Ltd. uses analytics to examine historical buying patterns and predict future clothing purchases. Northeastern pizza chain Papa Gino’s Inc. uses analytics to examine the use of its loyalty program and has succeeded in boosting the average customer’s online order size. As the use of analytics grows quickly, companies will need employees who understand the data. A … study from McKinsey & Co. found that by 2018, the U.S. will face a shortage of 1.5 million managers who can use data to shape business decisions.”
But as Kevin Weil, Product Lead for Revenue at Twitter, put it during a recent talk, “asking the right question is hard.” Which is the best explanation of why people like Kevin are so important. (As the head of the analytics team at Twitter, Weil is tasked with building distributed infrastructure and leveraging data analysis at a massive scale to help grow the popular micro-blogging service. With millions of monthly site visitors and much more interacting through API-based third party applications, Twitter has one of the world’s most varied and interesting data sets.)
“The fact is that even when the boundaries of a dataset are narrowly defined …, ” writes Stephen O’Grady, cofounder of RedMonk, “it’s easy to get lost in it. The trick is no longer merely being able to aggregate and operate on data; it’s knowing what to do with it. Find the people that can do that, whether they’re FTE’s or consultants, and you’ll have your competitive advantage. To [ask and] answer the right questions, you need the right people.”
Simply put: “Big Data becomes Big Intelligence (otherwise known as Business Intelligence) only when put in the hands of the right people enabled to ask the right questions at the right time, on a huge scale. Anything else risks the information becoming redundant and the BI worthless before it’s even discovered.” So comments industry analyst Mike Pilcher. (Note that most practitioners also insist that, along with asking the right questions, it is important to eliminate bias, and correlation from causality.)
Weil is correct that asking the right question (or questions) is not easy.
Productivity guru Tony Robbins notes that thinking is a process of asking and answering questions. He stresses the importance of asking the right questions to get the right answers and therefore the right results. The wrong questions lead to useless answers and no results, at least no positive results.
Leadership guru Michael Hyatt, says the same thing in his own way: “Questions are powerful tools. They can ignite hope and lead to new insights. They can also destroy hope and keep us stuck in bad assumptions. The key is to be intentional and choose our questions well.”
Perhaps a better phrase than ask the right question is innovate the right question. Innovation is key. Or, to resort to a cliche: be sure to think outside the box. (Einstein once said that if he only had an hour to solve a specific problem and his life depended on it, he’d devote the first 55 minutes to figuring out the right question to answer.)
Effective questioners look at an existing reality (data) from multiple (new) perspectives.
“Of course, it’s not just a matter of being willing to question – it’s also important to know how to question,” writes Warren Berger (author of CAD Monkeys, Dinosaur Babies, and T-Shaped People: Inside the World of Design Thinking). “Innovation is driven by questions that are original, bold, counter-intuitive, and perceptive. … Coming up with the right question, the one that casts a familiar challenge in a new light, is an art and a science in itself. It demands that the questioner be able to look at an existing reality from multiple viewpoints, including, perhaps most importantly, that of the ‘naive outsider.'”
Creative questioning is linked to the capacity to tolerate not knowing, to seek out paradoxes, to withstand the temptation of early closure, and to nurture the “courage of one’s own stupidity” in questioning commonly accepted assumptions.
“You don’t know what you don’t know,” says Bain consultant and partner Rasmus Wegener, “and if you don’t know, it is hard to come up with the right question. You need to be well-versed in both the business and the data.” Then you have to begin to bravely ask why and what.
Why are our digital subscription renewals down 10% in Boston, but booming everywhere else, and what available data can we merge and sort creatively in order to move toward an answer? What customer-appreciation program enhancements will best serve our purpose of improving user retention, and how can we leverage customer-appreciation-points usage data to infer an answer? What trends can we expect to see in vis-a-vis bandwidth usage on our network come Superbowl Sunday? (What spike did we see last year on the same day? What percentages of the spike represented cell phones, tablets, PCs? How have the hardware demographics of our users changed in the past 365 days. And what is the most efficient, logical way to correlate this data and infer an answer?)
In sum: Move forward bravely – but rationally – into the unknown. Think on your feet. Realize fully what data is at your fingertips. Think analytically and creatively about how to leverage that combined data to learn and predict. Reach for knowledge. Go for it.