Well, I think that…
Thinking is good; few of us think often enough. Too many of us follow learnt behaviours without questioning whether they are effective or appropriate to the task at hand. But launching a statement with “I think that…” doesn’t guarantee any real thinking has been performed – and even where it has, “I think that…” usually means “I believe that…”, and belief is a strange beast. Belief takes us into the realm of faith; and so “I think that…” sounds an alarm for me which makes me hypersensitive to the words that follow; are they really the results of critical thought, or a regurgitation of ill-considered opinion?
Many organisations are being seduced by the hoopla whipped up and labelled as Big Data. A popular meme has evolved which suggests that the existence of huge volumes of extremely detailed data enables insight into human behaviour, needs and motivation; if we just tap into Big Data we can learn everything we need to know about our customers, potential customers, competitors, employees and any other group we’re interested in. The truth is that very few organisations have the capability to generate such insight from Big Data, and these organisations are utterly data-driven; they collect, create, collate, cultivate and consume data – it is woven into all of their working practices, powers measurement of all their activity and the information reaped from its effective management drives their planning, execution and thinking. These data-driven organisations eat, drink and breathe information and to enable this they place a high value on data and treat it is as precious asset.
The vast majority of our businesses, however, are not data-driven; instead we are opinion-operated – that is, we hold beliefs about our organisations and argue our case for change, or maintaining the status quo, based on these opinions. Some of these opinions may well accord with reality – they may be accurate, rational representations of our world – but many may not; and rather than reaching for evidence to support or deny these beliefs, we often posit them with false certainty and strength.
Opinion-operated businesses have little if anything to gain from Big Data.
If we desire the insight, certainty and vision of the likes of Google, Facebook and LinkedIn, and we covet their ability to divine success from Big Data, then we need to understand the forces at work within them; we need to appreciate the scientific approach that they deploy and learn to utilise it to become data-driven like them. After all, Data Science is just that; a scientific approach to the use of data, yet opinion-operated organisations prefer to consider it alchemy and entertain tales of data transmuted into insight through mysterious, powerful software tools. The IT industry has a long and ignoble history of building bandwagons and then jumping on them; Big Data is certainly the high-profile vehicle of choice for many at present, and much of the hyperbole simply serves to remind just how many false idols IT has served up over the years. The reality is that the scientific method – observe, hypothesise, test, measure and repeat – is neither mysterious nor magical and can be applied to any aspect of business. The problem is that it demands imaginative exploration of data, rigorous and critical thinking, falsifiable experimentation and carefully detailed measurement – requiring creativity, tenacity and perseverance.
In short, it is hard work without quick fixes.
There are no shortcuts to data-driven business; it is a culture, an attitude, which pervades an organisation and influences all resources, all processes and all decisions. It doesn’t replace opinions, but the difference between the opinion-operated and data-driven business is evidence; data-driven business can back up their opinions with a rich selection of information drawn from analysis of evidence, explaining the cases for and against, and presenting the reasoned understanding behind each opinion. Too often, the opinion-operated business relies on the legends of its industry, its own organisational mythology and an assortment of ancient texts – the vestiges of religion, rather than the continuous test-and-measure of science. Building a body of knowledge takes time, effort and effective management; data is collected and validated, information is synthesised from its careful analysis, and knowledge emerges from shared understanding of information once it has been appraised, considered and communicated.
This is a dynamic, continuous process; scientific knowledge is always on the move, seeking greater understanding through challenging the status quo and looking for better answers to questions posed. Data-driven businesses understand and accept this – there is no sitting back on a job well done, carving lessons learned into stone tablets – assumptions are challenged, new hypotheses derived, tests defined, evidence gathered, results measured and analysis discussed. The process goes on; the body of knowledge is refined, understanding evolves and results continue to progress.
So to move from opinion-operated to data-driven takes time and commitment, but it makes little sense to dabble in Big Data without such a commitment – oases of evidence-based reason amid a desert of faith-based superstition are weak resources and prove too little sustenance for our travels. The spectacular rise of data-driven businesses over the past fifteen years suggests that whilst the journey is arduous, the destination is more than worthwhile.
Data Animator 
A great blog Guy, so such an important challenge for whole of life, not just business (although I expect we don’t normally need such a data driven approach to our daily lives, we certainly need to be evidence based). I was chatting a few weeks ago to a strategy consultant who is often placed in the position of having to ask “How do you know?” questions as senior executives are very happy to not only make, but work on the basis of claims that are merely wishful thinking. How does one go about getting evidence? Well I think “Big Data” is hype just like “Data Warehousing” if not considered carefully. Thus those who do understand data and are evidence based will understand the limitations and context of use for big data. Some evidence will have to come from elsewhere for some problems but will always rely on data driven evidence to be reliable. But interestingly in the philosophy of science it is well understood that stark inductive and deductive reasoning is insufficient as a basis for grounding scientific progress. It must start with imagination, dare, stabs in the dark or bad guess work. Business management, like art, is even more extreme. We often simply dream up what we want to do and as such, this is perfectly valid. I think of the vision shown by Tony O’Shaughnessy at the Culbone Inn for high quality local sourced food for the South West on the basis that the community there deserve better than what they were getting. Now this isn’t data driven management and is very insightful. But then how does one go about succeeding? How can one be sure that the South West of England really want this type of dining? What micro decisions are working and what decisions aren’t? These and a host of other strategic and operations decisions require the type of evidence based management you are talking of here. But there is a branch of modern science called complexity theory. You had some exposure to that in the book “A New Kind of Science”. While this book is not the best representation of this new science it does indicate the importance to realise that a whole genre of management “science” based around traditional planning and decision making and its more genuinely scientific decision science are flawed. That one can take more and more data in order to get an asymptotically closer alignment to reality and therefore the correct decision is an illusion based on what David Snowden calls “Linear Thinking”. How the data is captured, what it represents, where it comes from, how comes to the data, how it is interpreted and analysed and then how it is acted on are all very important. In a complex world probing, experiment, trial and error are all very important. Robustness is also key. One cannot even with all the data in the world guarantee that a key action will not fail. What one needs to guarantee is that one can safely fail as part of the learning and reality engaging process.
Out of interest, the most un or even anti-scientific point in the whole blog is the use of the term “meme” – there aren’t memes in any meaningful scientific sense, so one thing for sure, there isn’t a popular meme going around. I’d challenge anyone to find it, based on a scientific and empirically sound theory. Hence why there are no longer any journals dedicated to meme theory – it went out the door along with the Phlogiston theory of heat (see Laland and Brown “Sense and Nonsense”, Maria Kronfeldner “Darwinian Creativity and Memetics” and William Harms “Information & Meaning in Evolutionary Processes”
Pete – thanks for the response, detailed and knowledgeable as always. I plead guilty to your criticism of “meme” – I was using it lazily as shorthand for ‘an idea which circulates and gains traction without due consideration’. Consider me better informed now, and suitably chastened 🙂
Hey Guy ..great point that most of organisations are opinion driven due to obvious reasons … However same applies to consumers too , they act on perception or friends reference too than numbers many times I.e. data driven methodology must have science that can read end users opinions ?
No problems Guy – your blogs are excellent and very thoughtful. I’ve only discovered them so enjoying a backlog.
Pingback: Six reasons why big data science is not working for businesses | Technology Blog·
Pingback: Intent DataSix Reasons Why Big Data Science is Not Yet Working for Businesses·