Customer experience and innovation expert Woody Bendle is back today with his big, big, big, big strategic insights on big data analytics. Here's Woody!
Big Data is HOT!
Look at this Google Trends search on “Big Data” from this past Saturday (August 08, 2013). You don’t have to be a statistician or a Nobel Prize winning macro economist to see Big Data has been on an AMAZING upward trajectory since 2011!
There is no denying Big Data is in vogue right now. Some might even say it’s pretty darned sexy! But, as someone whose been tackling ‘Big Data’ for more than 20 years, I have to ask, “What the heck’s the big deal?” We’re just talking about data - albeit, more of it.
Look, Big is relative; and as long as you know what you’re doing, data size should not be an issue given the current state and price of computing technologies today. This leads me to think there is something more to all of this ‘Big Data’ chatter than simply terabytes.
Perhaps all of the hoo-hah surrounding Big Data analytics has to do with the different types of data out there.
Big Data encapsulates A LOT of different data types ranging from good (accurate and reliable) to bad (wrong and inconsistent). Big Data can also be structured (numbers, etc.) or unstructured (a video posted on YouTube with someone railing on your company). Also, some Big Data reflect location (latitude and longitude or a check-in on FourSquare) as well as things happening over time. WHEW!
And for good measure, I even have my own classification for the different types of data (Big or not) I regularly encounter – these are Woody’s data ABCs.
So sure, I’ll admit it, Big Data can be pretty complicated and complex. But this is what data analysis has always been about – for years. This leads me to think there is still something more to this whole Big Data thing than just data size and data complexity.
This is something I’ve been thinking more about lately, and I tend to believe the big deal has to do with the confluence of a number of technological trends that surprisingly snuck up on a lot of people, as well as the fundamental laws of demand and supply.
On the technology side, we now live a new hyper-digital era where due to advancements in computing capacity and speed, data capture and storage in conjunction with rapidly decreasing costs, virtually every move we make throughout every living moment of our lives is registered digitally. Billions of people all doing hundreds (or thousands) of different things every day – all captured and memorialized in some digital form in the cloud.
In addition, the Internet revolution has enabled all sorts of technological, consumer , and social innovations which now allow people to create and share more data in one day than many companies used to generate in a pre-digital era year!
Think about all of the data each of us create every day through emails, text messages, Facebook pages, Twitter, Instagram, Pinterest, Vine, Yelp, Wordpress, Blogger, etc. And, not to mention the continual digital wake we leave behind with the GPS transmitters in our smartphones and cars. All of these things individually and in combination generate more data than most of us can fathom – and certainly way more data than most in the business world are prepared to use. And by use, I mean actually utilize in such a manner as to create new value for consumers, society, and businesses.
So, technological innovation is enabling the creation and storage of exponentially growing mountains of Big Data. But this wouldn’t be happening if there weren’t market forces driving it. That is, demand for Big Data led to the supply of it!
On the demand and supply side of things, we’re dealing with at least two different phenomena. And this is where I think the real Big Data frenzy is stemming from. The first is a case of being careful about what you wish for (because you just might eventually get it!); and the second is gross deficiency in the supply of analytical talent in the US.
In terms of being careful about what you wish for, my sense is a lot of the Big Data issues many are living today probably involves a narrative similar to this:
1) For years, the lack of data (or the costs associated with obtaining data) has been used as an excuse for not knowing (or being able to answer) something – if we only had more data
2) Many, for whatever reason, erroneously believed that simply having more data would mean better and more valuable (insightful) data - these are probably the same people who believe you need to remove all of their blood in order to have a reliable blood test
3) Over the past 15-20 years it has become amazingly cheap and easy to create and house A LOT of data. As a result, there are now massive mountains of Big Data “out there” all over the place - making the people using the old “if we only had more data” excuse pretty nervous
4) Many (bosses, shareholders, government leaders, etc.) naïvely believe since Big Data is now relatively easy to capture and house, it also should be relatively easy and inexpensive to analyze - these are people who think Microsoft Excel is all anyone could ever possibly need to analyze anything – they also happen to be the same people who slept through their business calc and business stats courses and don’t know the difference between a T-Test and a T-Square (hint – one is used in carpentry)
5) Very few people (in decision making capacities) have actually spent much any time thinking about the types of questions they want to be able to answer with Big Data – let alone how someone would actually go about answering them
6) Even fewer people have spent much time thinking about how all of this Big Data should actually be configured. That is, how it should be structured in order for it to be analyzed; thus enabling it to help answer all of these yet-to-be-defined business questions
7) Many leaders are now nervously sitting on tons of Big Data and have come to the realization that they don’t have the right tools and/or the right talent within their organization to leverage their unwieldy Big Data asset
8) Meanwhile senior leadership, boards and shareholders continue to wonder when all of the Big Data magic is going to begin! – I mean come on all ready would you! You’ve got all of this data that you’d been asking for; so do something with it already - and make us tons of money! NOW!
This leads me to the second, and more problematic demand and supply issue surrounding Big Data - There simply aren’t enough well-trained Big Data analysts in the US labor market do anything of any value with all of this Big Data!
These are the types of degrees Big Data analysts will have; and unfortunately for organizations needing to hire Big Data analysts, this is down significantly from nearly 13.5% of all degrees awarded in the 1980-1981 academic year. If you are on the demand side of this Big Data equation, this is not the sort of trend you want to see in the face of the surging Big Data Tidal Wave! (Affiliate Link)
While I’m spreading all sorts of sunshine on our Big Data parade, here’s something else to keep in mind. Only a small fraction of those who have graduated with analytic or technical degrees in the past twenty or so years are actually in the Big Data analysis business, and very few of today’s technically-oriented undergrads are aspiring Big Data ninjas. The bottom line is an interesting reality where we are dealing with the rapid growth in demand for competent Big Data analysts in the face of a woefully insufficient supply. I suspect it will take a good number of years before natural market forces arrive at equilibrium – that is, when the supply of Big Data slayers will equal market demand.
There is absolutely no doubt that Big Data is finally here, and that it is truly here to stay. There is also no denying that there are a lot of Big Data challenges that need to be better understood and dealt with. However, if we make the proper investments in Planning, Preparing and Organizing for Big Data, we can begin to realize the value that is promises. I, by no means intend to trivialize or undersell the time, effort, and resources that will be required along the way. This will be a big effort – after all, we are talking about Big Data. However this dilemma is a bit like the question about when is the best time to plant a tree. The best answer of course is yesterday and the second best answer is today. Regardless of your current Big Data state, better Planning, Preparation and Organization today will ensure a better Big Data tomorrow.
So what do you think? Are we really in the midst of a Big Data dilemma? Or, is all of this Big Data stuff much ado about nothing?
And hey - on the bright side of things, if you happen to be a budding Big Data ninja… Your future’s so bright, you’d better get shades (cue up Timbuk3). - Woody Bendle
