Big Data Series 3 - Do I need it?

---

One of the occupational hazards of being a management consultant is that sometimes you have to be the harbinger of bad news. It’s depressing, to say the least. But, you have no other choice. It’s always better to be direct and not dance around the issue. It’s all part of the package, you see. If a consultant tells you that he or she has never encountered a situation like that, don’t believe ‘em.

Let me explain with an example. A medium-sized firm came to us and asked us to help them develop a “Big Data” strategy to transform their business. After careful analysis of the case, I made the call that there was absolutely no need for them to implement a “Big Data Solution” as there was no real “problem”. Obviously, the CEO did not like my answer and dismissed our findings and as is normally the case, gave me a lecture about how consultants are assholes and that we are nothing more than glorified promoters of Microsoft Powerpoint. You sort of get thick skin after a year or two in consulting business, so I am used to occasionally hearing such things. Of course, we cannot be right all the time and when we err it is important to own up to it. Unfortunately, a few of us don’t like to own up to our fuck-ups and resort to pointing fingers at others. This is bad and makes us all look evil. That’s why people hate consultants, in my opinion. Anyway, the news I gave him was not necessarily bad, but his response was what made it look worse. They threw my report out the window and decided to go ahead with their plan. It turned out to be a very costly mistake and the CEO got booted out within just 7 months since this incident.

The problem in the example is scale, you see. It is the primary reason why many businesses have struggled transforming their early analytics enthusiasm into something truly of value. Say A does it successfully. There is B who wants to replicate what A does. But B cannot compare with A’s scale, so why imitate A’s data strategy? The firm that came to us was B. This is not When Harry met Sally, where you wanna have what Sally’s having. Building a “Big Data” infrastructure is expensive and requires an across the board transformation. If your company generates such data, then it’s absolutely worth it, say about 80%. But if you don’t, it’s a monumental waste of money and resources.

Do I need “Big Data”?

Big Data, Analytics, Data Science, Machine Learning, Artificial intelligence - These are the most common themes discussed amongst businesses today. Everyody is hopping on to the Big Data bandwagon without even understanding what it is. In my earlier blog post, I mentioned that the key thing to remember is that this is high velocity data and that’s what makes it “Big”. Many organizations have started to assume that big data is indispensable as the holy grail when in fact, ths so-called “small” operational data is all that they need. Analytical treatment of just this internal data can give you excellent pointers to improve your business model. Big Data is only sporadically collected internally (Manufacturing is a different thing. More on that later). Buying it is a more common strategy these days.

The first thing one needs to do is to figure out whether big data is necessary or not. The problem is compounded when you don’t produce the data yourself. Setting aside the notorious difficulty in wrangling with the data, thanks to its size and complexity, preparing the data for report generation and analysis can set you back by hundreds of thousands of dollars, or in some cases, by the millions. Furthermore, no amount of fancy analytics tools can deliver you the golden nugget if you don’t have a clue about extracting information that will lead to it. In my decades long experience, I can tell you this - retrieving actionable information with data mining requires accurate analysis, receptive/attentive management and sustained adjustment. Buying SaaS and other fancy tools alone will not cover for the shortage of skills necessary to master these processes overnight. Don’t even get me started on those useless 100 dollar week long courses in Data Science, ML and AI that purports to make you the next fucking Pearson! I am dedicating an entire blogpost on this phenomenon for later.

2018 was the year of big data mishaps and this has resulted in scrutinizing how organizations gather and use data. Remember Facebook and Cambridge Analytica? Despite this criticism, big data does still offer invaluable insights for some situations and challenges — what you need to figure out is whether yours are among them. If you’re thinking about investing in big data for your organization, take the following considerations into account to ensure you’re truly working toward the goal you seek.

Needing a lot of data doesn’t always mean you need big data

Before jumping in, make sure the problem you’re trying to solve or the goal you’re hoping to achieve actually requires big data rather than just a lot of data. Although the term seems to emphasize volume over anything else, “big data” actually just describes a quantity of data that requires new tools to process it. Typically, big data utilizes multiple physical or virtual machines working together. If you’re merely storing and retrieving large volumes of files in and from a data warehouse, you’re facing a different kind of challenge. Huge data sets are an issue that many organizations have been dealing with for years, and the quantity of data by no means indicates a “big data” problem.

Even with big data, operational data remains critical

It’s a common misconception that organizations must choose between big data and small operational data. In fact, a complete big data solution could depend on combining them. Big data is most commonly used retrospectively, and analytical big data technologies such as Hadoop can generate valuable insights after data has been collected. However, operational big data systems are still responsible for importing and storing data via real-time workloads. Incorporating both types of data will ensure your data efforts produce the most effective results.

The payout from big data requires big changes

The hype surrounding big data has inflated expectations, in many cases well beyond what’s reasonable. Gaining a competitive advantage from big data can also require enormous changes that are impossible or impractical for many organizations to make. For instance, big data helped a retailer see that by keeping items on the showroom floor for a longer period, both before and after discounting them, it could increase its profits significantly. Unfortunately, this change had far-reaching supply chain implications, and the company was unable to put it into practice.

The insights generated through big data analytics can be easy to replicate, so it’s possible that consultants in your industry might already provide the services you’re looking to glean from big data. Be sure to do your homework before you spend the money on a big data initiative. Although big data is everywhere you look, it may not actually be the right solution for your organization. Big data can be insightful, but these insights are distilled after the data has been collected and analyzed. Ultimately, before you go chasing big data, you might want to focus on better using the operational data you already have. Even if you end up needing big data after all, you’ll be better prepared for it after you get a handle on your in-house data.