IT vendors these days are making a lot of noise about “Big Data”. That comes as no surprise, since Big Data allows selling masses of expensive hardware, software, and services. But does it really make that much sense for the customer?
The sales pitch for Big Data is that companies can better do business based on that approach. They can do better marketing based on analyzing more data about their customers. They might provide better security services on analyzing more data. They might need it to deal with machine-generated data in the connected vehicle.
However: better marketing is not about still failing to understand the intention of the customer based on more data (I just recently blogged about that). For marketing it is about changing the fundamental principles and accepting that there is no way of reliably predicting the current intention of a customer based on historical data.
For security, analyzing more data and understanding the patterns which might become visible when you look at data from your endpoint security solutions, your firewalls, and some log files (and not only one portion of that data) is a clear argument. However, even here I don’t think that we necessarily have to put all that data into one place.
And if we look at machine-generated data like the connected car, there is no single application which needs to have access to all that data. There is a lot of data. There are many applications which might require some of that data. But there is no need for the single Big Data store. The connected vehicle (and some other elements of the upcoming “smart world”) are topics I’ll touch more in depth in upcoming reports and blog posts.
I fully agree that the volume of data is increasing. But I doubt that we really need to deal with an exponential growth beyond what is covered by the growth of computing capacity described in Moore’s law.
There’s also this: building bigger data stores inevitably leads to compliance issues. One of the favored targets of today’s auditors is SAP Business Warehouse. A lot of data ends up in there and it is hard to understand and control with which other data it is combined and where it ultimately goes. So betting on Big Data may also mean provoking new audit findings.
My perspective is that we should focus much more on getting smarter in dealing with data. If we know what we want to do with data than we can request the data we need from the existing data sources. We can even request pre-processed data to become smarter. A foundation for that is what my colleague Craig Burton recently described in his report on The Open API Economy.
The Open API Economy is about providing simple (or smart) APIs, like RESTful APIs. They are simple to use. If we add such APIs to our existing data we can consume different APIs and then extract the data we really need for analytics. This means that we don’t need to build the big, fat data stores. We extract what we need. The API might provide pre-processed data. We could also import data if we need it more frequently.
Yes, there are aspects which we have to keep in mind, like latency issues and the amount of data which has to be moved across the wire. But overall there are many situations where being smart will be a better approach than being big.
I don’t say that there is not a need for some bigger data in some cases. But I doubt that there is any value by itself in Big Data. It is just a marketing buzzword. And it’s time to start thinking about the real value of Big Data business cases and smarter approaches. Smart data and picking up concepts like The Open API Economy can help saving money and getting more results for less investment.
I would also say that programmatic access to data also allows the ability for an entity to consider "contextual" or "salient" data at the time of an event. This design also allows a given entity to ignore any incoming data that isn't salient and would therefore preempt the need for "Big Data" stores. On the other hand, there may be instances where an entity can ignore the event at a given time, but needs to be able to go back in time if needed to see what happened.
All this lead to a common theme throughout our KuppingerCole philosophy, "freedom of choice". If the Big Data approach works, then by all means, use it. If you don't need it, ignore it. It's not about whether Big Data is wrong. It is about the enterprise or individual having a choice and doing what is best for him. Be smart and think about Smart Data!