Big Data After the Hype

Total Data Management

This year, I’ve been following the meteoric rise of big data. It has been a boon for vendors who are venturing into this area.  It has produced countless start-ups and much buzz in the data management world.

However, when it comes down to it, what we’re really talking about here is data management and data governance.  Whether you have to deal with big data, enterprise data or spread-marts, data needs to be managed no matter what size. The tides are turning for a total data management approach. Recent surveys shows that despite the market hype, most technologists and business users feel that big data is an off-shoot of data management, not a branch of technology in itself. 

So, why the hype?  I'm convinced it is mostly vendor-generated. In 2010, when big data began to gain notoriety, there was a disconnect for some vendors.  While partnered with traditional enterprise data management companies like the Oracle and IBM’s of this world, not all vendors were prepared for the growing popularity of open source and Hadoop. Others were (and still are) better positioned. They began talking about big data as a product differentiator. Vendors who don’t have the basic architecture for managing data in Hadoop have been and will continue to struggle. 

For example, ETL tools that have a basic connection to move data in and out of Hbase, Hortonworks and Cloudera can’t stop there.  The power of Hadoop must be harnessed, and it’s not always an easy thing to do when your technology requires executables tied to CPUs.  One of the powerful things about Hadoop is that it scales based on a languages like PIG, Sqoop and Java without having to install anything.  Want to expand the number of servers?  Add a datanode server, tell the name namenode and rebalance - and your off and running.  However, even this simple innovation is more difficult on some vendors’ architectures than others.

Another rethinking that is taking place in the market is long-standing CPU-based pricing structure.  Vendors who they keep their pricing structure based on core processors for Hadoop will continue to struggle because it runs counter to the power of Hadoop. You hear about the volume, velocity and variety.  Technically, if you want to step up the volume with another datanode, it’s no big deal. However, it becomes a big deal if you have to renegotiate a vendor contract each time.

Last year, around this time, I did write about the various costs associated with the scale of data. In summary, the costs of licenses and connectors are the bigger for enterprise data, while the costs associated with skills are more likely to affect you with big data.  There will come a time where the skills gap will be closed, however.

In the year 2013, we’ll begin to see the un-hyping of big data in favor of this total data management approach. For buyers, big data will be a tick-box in their RFP’s in the effort to manage data, no matter what the size.