The Format War for Hadoop Structured Data

A war is raging that pits Hadoop distribution vendors against each other in determining exactly how to store structured big data. The battle is between the ORC file format, spearheaded by Hortonworks, and the Parquet file format, promoted by Cloudera.

ORC and Parquet are separate Apache projects with the similar goal of providing very fast analytics. To achieve performance, the formats have similar characteristics in that they both store data in columns rather than rows. This enables a majority of analytics to run faster than if the data was stored in rows or some semi-structured format. They also both support compression; when you store data in columns, it tends to compress very efficiently.  It’s easier to compress a column of dates, for example, than it is to compress mixed numbers, dates and strings. Compression saves you intensive disk access, a common bottleneck for analytics.

If you’re part of the HPE Vertica community, the goals of ORC and Parquet may sound familiar.  Columnar databases, including Vertica, have had columnar formats as part of the core product since the beginning. Before ORC and Parquet were in incubation, Vertica developed the ROS format for columnar, compressed big data storage.  Over the years, we have tuned and enhanced the format by adding a large number of compression algorithms designed to make the data storage and retrieval very efficient.  We’ve thought through features like backup and restore. After all, with a columnar store database, the concept of incremental backup/restore changes quite a bit.  We’ve had time to think through security, encryption and a long list of challenges when managing data in columnar format.

Orc vs Parquet – War, what is it good for?

Which format is better? Hortonworks has argued that ORC is ahead of Parquet in its capabilities to do predicate pushdown.  In layman terms, this claim is about performing analytics closer to where the data sits rather than spurring on excess network traffic. Cloudera has argued for Parquet in its efficient C++ code base.  It also argues that ORC data containers are primarily described with HIVE, while Parquet’s data containers can be described using HIVE, Thrift and AVRO.  The important thing to remember is that if you have chosen Hortonworks as your Hadoop distribution, it may be a little tricky to perform analytics on Parquet.  Accessing ORC files from Cloudera might also be a challenge.

At HPE, our goal is to seamlessly support ORC, Parquet and ROS as part of the Vertica analytics platform. Vertica has developed an ORC reader, in collaboration with Hortonworks, to be super-efficient at performing analytics on ORC files.  Just this week we also announced certification of Vertica on the CDH 5 platform and we have connectors into Parquet via our HDFS connector.  We’re also working with Cloudera to continuously optimize our Parquet file access. The goal is to read, write and federate multiple formats to minimize unnecessary data movement and transformations. For the information workers who need to run analytics, it shouldn’t matter where the data sits or in what format.

On the Horizon – Kudu

The aforementioned file formats are tied to analytical use cases.  In other words, if you have petabytes of data in your data lake and you need to crunch through it in short order, ORC, Parquet and ROS are valuable.  However, Cloudera recently announce a new data structure and project called Kudu (link) that also addresses the needs of an operational analytics use case – one where you need to small queries on the smaller data sets, particularly as they are ingested into the data lake. It’s still in incubation, but if the vision is realized, it will mean better efficiency and easier implementation for companies who need to do both analytical and operational systems.  We’ll explore this and its tie to Kafka and Spark in my next post.

What’s in store for Big Data Analytics in 2016

It’s the time of year again for predictions on all sorts of topics. Worthy, solid predictions are often based on the past and present trends and then projecting those trends into the coming year. Since I spend a lot of time studying trends of big data and analytics, I’m going to offer my predictions for the upcoming year.

Big Data will Triumph over Global Troubles
While there were awesome use cases, big data in 2015 was still somewhat a science experiment. This year there is hope for major breakthroughs in solving some of the world’s most challenging problems with big data.  Organizations are already doing amazing things, but we’re just scratching the surface of what we can accomplish with big data.  I’ve had several conversations with clients who are looking to map the human genome and tackle problems like cancer, Alzheimer’s disease and more by mapping the genes linked to them. I believe there are eminent breakthroughs here credited to our ability to handle huge data volume and perform faster and faster analytics improves.

But that’s not all.  People are using big data science for transportation research, making planes, trains and automobiles smarter and more efficient.  Non-profits are using big data to drive decisions about conservation and ecology with big data. We have a real opportunity this year to make the world a better place with big data.  Data is the new currency in scientific breakthroughs. The capability we now have to crunch through it with our algorithms is the disruptor.

Algorithms will be the New Edge
2016 is sure to be a year for using algorithms, specifically predictive analytics, to boost company revenue. Analysts like Gartner predict that differentiated algorithms alone will help corporations achieve a boost of 5% to 10% in revenue in the near future. Algorithms will make the best use of huge volume of customer-generated data that we get from our phones, devices and the internet of things to formulate more helpful, targeted offers for prospects and customers.  New, younger companies will leverage predictive analytics to disrupt their markets and potentially unseat the established leaders.  Predictive analytics can serve to update power delivery and consumption, medical research and treatment, and other lofty human problems, in addition to generating new revenue.

It’s difficult to see whether the algorithms themselves will be an emerging market, as some analysts say, or whether we will share most of our algorithms in our communities of data scientists. I think society will benefit more from an open source approach here, and the young minds who develop the algorithms will probably be more willing to take an open approach. Think about it, if you could predict Alzheimer’s disease with your algorithm, wouldn’t you want to share it with the world?

Hybrid Architectures will Rule in 2016
Companies are adopting a strategy where they use the right tool for the right job when it comes to big data analytics. This means that daily analytics and proprietary data is analyzed on-premise in ever growing data warehouse data volumes. Small, short-lived projects are often deployed on the cloud, and Hadoop is often used to keep costs low on data that is important, or data that needs to be farmed for mission-critical information. Finally, technologies like Spark are in their infancy to help with real-time, operational analytics.

It will be up to the vendors and open source community to provide some consistency across these different deployment strategies. Information workers really won’t care where it is running, just that they can use their favorite visualization tools, SQL, R and Python. Sometimes these workloads run in their own environment, but vendors can help reduce the work involved if, for example, you want to move your cloud project to on-premise. By offering a consistent SQL, for example, across these deployment architectures, you can avoid the headaches of a hybrid environment.

Open Source will Attain New Maturity
I’ve written many times about the hype around Hadoop and the maturity of the Hadoop platform by comparison to commercially available software. Let’s face it, many open source solutions for big data analytics were somewhat immature in 2015. As I mentioned in my last post, it’s a matter of taking software that is extremely useful and spending a few years to overcome shortcomings and build out a complete platform for big data analytics.  This year, the Hadoop community will build it out to be a more complete platform.  My prediction is that we’ll see greater maturity in 2016. With greater maturity will come wider adoption.

That said, I have observed that the open source community tends to focus on the start and not the finish. For example, over the past few years, SQL users have heard about many flavors of SQL on Hadoop.  Spark seems to be the latest and coolest new project offering SQL analytics on big data and it show great promise. However, the shift seems to be toward new projects and away from making the legacy projects work better.

Hewlett Packard Enterprise Role
I was inspired to write these predictions by a webinar that I attended in which some of the executives of Hewlett Packard Enterprise and influencers gave their vision of 2016.  For more information, watch the replay video here. Hewlett Packard Enterprise (HPE) has a role to play in making these predictions come true. HPE’s vision starts with the understanding that data fuels the new style of business driving the idea economy. Data will distinguish disruptors from the disrupted. Big data promises new customers, better experiences and new revenue streams. But all opportunities come with challenges. The recipe for success is continuously iterating on what questions to ask, which data to analyze and how to use the insights at all levels of your organization.