Efficient Data Processing and Analytics with Apache Hive

Size matters. When your enterprise employs data sets that are too large or too complex to be dealt with by traditional data processing software, you’ve got big data. And though analysis of big data can create great insights and benefits, finding tools that can deal with such dense and complex data can be tricky.

How do I know if I’ve got big data?

Look at the following characteristics:

• Volume: The quantity of generated and stored data. Big data is usually larger than terabytes and petabytes.
• Variety: The type and nature of the data. Big data employs information collected via social media, log files, sensors, drawing from text, images, audio, video. It’s more complex and generated much more quickly than the structured data of the 90’s.
• Velocity: The speed and frequency at which the data is generated. Big data is often available in real time.
• Veracity: The truthfulness or reliability of the data. (The data quality of captured data impacts the ability to analyze it accurately.)
• Variability: Big data analysis can integrate raw data from multiple sources. This may create the need to transform unstructured data to structured data.

When you’ve got big data, and your enterprise needs to simplify large-scale data management, unlock efficient data processing, and supply reliable analytics (with fewer false discovery rates), you should consider Apache Hive.

What is Apache Hive?

Apache Hive is a fault-tolerant, distributed data warehouse system that enables data analytics at a massive scale. It enables data scientists and system administrators to read, write, and manage petabytes of data residing in distributed storage using its own version of SQL, Hive Query Language SQL.

Hive is built on top of Apache Hadoop. It supports storage on Amazon S3, Azure Data Lake Storage (ADLS), GoodSync (GS), and others through the Hive Distributed File System (HDFS).

This means that Apache Hive plays nicely with most contemporary data storage systems, and is easy to learn.

Apache Hive’s core features include:

• Hive-Server 2 (HS2)
• Hive Query Language
• Hive Metastore Server (HMS)
• Hive ACID
• Hive Replication

It’s tailor-made to address some of the thorniest requirements of big data warehousing and analytics, including the ability to:

• Properly index or identify all the data
• Work with data that includes common fields (which makes it easier to conjoin different data sets for meta-analysis)
• Work with data with extensional fields
• Expand the system rapidly

How Apache Hive works

Hive mainly consists of three core parts:

• Hive clients: Comprised of an application and a driver, which communicate with the Hive Server. Clients propel queries to Hive Services, which process the queries and then write them to a metastore, a file system, or an execution engine.
• Hive services: Comprised of the Hive Server (which interacts with client queries), the Hive web interface, and CLI. Hive services act on incoming data and propel it to the proper storage and compute facility.
• Hive storage and computing: Comprised of file system, job client, and meta store services that communicate with Hive storage and store metadata table information and query results.

What Hive Metastore does

When new data is saved to object storage, it is registered into the Hive Metastore via the metastore API. This “registration” maps a set of objects in the object store to a table exposed by Hive. Part of registration includes specifying the schema of the table held in the file, with some metadata describing the columns.

Using Hive Metastore in this way provides benefits related to virtualization, discoverability, schema evolution, and performance.

Virtualization

Data analysts using SQL usually aren’t interested in the details of object storage and its access patterns. They just want the analytical workflows to work, thank you! Hive’s analytical workflows, dependent on these defined objects, have remained stable for over a decade, while other Hadoop components have evolved. In fact, every new technology introduced since HMS was created has been careful not to break HMS.

Discoverability

Hive Metastore naturally becomes a catalog of all the collections held in object storage when exposing new data is accompanied by updating it. If well-maintained, this allows for the discovery of data sets available to query.

Supplemental information, such as data update frequency and ownership, can also be saved in the metastore.

Schema evolution

One of the challenges of managing datasets over time is their mutability. Records may change over time with respect to the existing columns describing their attributes. The set attributes themselves may also change over time, resulting in a change to the schema of the table.

The registration process described above provides a record of the schema for each additional data file that belongs to the table. This means that if the schema ever changes, it will be recorded within the Hive Metastore. The data can be accessed with the appropriate schema.

It also provides a good basis to validate a schema if it should not have changed, and send an alert. Hive holds the information to create such a test.

Performance

Since Hive Metastore maps the table to the underlying object, it allows the representation of partitions according to the primary key supported by the object storage. The user can set the granularity of the partitions. If the partitions are balanced and their number is reasonable, this mapping improves query performance.

This is often referred to as “partition pruning”, which allows a query engine to identify data files that can be skipped.

How Hive differs from traditional relational databases

The following table provides a quick comparison between traditional RDBMS and Hive.

Main Apache Hive components

Hive-Server 2 (HS2)

A service that facilitates the execution of queries against Hive over the web, via a JDBC driver such as the CData Apache Hive JDBC Driver.

Because it is based on Apache Hadoop, it leverages Hadoop’s massive scale-out and fault-tolerance capabilities for data storage and processing on commercial servers.

Hive is designed to enable easy data summarization, ad-hoc querying and analysis of large volumes of data. It provides HiveQL, a SQL-like query language that enables users to do ad-hoc querying, summarization, and data analysis easily. It also makes it easy for developers to integrate their own functionality to perform custom analysis via user-defined functions.

Hive Query Language (HiveQL)

A query language in Apache Hive for processing and analyzing structured data. It reuses common concepts from relational databases, such as tables, rows, columns, and schema, to make it easy for people who are familiar with RDBMS and SQL to learn. There is also a Command Line Interface (CLI) for writing Hive queries using HiveQL.

HiveQL supports the TEXTFILE SEQUENCEFILE, ORC (Optimized Row Columnar), and RCFILE (Record Columnar File) formats. Metadata is stored in the Hive Metastore Server (HMS) via either a built-in single-user Derby database or a shared MYSQL database.

Hive Metastore Server (HMS)

The central repository of metadata for Hive, organized as tables and partitions in a relational database. It provides clients (like Hive, Impala and Spark) access to this information via the metastore service API.

HMS has become a key component for data lakes that pull from the diverse world of open source software. The original stack that comprised HMS in the early 2010s, was revolutionary at that time, but even as newer technologies have dismantled that stack, most organizations featuring data lakes still include a Hive Metastore.

For a description of how HMS works, see “How Hive Works”, above.

Hive ACID

Apache Hive provides transaction semantics to support the...

• Atomicity: An operation either succeeds, or it fails
• Consistency: Once an application performs an operation, the results are visible to it in every subsequent operation
• Isolation: An incomplete operation by one user does not cause unexpected side effects for other users
• Durability: Once an operation is complete it is preserved even if the machine or system fails

...of its database transactions.

ACID contributes to the stability of the streaming ingestion of data, slow-changing dimensions, data restatement, and bulk updates using the SQL MERGE statement.

Hive Replication

Hive Replication supports both bootstrap and incremental replication of clusters for backup and recovery. It provides a framework for replicating Hive metadata and data changes between clusters without the need for the replica to run in the same Hadoop distribution, Hive version, or metastore RDBMS. The source cluster and the replica are very loosely coupled. The HMS Thrift service acts as an integration point between the two.

Benefits of Apache Hive

Some high-level benefits of Apache Hive include:

• Fast processing of petabytes of data via batch processing
• Familiar SQL-like interface (Hive CLI, HiveQL) that is accessible to both developers and non-developers
• Easy scalability
• Improved performance, as a distributed data warehouse system that provides SQL-like querying capability, over NoSQL distributed databases
• Provides defined schemas for all tables
• Supports both structured and unstructured data, including native support for common SQL data types such as INT, FLOAT, and VARCHAR

CData Hive drivers & connectors for data integration

CData’s Apache Hive drivers & connectors enable you to connect to Apache Hive-compatible distributions from BI, analytics, reporting, ETL tools, and custom solutions. Whether you need to build reports and visualizations in Power BI, analyze your Hive data in Microsoft Excel or any other data task, CData's drivers and connectors let you work with Apache Hive data exactly where you want.