Data Exploration: Definition, Importance & How It Differs from Data Analytics

It’s a common plot device: Our hero confronts a huge pile of raw data that must be analyzed quickly to clear an innocent person from being convicted of a crime. The information is a collection of interview transcripts, drawings, photographs, video clips, audio clips, hand-written notes, and printed matter. There’s an answer in there somewhere, but nobody else has ever been able to find it.

Fortunately, our hero is secretly a metahuman with super speed. She analyzes and ingests each type of data in turn, reconciles all contradictions, and emerges with evidence that exonerates the poor guy. Justice is served!

Unfortunately, in real life, ingesting all the information doesn’t always help you see the data landscape as a whole. Sometimes there are clues hidden in the raw data that are obscured once the data’s processed. And sometimes, when data is ordered and organized, you lose contact with corner cases and outliers that can provide special insight.

Is insight about data always deferred until we can port it into a data warehouse for analysis? Frequently, yes. But it doesn’t have to be.

What is data exploration?

Data exploration is a review of raw data to observe data set characteristics and patterns, and hopefully identify relationships between different variables. It helps to expose:

• The structure of the dataset
• The presence of outliers
• The distribution of data values

These characteristics reveal patterns in the data and identify points of interest that enable data analysts to gain insight into the data before it is ported into a data warehouse.

The objectives of data exploration are to foster a broader understanding of the data’s usefulness to the business, and to speed up time to answers. Users can examine the data visually and drill down on it while it is still raw, determining which data is the most important and which may distort the analysis (and should therefore be removed) before it’s ever ingested into a database, data warehouse, or data lake.

In this way, data exploration helps decrease the time spent on less valuable analysis by selecting the right path forward from the very beginning.

How does it work?

Data is examined using a combination of manual workflows and automated exploration techniques that display data in visually distinct ways, enabling data analysts to better infer meaning from it.

It’s a three-step process.

1. Data overview: To gain a deep understanding of the data (so that it can be handled efficiently in future machine learning tasks), we begin to identify preliminary statistics. We characterize the data with respect to the number of observations, number and types of features (numeric and categorical), overall missing rate, and percentage of duplicate observations.
2. Feature assessment and visualization: We focus on both the individual properties of the data (univariate analysis), and their interactions and relationships (multivariate analysis). This helps identify the behavior(s) of interest and assess them concerning their strength.
3. Assess data quality: We identify possible complicating factors that might affect the quality of this dataset. This includes missing data, imbalance data, constant values, duplicates, highly correlated features, redundant features, and “noisy” data.

Why data exploration is important

Because data exploration is inherently visual, it helps data analysts and others literally see patterns in the data that they might not be able to discern if the same data was moved into a giant spreadsheet.

Data exploration and visual analytics tools encourage users to explore data in any visualization. This democratizes access to data and provides governed self-service analytics. Users can generate thoughtful questions that might spur deeper analysis.

Data exploration is useful in a wide range of industries including engineering, education, and insurance, but it’s especially useful in:

• Software development – including Geographic Information Systems (GIS), machine learning, and the exploration of big data.
• Healthcare and medicine – including epidemiology, drug research, treatment methodologies, and fraud detection.
• Finance – including sales, offer acceptance, fraud detection.

Is there a difference between data analytics and data exploration?

Data exploration is sometimes called exploratory data analysis and defined as “a statistical technique used to analyze raw data sets to observe their broad characteristics”. This makes it especially easy to conflate data exploration with data analytics. But they’re not the same.

Data analytics is the broad field of using data and tools to make business decisions, executed by data experts (data scientists, engineers, and analysts), to help the rest of the business access and understand their findings. It covers the full life cycle of the data.

Data exploration is strictly performed on raw data. Once the data has been explored, it may wind up in a database, data lake, or data warehouse, but data exploration is longer happening.

Data exploration techniques

This section describes some common techniques for data exploration.

Visual data exploration

Visual data exploration investigates the data set's characteristics by creating many distribution plots of the data, interactively. Plots can include:

• Measures of location
• Measures of spread
• Asymmetry
• Outliers
• Gaps

Data can be viewed in something as exotic as a topological plot, or in something as simple as a histogram (x-y axis grid). These can quickly show which values are the most frequent, and whether values cluster in a particular location in the dataset.

Data notebook exploration

Data notebooks that run on a Python computational engine are an easy way for a data analyst with some development skills to load and interact with raw data. Data can be loaded as a .csv file, and then instantly “printed” (displayed) within the notebook. Data scientists can gather basic information, then calculate descriptive statistics such as central tendency, data dispersion, percentile, and NaN (not-a-number) values to preview the data and possibly cull information that is less useful.

The most commonly used data notebook is the Jupyter computational notebook, running a feature-rich library of functions such as Pandas.

Central tendency measures

Measures of central tendency are single values that represent the center point of a dataset. The most common measures of central tendency, familiar to most high school statistics students, are the mean, the median, and the mode.

Central tendency measures are good for determining both typical information and outlier information.

Variance analysis

An analysis of variance (ANOVA) is used to determine whether there is a statistically significant difference between the means of three or more independent groups. This helps data analysts determine whether the data for those three groups are found together, or if they are interrelated.

Data exploration tools

This section described some of the most frequently installed tools for exploring raw data through a simple visual interface.

• Microsoft Excel: Excel spreadsheets support the creation of basic charts for data exploration, to view raw data, and to identify the correlation between variables. You can use the Excel CORREL() function to compare two continuous variables or identify the correlation between two categorical variables using a two-way table, a stacked column chart, or a chi-square test.
• Google Looker: Google Looker is a product that helps data scientists explore, share, and visualize raw data. It includes LookML, a language for querying SQL data; the ability to create dashboards and reports. Users can also interactively explore data from Look ML models through a connected Google Sheets spreadsheet.
• Mode Analytics: Mode analytics supports ad hoc data analysis, self-service reporting, data applications, and advanced analytics with SQL, R, Python, and data viz. It includes a Visual Explorer that runs on Mode’s underlying data viz engine.
• Power BI: Microsoft Power BI makes it easy for anyone – not just a data scientist – to query your data using natural language. Users can drill down on a visual representation by asking a question, such as “What are the total units by region?” There are literally scores of ways to explore data, share insights, and interact with the visuals.
• Python Libraries: Libraries of functions, written in Python to be used in a computational notebook, have been used in academia for more than a decade. The most popular function library is Pandas, which is open source, and growing through a large contributing community of data scientists. (For more details, see “Data notebook exploration”, above.)
• Qlik: Qlik is SaaS software with very flexible deployment options. It features DataOps automation to protect raw data and keep it trusted and actionable. The available analytics tools work well for data scientists, AI/machine learning, and applications.
• Tableau: Tableau is open-source software that plays well with function libraries such as matplotlib and Seaborn. It is dashboard-driven, and capable of univariate analysis, histogram and box plots, bi-variate analysis, scatter plots, and more.

Self-service data exploration with CData Connect Cloud

CData Connect Cloud makes it easy to explore and manipulate raw data collected directly from your favorite data exploration tools and platforms. The built-in data explorer gives you access to a live data model of SaaS, big data, and NoSQL sources where you can create customized queries and views into your data for access across the organization.

These features let you generate custom service-specific support for several popular data exploration tools. When paired with purpose-built native clients for Power BI, Tableau, Microsoft Excel, Google Sheets, Looker, Python, and more, you can access live data and interact with it from anywhere in your enterprise. No transformation required!