How to Build an ETL App for Neo4J Data in Python with CData

The rich ecosystem of Python modules lets you get to work quickly and integrate your systems more effectively. With the CData Python Connector for Neo4J and the petl framework, you can build Neo4J-connected applications and pipelines for extracting, transforming, and loading Neo4J data. This article shows how to connect to Neo4J with the CData Python Connector and use petl and pandas to extract, transform, and load Neo4J data.

With built-in, optimized data processing, the CData Python Connector offers unmatched performance for interacting with live Neo4J data in Python. When you issue complex SQL queries from Neo4J, the driver pushes supported SQL operations, like filters and aggregations, directly to Neo4J and utilizes the embedded SQL engine to process unsupported operations client-side (often SQL functions and JOIN operations).

Connecting to Neo4J Data

Connecting to Neo4J data looks just like connecting to any relational data source. Create a connection string using the required connection properties. For this article, you will pass the connection string as a parameter to the create_engine function.

To connect to Neo4j, set the following connection properties:

• Server: The server hosting the Neo4j instance.
• Port: The port on which the Neo4j service is running. The provider connects to port 7474 by default.
• User: The username of the user using the Neo4j instance.
• Password: The password of the user using the Neo4j instance.

After installing the CData Neo4J Connector, follow the procedure below to install the other required modules and start accessing Neo4J through Python objects.

Install Required Modules

Use the pip utility to install the required modules and frameworks:

pip install petl
pip install pandas

Build an ETL App for Neo4J Data in Python

Once the required modules and frameworks are installed, we are ready to build our ETL app. Code snippets follow, but the full source code is available at the end of the article.

First, be sure to import the modules (including the CData Connector) with the following:

import petl as etl
import pandas as pd
import cdata.neo4j as mod

You can now connect with a connection string. Use the connect function for the CData Neo4J Connector to create a connection for working with Neo4J data.

cnxn = mod.connect("Server=localhost;Port=7474;User=my_user;Password=my_password;")

Create a SQL Statement to Query Neo4J

Use SQL to create a statement for querying Neo4J. In this article, we read data from the ProductCategory entity.

sql = "SELECT CategoryId, CategoryName FROM ProductCategory WHERE CategoryOwner = 'CData Software'"

Extract, Transform, and Load the Neo4J Data

With the query results stored in a DataFrame, we can use petl to extract, transform, and load the Neo4J data. In this example, we extract Neo4J data, sort the data by the CategoryName column, and load the data into a CSV file.

Loading Neo4J Data into a CSV File

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'CategoryName')

etl.tocsv(table2,'productcategory_data.csv')

With the CData Python Connector for Neo4J, you can work with Neo4J data just like you would with any database, including direct access to data in ETL packages like petl.

Free Trial & More Information

Download a free, 30-day trial of the CData Python Connector for Neo4J to start building Python apps and scripts with connectivity to Neo4J data. Reach out to our Support Team if you have any questions.

Full Source Code

import petl as etl
import pandas as pd
import cdata.neo4j as mod

cnxn = mod.connect("Server=localhost;Port=7474;User=my_user;Password=my_password;")

sql = "SELECT CategoryId, CategoryName FROM ProductCategory WHERE CategoryOwner = 'CData Software'"

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'CategoryName')

etl.tocsv(table2,'productcategory_data.csv')