How to load Zuora data into Elasticsearch via Logstash

Elasticsearch is a popular distributed full-text search engine. By centrally storing data, you can perform ultra-fast searches, fine-tuning relevance, and powerful analytics with ease. Elasticsearch has a pipeline tool for loading data called "Logstash". You can use CData JDBC Drivers to easily import data from any data source into Elasticsearch for search and analysis.

This article explains how to use the CData JDBC Driver for Zuora to load data from Zuora into Elasticsearch via Logstash.

Using CData JDBC Driver for Zuora with Elasticsearch Logstash

• Install the CData JDBC Driver for Zuora on the machine where Logstash is running.
• The JDBC Driver will be installed at the following path (the year part, e.g. 20XX, will vary depending on the product version you are using). You will use this path later. Place this .jar file (and the .lic file if it's a licensed version) in Logstash.
  C:\Program Files\CData\CData JDBC Driver for Zuora 20XX\lib\cdata.jdbc.zuora.jar
  
• Next, install the JDBC Input Plugin, which connects Logstash to the CData JDBC driver. The JDBC Plugin comes by default with the latest version of Logstash, but depending on the version, you may need to add it.
  https://www.elastic.co/guide/en/logstash/5.4/plugins-inputs-jdbc.html
• Move the CData JDBC Driver’s .jar file and .lic file to Logstash's "/logstash-core/lib/jars/".

Sending Zuora data to Elasticsearch with Logstash

Now, let's create a configuration file for Logstash to transfer Zuora data to Elasticsearch.

• Write the process to retrieve Zuora data in the logstash.conf file, which defines data processing in Logstash. The input will be JDBC, and the output will be Elasticsearch. The data loading job is set to run at 30-second intervals.
• Set the CData JDBC Driver's .jar file as the JDBC driver library, configure the class name, and set the connection properties to Zuora in the form of a JDBC URL. The JDBC URL allows detailed configuration, so please refer to the product documentation for more specifics.

Zuora uses the OAuth standard to authenticate users. See the online Help documentation for a full OAuth authentication guide.

Configuring Tenant property

In order to create a valid connection with the provider you need to choose one of the Tenant values (USProduction by default) which matches your account configuration. The following is a list with the available options:

• USProduction: Requests sent to https://rest.zuora.com.
• USAPISandbox: Requests sent to https://rest.apisandbox.zuora.com"
• USPerformanceTest: Requests sent to https://rest.pt1.zuora.com"
• EUProduction: Requests sent to https://rest.eu.zuora.com"
• EUSandbox: Requests sent to https://rest.sandbox.eu.zuora.com"

Selecting a Zuora Service

Two Zuora services are available: Data Query and AQuA API. By default ZuoraService is set to AQuADataExport.

DataQuery

The Data Query feature enables you to export data from your Zuora tenant by performing asynchronous, read-only SQL queries. We recommend to use this service for quick lightweight SQL queries.

Limitations
• The maximum number of input records per table after filters have been applied: 1,000,000
• The maximum number of output records: 100,000
• The maximum number of simultaneous queries submitted for execution per tenant: 5
• The maximum number of queued queries submitted for execution after reaching the limitation of simultaneous queries per tenant: 10
• The maximum processing time for each query in hours: 1
• The maximum size of memory allocated to each query in GB: 2
• The maximum number of indices when using Index Join, in other words, the maximum number of records being returned by the left table based on the unique value used in the WHERE clause when using Index Join: 20,000

AQuADataExport

AQuA API export is designed to export all the records for all the objects ( tables ). AQuA query jobs have the following limitations:

Limitations
• If a query in an AQuA job is executed longer than 8 hours, this job will be killed automatically.
• The killed AQuA job can be retried three times before returned as failed.

Executing data movement with Logstash

Now let's run Logstash using the created "logstash.conf" file.

A log indicating success will appear. This means the Zuora data has been loaded into Elasticsearch.

For example, let's view the data transferred to Elasticsearch in Kibana.

    GET zuora_table/_search
    {
        "query": {
            "match_all": {}
        }
    }

We have confirmed that the data is stored in Elasticsearch.

By using the CData JDBC Driver for Zuora with Logstash, it functions as a Zuora connector, making it easy to load data into Elasticsearch. Please try the 30-day free trial.