How to work with SAP Netweaver Gateway Data in Apache Spark using SQL

Apache Spark is a fast and general engine for large-scale data processing. When paired with the CData JDBC Driver for SAP Netweaver Gateway, Spark can work with live SAP Netweaver Gateway data. This article describes how to connect to and query SAP Netweaver Gateway data from a Spark shell.

The CData JDBC Driver offers unmatched performance for interacting with live SAP Netweaver Gateway data due to optimized data processing built into the driver. When you issue complex SQL queries to SAP Netweaver Gateway, the driver pushes supported SQL operations, like filters and aggregations, directly to SAP Netweaver Gateway and utilizes the embedded SQL engine to process unsupported operations (often SQL functions and JOIN operations) client-side. With built-in dynamic metadata querying, you can work with and analyze SAP Netweaver Gateway data using native data types.

Install the CData JDBC Driver for SAP Netweaver Gateway

Download the CData JDBC Driver for SAP Netweaver Gateway installer, unzip the package, and run the JAR file to install the driver.

Start a Spark Shell and Connect to SAP Netweaver Gateway Data

1. Open a terminal and start the Spark shell with the CData JDBC Driver for SAP Netweaver Gateway JAR file as the jars parameter: $ spark-shell --jars /CData/CData JDBC Driver for SAP Netweaver Gateway/lib/cdata.jdbc.sapgateway.jar
2. With the shell running, you can connect to SAP Netweaver Gateway with a JDBC URL and use the SQL Context load() function to read a table.

   SAP Gateway allows both basic and OAuth 2.0 authentication. You can use basic authentication to connect to your own account, or you can use OAuth to enable other users to retrieve data from your service with their accounts. In addition to authenticating, set the following connection properties to access SAP Gateway tables.

   • Url: Set this to the URL of your environment, or to the full URL of the service. For example, the full URL might appear as: https://sapes5.sapdevcenter.com/sap/opu/odata/IWBEP/GWSAMPLE_BASIC/. In this example, the environment URL would just be: https://sapes5.sapdevcenter.com.
   • Namespace: Set the appropriate Service Namespace. In the example above, IWBEP is the namespace. It is optional if the full URL to the service is specified.
   • Service: Set this to the service you want to retrieve data from. In the example above, the service is GWSAMPLE_BASIC. It is not required if the full URL is specified.

   Authenticate via Basic Authentication

   In basic authentication, you use your login credentials to connect. Set the following properties:

   • User: This is the username you use to log in to SAP Gateway.
   • Password: This is the password you use to log in to SAP Gateway.

   Authenticate via OAuth Authentication

   You can connect to SAP Gateway using the embedded OAuth connectivity (without setting any additional authentication connection properties). When you connect, the OAuth endpoint opens in your browser. Log in and grant permissions to complete the OAuth process. See the OAuth section in the online Help documentation for more information on other OAuth authentication flows.

   Built-in Connection String Designer

   For assistance in constructing the JDBC URL, use the connection string designer built into the SAP Netweaver Gateway JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

   java -jar cdata.jdbc.sapgateway.jar

   Fill in the connection properties and copy the connection string to the clipboard.

   Configure the connection to SAP Netweaver Gateway, using the connection string generated above.

   scala> val sapgateway_df = spark.sqlContext.read.format("jdbc").option("url", "jdbc:sapgateway:User=user;Password=password;URL=https://sapes5.sapdevcenter.com/sap/opu/odata/IWBEP/GWSAMPLE_BASIC/;").option("dbtable","SalesOrderLineItems").option("driver","cdata.jdbc.sapgateway.SAPGatewayDriver").load()
3. Once you connect and the data is loaded you will see the table schema displayed.

4. Register the SAP Netweaver Gateway data as a temporary table:

   scala> sapgateway_df.registerTable("salesorderlineitems")

5. Perform custom SQL queries against the Data using commands like the one below:

   scala> sapgateway_df.sqlContext.sql("SELECT ProductID, Quantity FROM SalesOrderLineItems WHERE Quantity = 15").collect.foreach(println)

   You will see the results displayed in the console, similar to the following:

Using the CData JDBC Driver for SAP Netweaver Gateway in Apache Spark, you are able to perform fast and complex analytics on SAP Netweaver Gateway data, combining the power and utility of Spark with your data. Download a free, 30 day trial of any of the 200+ CData JDBC Drivers and get started today.