Access CSV Data in SharePoint External Lists

You can use the CData API Server with the ADO.NET Provider for CSV (or any of 200+ other ADO.NET Providers) to give your users the capabilities to access CSV data in SharePoint. This article shows how to create an external list of CSV data that is always up to date. You will use an external content type as a template to create the external list. The external content type enables connectivity through OData, a real-time data streaming protocol for mobile and other online applications. The API Server is an OData producer of CSV feeds.

After setting up the API Server, creating an external list with connectivity to CSV data consists of three basic steps:

1. Create the External Content Type
2. Import the External Content Type
3. Create the External List

This article also covers how to accomplish the following tasks:

• Limit Results
• Configure Pass-Through Authentication
• Configure Credentials Authentication

Set Up the API Server

Follow the steps below to begin producing secure CSV OData services:

Deploy

The API Server runs on your own server. On Windows, you can deploy using the stand-alone server or IIS. On a Java servlet container, drop in the API Server WAR file. See the help documentation for more information and how-tos.

The API Server is also easy to deploy on Microsoft Azure, Amazon EC2, and Heroku.

Connect to CSV

After you deploy the API Server and the ADO.NET Provider for CSV, provide authentication values and other connection properties needed to connect to CSV by clicking Settings -> Connections and adding a new connection in the API Server administration console.

The DataSource property must be set to a valid local folder name.

Also, specify the IncludeFiles property to work with text files having extensions that differ from .csv, .tab, or .txt. Specify multiple file extensions in a comma-separated list. You can also set Extended Properties compatible with the Microsoft Jet OLE DB 4.0 driver. Alternatively, you can provide the format of text files in a Schema.ini file.

Set UseRowNumbers to true if you are deleting or updating in CSV. This will create a new column with the name RowNumber which will be used as key for that table.

SharePoint consumes OData 3.0 data sources. You can configure the API Server for compatibility with SharePoint by clicking Settings -> Server and then selecting OData 3.0 in the Default OData Version menu.

You can then choose the CSV entities you want to allow the API Server to access by clicking Settings -> Resources.

Authorize API Server Users

After determining the OData services you want to produce, authorize users by clicking Settings -> Users. The API Server uses authtoken-based authentication and supports the major authentication schemes. Access can also be restricted based on IP address; by default only connections to the local machine are allowed. You can authenticate as well as encrypt connections with SSL.

Create the External Content Type

The external content type is a schema that will provide the core connectivity to CSV data from any SharePoint app. You can create a schema for any OData query. You can pass in the required options with the $sharepoint query string parameter. Below is an example request, which will return the schema in an .ect file:

Note that for simplicity, the authtoken is passed in the query string to authenticate the request for the .ect. This is not enabled by default; if you would like to use this method to authenticate to the API Server, you will need to add an entry like the following to your settings.cfg file:

The settings.cfg file is located in the data directory. In the .NET edition, the data directory is located in the app_data subfolder of the application root. In the Java edition, the location of the data directory depends on your operating system:

• Windows: C:\ProgramData\CData
• Unix or Mac OS X: ~/cdata

Import the External Content Type

After you have created the .ect, you can follow the steps below to import it into SharePoint Online or an on-premise SharePoint installation.

SharePoint 2013

Navigate to the SharePoint central administration portal and click the link to manage service applications. In the resulting page, click Business Data Connectivity Service. Select External Content Types in the menu and click Import. In the BDC Model section, click Choose File to select the .ect file in the dialog.

SharePoint Online

In the SharePoint admin center, click BCS from the quick launch menu and then click Manage BDC Models and External Content Types. On the resulting page, select External Content Types in the menu and click Import. In the BDC Model section, click Choose File.

Create the External List

You can now create SharePoint apps that can access and modify CSV data:

1. Navigate to your SharePoint site and choose Site Contents -> Add an App -> External List.
2. Click the Select External Content Type icon and choose the external content type that you created in the previous section.
3. Click Create.

Limit Results

SharePoint has limits on how much data can be retrieved from external lists of OData sources. External lists display results in pages of 30 items by default. To modify the paging size, you can set the Item Limit property in the settings for the default view. Alternatively, you can build a custom Web part to view the data from the external list.

In SharePoint 2013, requests to external data sources are limited by bandwidth throttling controls, which can be changed using the Set-SPBusinessDataCatalogThrottleConfig command. In SharePoint Online, requests to external data sources are limited by your Server Resource Quota and by bandwidth throttling controls; to work around this, you can apply filters in the request for data.

The API Server sets the default limit for the number of returned rows to be 500. You can disable this limit by adding the limit option to the $sharepoint query string parameter and setting its value to 0.

Configure Pass-Through Authentication

To use pass-through authentication for accessing your external content type, set the AuthMode option to pass-through.

If you are using Kerberos authentication, you need to add these users to the API Server. If you are not using Kerberos authentication, you are likely using another form of Windows authentication such as NTLM.

When users are authenticating via NTLM authentication and using pass-through authentication in the Business Connectivity Service (BCS) in SharePoint, SharePoint connects to the external Web service using the default IIS account. This account is often the NT AUTHORITY\IUSR account. Give this user access to the CData API Server.

See the help documentation for a guide to enabling Windows authentication for the API Server.

Configure Credentials Authentication

In the following sections, you will first create a secure store target application that authenticates SharePoint users to the API Server with the credentials for a user who has been added to the API Server. Next, you will create the external content type and configure it to authenticate with the credentials in the secure store.

• SharePoint Online
• SharePoint 2013

SharePoint Online

1. In the SharePoint Online administration center, click secure store from the quick launch bar and then click New.
2. In the Target Application Settings section, enter the target application Id, display name, and contact email.
3. In the Credential Fields section, add the credentials for a user in the API Server.
4. In the Target Application Administrators section, choose an administrator who can access the connection settings for the external content type. The account for the SharePoint Online administrator is usually specified here.
5. In the Members section, enter the users in SharePoint who are authorized to access CSV data.

After you create the target application, save the credentials of The API Server user into the secure store:

1. In the quick launch bar, click secure store.
2. Click Set Credentials in the menu for the target application.
3. Enter the username and password of a user in the API Server.

Next, create a new connection settings object:

1. Click BCS from the quick launch bar and then choose the option to manage connections to online services.
2. Click Add. Name the connection and enter the URL of the OData endpoint, https://my-server/api.rsc.
3. Select the option to use credentials stored in SharePoint. Enter the application Id for the target application you created.

Finally, create the external content type, import it into SharePoint Online, and create the external list:

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter. Set the AuthMode option to Credentials. Specify the target application Id and the Id of the connection settings object. For example:
   https://my-server/api.rsc/Customer?$filter=FirstName eq 'Bob'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
2. Import the resulting .ect file: Open BCS from the quick launch bar and click the link to manage BDC models and external content types. Click Import.
3. You can now create the external list.

SharePoint 2013

1. Browse to the central administration area for your SharePoint site and click the link to manage service applications.
2. Click Secure Store Service and then click New.
3. Enter the application Id, display name, and contact email to configure the new secure store target application. Set the target application type to Group.
4. On the next page, add the field names and field types to display when the user enters their username and password. Select the username and password field types.
5. Next, in the Target Application Administrators section, choose administrators who are authorized to access connection settings for the external content type. In the Members section, enter the users in SharePoint who are authorized to connect with the credentials of the API Server user.

After you create the new secure store target application, follow the procedure below to set the credentials that users in SharePoint will provide when accessing the API Server:

1. Click Set Credentials in the menu for the target application.
2. In the resulting menu enter the username and password (authtoken) of a user in the API Server.

3. Open an instance of the SharePoint management shell and initialize a new SharePoint connection object. Note that the connection object is site specific.

   New-SPODataConnectionSetting -AuthenticationMode Credentials -ServiceAddressURL http://my-server/api.rsc -ServiceContext http://myspsite/ -Name MyODataConnectionSettingsId -SecureStoreTargetApplicationId my-target-application-Id

   The output below shows that the command has executed successfully:

The final steps involve creating the external content type for the table you want to expose in SharePoint, configuring it to authenticate with credentials from the secure store, and then importing it into SharePoint.

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter: Below is an example request:
   http://my-server/api.rsc/Account?$filter=Industry eq 'Floppy Disks'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
2. To import the external content type into SharePoint, point your browser to the SharePoint central administration portal and click the link to manage service applications.
3. Click Business Data Connectivity Service in the resulting page.
4. Click Import and select the .ect file in the dialog.
5. You can now create the external list. On the Site Contents page on your SharePoint site, click the button to add an app and then click the icon to choose the external content type.