Data & Insights Gateway consists of both Agents and Plugins. The agent is set up on a server that has access to the source system hosting your data. Various plugins exist for your agent to connect to different source systems both on the cloud (Esri, AWS S3, etc) as well as on-premise (MS SQL, Excel, etc).

• Provision Agent
• Creating Plugin
• Creating a Dataset
• Schedule Dataset Updates
• Managing Agent and Plugin

The first step to setting up your Data & Insights Gateway will be to navigate to Gateway from the Data & Insights admin page on your domain. This can also be navigated to directly at https://domain/admin/gateway.

Provision Agent

Provision and download agent

Once you are in the Gateway section, you will need to provision a new agent. To do so, click the Provision Agent button at the top right of the page. This will launch a new window with additional instructions. The first thing you will do is to name your agent and download it. The agent will download as a .zip file containing an executable .jar program and a README* file. Once your agent is downloaded, move on to the Next page.

*Tip: The README file is a great resource with instructions for MacOS, Windows and Linux.

Setup Agent

Once you have downloaded the agent, the next step will be to set it up. The window will allow you to view instructions for Windows, Mac, or Linux machines. In general, the steps to setting up a new agent will be as follows:

1. Place the downloaded folder on the server or computer you are connecting to Data & Insights (if it’s not there already).
2. Extract/Unzip the downloaded file.
3. Open the new subfolder containing the zipped contents of the download.
4. Open the Folder specific to your OS
5. Run the agent as a service
   1. For Windows this is install.bat
   2. For MacOS this is the setup file

Once the agent is running, go back to the Agent Stepper and click refresh at the bottom of the page, this will prompt Data & Insights to check to see if the agent is running. You will not be able to move to the next page until the agent is confirmed to be running. Once it is confirmed that the Agent is running, click Next in the window.

Next Steps

You will now have the option to either set up a plugin now or choose to do it a later time, the following section will walk through creating a plugin. The Gateway page will also now list your new Agent.

Create a Plugin

A plugin is used to connect directly to a specific source system. Once a plugin is configured, users can create datasets from any data the plugin is configured to access for that source. You can set-up a plugin on the final page of the Agent set up tool within the gateway page or at any point with the Add Plugin button.

Likewise, agents and plugins can be created in the Data & Insights Dataset Management Experience when creating a new dataset.

When adding a plugin, the first thing you will do is search or browse the list to determine the plugin associated with your source system. In this example, I have chosen the CSV option. Selecting Set-Up will open a new window with steps to configuring your plugin.

The first page is a plugin overview, it will contain a description and any required fields needed to run the plugin:

On the second page, you will name your plugin, the name should help you understand the data source you are connecting to.

The final page will give you the Set-up instructions for the plugin.

1. During the agent set up process, if you didn’t note the path of your downloaded agent file, gather that path now. It will be called <Gateway_The name you called your agent>.
2. Open the command prompt (Windows) or Terminal (Mac) and navigate to that folder where you have placed the agent. This can be done using the cd (change directory) command.
3. Copy and paste this command into the command prompt/terminal.

java -jar socrata-ingress-agent.jar --configure-plugin csv:Andrew_CSV_1

4. Run the command by hitting enter or return. This will download the plugin from Data & Insights, verify it, and run it on your server. If you get an error, try changing directories.
5. When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Data & Insights.

6. Fill out the configuration information and click “OK”. 

Once completed you can return to Data and Insights, the Done option in the setup window should now be available. The plugin will also be reflected on the Gateways page:

 

Creating a Dataset 

To create a new dataset that sources data from a plugin, start in the Data & Insights Dataset Management Experience select Add Data > Connect to an External Data Source. This will bring you to a page with a list of available plugins.

If you have not created your agent or plugin yet, you can also do that from this screen following the same instructions as above.

Schedule Dataset Updates

Once the dataset has been published, access the Primer page and open the dropdown menu to find the Schedule Updates option.

This will open a modal where you can configure the frequency of update in days, the time of day the update will occur as well as view or change the datasource settings. In the below example, the dataset would update at 6 AM Pacific time every day:

Once the time and frequency are set, click Save Schedule. The schedule can be modified at any time by choosing the Schedule Updates option.

Congratulations! You now have a fully automated Gateway-powered dataset!

 

NOTE: If the query powering Gateway automation is updated, in order to keep the automation working, it is necessary to manually create a draft of the dataset and reload the dataset in the Data & Insights Dataset Management Experience then publish it. 

Manage Agents and Plugins

Agents and Plugins can be managed from the Gateway admin page. At any point, you can Provision a new Agent or add a Plugin. This will show all agents and plugins set up on your domain.

The actions menu also shows two different options: Delete agent and Transfer Ownership. Delete will completely remove the agent, whereas Transfer Ownership gives an option to assign the point of contact for the agent, who can configure or delete it. Deleting an agent will pause all scheduled updates for all datasets created from all plugins associated with that agent.

You can also view all plugins connected to a single agent on this page.