Socrata Gateway for Data Publishers

Data Publishers can use Socrata Gateway to create and update datasets using the Gateway agent and plugins. Both the Agent and Plugin can be created in the Socrata Dataset Management Experience (SDMX).

Connect to an External Data Source

The first thing you will do will be to either create a new dataset or edit an existing one - taking you to the main page in SDMX.

You will then need to select Add Data in the Data Actions section.

Pasted_Image_8_2_19__11_03_AM.png

From there, you will want to choose the option Connect to an External Data Source.

 mceclip0.png

You will now be on the Socrata Gateway page for your dataset. This page will list any plugins already created to choose from. If your plugin is already created skip to Connect Datasource. Otherwise, your next step will be to Provision a new agent.

Provision Agent

The first step you will take is to provision a new agent. To do so, click the Provision Agent button at the top right of the page.

Pasted_Image_8_2_19__11_12_AM.png

This will launch a new window with additional instructions. The first thing you will do is to name your agent and download it. The agent will download as a .zip file containing an executable .jar program and a README* file. Once you agent is downloaded, move on to the Next page.

Pasted_Image_8_2_19__11_32_AM.png

*Tip: The README file is a great resource with instructions for MacOS, Windows and Linux.

Setup Agent

Once you have downloaded the agent, the next step will be to set it up. The window will allow you to view instructions for Windows, Mac, or Linux machines. In general, the steps to setting up a new agent will be as follows:

  1. Place the downloaded folder on the server or computer you are connecting to Socrata (if it’s not there already).
  2. Extract/Unzip the downloaded file.
  3. Open the new subfolder containing the zipped contents of the download.
  4. Run the agent as a service

Once the agent is running, go back to the Agent Stepper and click refresh at the bottom of the page, this will prompt Socrata to check to see if the agent is running. You will not be able to move to the next page until the agent is confirmed to be running. Once it is confirmed that the Agent is running, click Next in the window.

Pasted_Image_8_2_19__11_33_AM.png

Next Steps

You will now have the option to either set up a plugin now or choose to do it a later time, the following section will walk through creating a plugin.

Create a Plugin

A plugin is used to connect directly to a specific source system. Once a plugin is configured, users can create datasets from any data the plugin is configured to access for that source. You can set-up a plugin on the final page of the Agent set up tool or at any point with the Add Plugin button.

When adding a plugin, the first thing you will do is search or browse the list to determine the plugin associated with your source system.

mceclip0.png

In this example, I have chosen the CSV option. Selecting Set-Up will open a new window with steps to configuring your plugin.

The first page is a plugin overview, it will contain a description and any required fields needed to run the plugin:

Pasted_Image_8_2_19__11_27_AM.png

On the second page, you will name your plugin, the name should help you understand the data source you are connecting to.

The final page will give you the Set-up instructions for the plugin.

  1. During the agent set up process, if you didn’t note the path of your downloaded agent file, gather that path now. It will be called <Gateway_The name you called your agent>.
  2. Open the command prompt (Windows) or Terminal (Mac) and navigate to that folder where you have placed the agent. This can be done using the cd (change directory) command.
  3. Copy and paste this command into the command prompt/terminal.
java -jar socrata-ingress-agent.jar --configure-plugin csv:Andrew_CSV_1
  1. Run the command by hitting enter or return. This will download the plugin from Socrata, verify it, and run it on your server. If you get an error, try changing directories.
  2. When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on Socrata.

Pasted_Image_8_2_19__11_49_AM.png

  1. Fill out the configuration information and click “OK”. 

Once completed you can return to Socrata, the Done option in the setup window should now be available.

Creating a Dataset

Now that your plugin is available, you can select it and connect it to the data source you are using for the dataset.

mceclip0.png

In this example, I will select the CSV plugin, there is an arrow dropdown that will show a list of all available files that can be published to your dataset. To choose the source data, select Use this data source.

mceclip1.png

Selecting the data source will then load the data into Socrata, and you can proceed to set up the dataset just as you would any other!

Schedule Dataset Updates

Once the dataset has been published, access the Primer page and open the dropdown menu to find the Schedule Updates option.

 Pasted_Image_8_2_19__10_34_AM.png

 

This will open a modal where you can configure the frequency of update in days, the time of day the update will occur as well as view or change the datasource settings. In the below example, the dataset would update at 6 AM Pacific time every day:

Once the time and frequency are set, click Save Schedule. The schedule can be modified at any time by choosing the Schedule Updates option.

Pasted_Image_8_2_19__11_52_AM.png

Congratulations! You now have a fully automated Gateway powered dataset! 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.