1. Data & Insights Client Center
  2. Data Publishing
  3. Gateway Integration

Gateway Readme for Linux

Avatar
Andrew
Last Updated:
Follow

Table of Contents

  1. Overview
  2. System Requirements
  3. Set Up Instructions
    1. STEP 1: Create an Agent
      1. Starting the Agent
      2. Managing the Agent
    2. STEP 2: Add and Configure a Plugin 
      1. Option 1: Start from the Command Line
      2. Option 2: Start from the Data & Insights data site
      3. Reconfigure a plugin
      4. Removing a plugin

Overview

Establishing a Data & Insights Gateway connection is a two-step process:

  1. Create an agent
  2. Add a plugin (or plugins) to that agent

Agents are small programs that establish a secure connection to the Data & Insights platform and act as a manager for a plugin or set of plugins.

After the agent is downloaded and set up, we’ll help you connect to your individual data sources with plugins for specific data types.

You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

System Requirements

The agent requires Java 8 or newer

  • Install Java from your system package manager
    • Debian/Ubuntu: sudo apt install openjdk-11-jre
    • Red Hat or CentOS: sudo yum install java-11-openjdk

Networking
The host must be allowed to make connections to the Data & Insights platform on port 443

SSL
Special SSL and Self-Signed Certificates must be added to the Java trust store. For more information, please see: http://www.thesqlreport.com/?p=576

User Permissions
Sudo (root) or Administrator access is required to configure the service

Set Up Instructions

STEP 1: Create an Agent

Starting the Agent

  1. From your Data & Insights data site click the “Provision Agent” button on either the:
    1. Admin/Gateway page ({Your_Domain}/admin/gateway) OR
    2. When you're choosing your data source while creating or editing a dataset
  2. Follow the agent naming and download process.
  3. Place the downloaded folder in a directory on the server or computer you are connecting to the Data & Insights platform (if it’s not there already). Note or copy this folder path. You’ll need it when you set up a plugin later.
  4. Unzip the downloaded file.
  5. Run `linux/install` in that directory. It will ask a few questions about where and how to run the service, and then set it up to be run automatically by `systemd`.

     

 $ linux/install
 systemd service name: example
 User to run as: myuser
 Directory to run in: /home/myuser/example-agent
 Data directory: /home/myuser/.local/share/socrata-ingress-agent
 Here is the unit file we will generate:
 
 [Unit]
 Description=The Socrata Ingress Agent
 StartLimitIntervalSec=1
 StartLimitBurst=5
 [Service]
 User=myuser
 Group=myuser
 WorkingDirectory=/home/myuser/example-agent
 ExecStart=/usr/bin/java -jar "/home/myuser/example-agent/example.jar" --data-dir "/home/myuser/.local/share/socrata-ingress-agent"
 Restart=on-failure
 RestartSec=1
 PrivateTmp=true
 PrivateDevices=true
 [Install]
 WantedBy=multi-user.target
 
 Is this correct? [Y/n] Y
 created /home/myuser/Downloads/gateway_example/example.service
 created /home/myuser/Downloads/gateway_example/example-control
 
 Setting up the service. The commands being run will be displayed.
 You may be asked for the root password, depending on your local
 sudo configuration.
 
 sudo mkdir -m0755 -p -- /home/myuser/example-agent
 sudo cp -- /home/myuser/Downloads/gateway_example/socrata-ingress-agent.jar /home/myuser/example-agent/example.jar
 sudo chmod 644 -- /home/myuser/example-agent/example.jar
 sudo mkdir -m0700 -p -- /home/myuser/.local/share/socrata-ingress-agent
 sudo chown -- myuser:myuser /home/myuser/.local/share/socrata-ingress-agent
 sudo cp -- /home/myuser/Downloads/gateway_example/example.service /etc/systemd/system/example.service
 sudo systemctl enable -- example.service
 Created symlink /etc/systemd/system/multi-user.target.wants/example.service → /etc/systemd/system/example.service.
 sudo systemctl start -- example.service
 
 Setup complete. You may use the /home/myuser/Downloads/gateway_example/example-control
 script to control, configure, or uninstall the service.

 

On your Data & Insights data site you can check the agent status in the following places:

  • In the “Provision Agent” set up stepper (On the “Set up instructions” page there is an “Am I connected?” widget. Clicking “refresh” will check your connection.
  • On the Admin/Gateway page ({Your_Domain}/admin/gateway), it will appear in the table
  • When you're choosing your data source while creating or editing a dataset it will appear in the list as “Connected”

Managing the Agent

After installing the agent, you can use the generated control script
to manage it:

$ ./example-control
 Usage: ./example-control stop|start|status|logs|run|uninstall
 
 stop - stop the service 
 Important: All plugins associated with this agent will also stop and any schedules associated with datasets created under this agent will stop/break
 start - start the service
 status - display information about the service
 logs - show the service's logs
 -f, --follow output appended data as more is generated
 run ARGUMENTS - run a command on the service jar
 Use "./example-control run --help" for more information
 uninstall - stop the service and delete all its data files

 $ ./example-control start
 $ ./example-control stop
 $ ./example-control status


STEP 2: Add and Configure a Plugin

TIPS:

  • ”PLUGIN_TYPE” is a type from a list you’ll be presented upon set up.
  • “PLUGIN_NAME” is a name you select to identify this plugin on the Data & Insights data site.
  • Configuration for the plugin type and plugin name are case sensitive. Please ensure you’re using the case presented at set up.
  • You can add as many plugins as you like to a single agent, including multiple plugins of the same type.

Option 1: Start from the Command Line

Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.

  1. To list available plugin types
    1. $ ./example-control run --list-available-plugins
  2. To install or reconfigure a plugin
    1. $ ./example-control run --configure-plugin PLUGIN_TYPE:PLUGIN_NAME
  3. When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on the Data & Insights platform.
  4. Fill out the configuration information and click “OK”. Return to the data site and your plugin will then be ready to use!


Option 2: Start from the Data & Insights data site

  1. Click "Add Plugin" either
    1. On the Admin/Gateway page ({Your_Domain}/admin/gateway)
    2. When you're choosing your data source while creating or editing a dataset
  2. Choose your plugin.
  3. Name your plugin.
    1. Naming tip: Who is responsible + Data source description + Data source type
    2. Example: Finance_Risk_CSV
    3. Non alphanumeric characters will be converted to underscores
  4. Open the command prompt on your computer or server and navigate to the directory where the downloaded agent jar file resides.
  5. Copy and paste the plugin command into the command prompt.
  6. Run the command by hitting enter or return. This will download the plugin from the Data & Insights data site, verify it, and run it on your server.
    Tip: If you get an error, try changing directories.
  7. When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on the Data & Insights platform.
  8. Fill out the configuration information and click “OK”. Return to the data site and your plugin will then be ready to use!


Reconfigure a plugin

  1. On the computer or server your plugin has been configured, in the command prompt enter:
    1. $ ./example-control run --configure-plugin PLUGIN_TYPE:PLUGIN_NAME
  2. When the plugin is run it will display a window on your server asking for plugin-specific configuration and credentials (if needed). This information will not live on the Data & Insights platform.
  3. Fill out the configuration information and click “OK”. Return to the data site and your reconfigured plugin will then be ready to use!

Removing a plugin

Important: All schedules associated with the dataset(s) created under this plugin will break when you remove the plugin.

  1. On the computer or server your plugin has been configured, in the command prompt enter:
    1. $ ./example-control run --delete-plugin PLUGIN_TYPE:PLUGIN_NAME
Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.

Articles in this section

See more