Guide to using Automate This! with FME

Create Dataset on Platform

For this guide, the platform UI will be used to build the shell of a dataset that FME will write data to. Configure the dataset's schema, metadata, etc. and make sure to include the computed column that will store geocoded data as a point.

Set Up FME Workspace

Note: The attached file at the end of the article contains the FME transformers contained in a Bookmark.

• Set Python Interpreter (Preferences > Translation > Python Interpreter: Python 3.5+)
• Add the Socrata-py library to FME:
  • Download the Socrata-py .zip located here and unzip the file (it will appear as socrata-master)
  • Navigate to the python folder within FME, likely similar to ‘/program files/FME/2018.1/python’
  • Grab the socrata folder within socrata-master and add the folder into the python directory
•  Set the FME workspace to read from Python 3.5+. (Navigation Pane > Workspace Parameters > Scripting > Python Compatibility = Python 3.5+)
•  Add the Bookmark containing the FeatureWriter and AutomateThisPython transformers to the workspace.
• Set workspace parameters:
  • domain: domain to publish data to (no https://)
  • username: Data & Insights username
  • password: Data & Insights password
  • dataset_uid: The dataset’s 4x4 that will be updated/replaced
  • upload_config: The configuration name for uploading data (See step: choosing upload config)
  • csv_name: The name of the csv that will be created and uploaded to the data platform
  • csv_folder: The folder the csv will be output for writing to the data platform

Note: The workspace parameters are set on the workbench itself. Editing the Python Script directly will cause the FME workflow to break.

Get Upload Config

Note: This will only need to be done once, since a saved config can be re-used.

In FME:

• Open the FME workspace used for automation.
• Run the workspace, writing the final data to a .csv with the same columns as the payload that the Socrata Writer sends.

On platform:

• Navigate to the dataset that will be updated in the Data & Insights platform.
• Open a new revision by clicking the Edit button.
• On Add Data, choose whether you plan to Update or Replace the data
• Upload the .csv created by the FME workspace
• Click the "Automate This" button at the bottom
• Choose whether the script should be used to update or replace data (this decision is what will be used during automation, so choose with that action in mind)
• See the line starting with “(ok, job) = socrata.using_config(…” the first argument in that line is green text and will have a name similar to the csv you uploaded and include today’s date. Copy that first argument, the green text between the two single apostrophes.

In FME:

• Paste this into the FME workspace as the parameter ‘upload_config’.

On platform:

• In the browser with the Automate This script, click "Done."
• Click the “X” in the upper right of the Data Preview window, to return to the revision source.
• Click the ellipsis (…) next to the Update button and select "Delete This Draft" (this because only the config is needed)

Run Workspace in FME

• Connect the final transformer of the workspace to the FeatureWriter in the new bookmark.
• Disable any existing Socrata Writer.
• Run workspace. Workspace will write a .csv then execute Python. The log will show the job progress until the message "Job progress: successful." is received
• The translation will be successful and finish.