Guide to using Automate This! with FME

Create Dataset on Platform

For this guide, the platform UI will be used to build the shell of a dataset that FME will write data to. Configure the dataset's schema, metadata, etc. and make sure to include the computed column that will store geocoded data as a point.

Set Up FME Workspace

  • Set Python Interpreter (Preferences > Translation > Python Interpreter: Python 3.5+)
  • Add the Socrata-py library to FME:
    • Download the Socrata-py .zip located here and unzip the file (it will appear as socrata-master)
    • Navigate to the python folder within FME, likely similar to ‘/program files/FME/2018.1/python’
    • Grab the socrata folder within socrata-master and add the folder into the python directory
  •  Set the FME workspace to read from Python 3.5+. (Navigation Pane > Workspace Parameters > Scripting > Python Compatibility = Python 3.5+)
  •  Add the Bookmark containing the FeatureWriter and AutomateThisPython transformers to the workspace.
  • Set workspace parameters:
    • domain: domain to publish data to (no https://)
    • username: Socrata username
    • password: Socrata password
    • dataset_uid: The dataset’s 4x4 that will be updated/replaced
    • upload_config: The configuration name for uploading data (See step: choosing upload config)
    • csv_name: The name of the csv that will be created and uploaded to Socrata
    • csv_folder: The folder the csv will be output for writing to Socrata

Get Upload Config

Note: This will only need to be done once, since a saved config can be re-used.

In FME:

  • Open the FME workspace used for automation.
  • Run the workspace, writing the final data to a .csv with the same columns as the payload that the Socrata writer sends.

On platform:

  • Navigate to the dataset that will be updated in Socrata.
  • Open a new revision by clicking the Edit button.
  • On Add Data, choose whether you plan to Update or Replace the data
  • Upload the .csv created by the FME workspace
  • Click the "Automate This" button at the bottom
  • Choose whether the script should be used to update or replace data (this decision is what will be used during automation, so choose with that action in mind)
  • See the line starting with “(ok, job) = socrata.using_config(…” the first argument in that line is green text and will have a name similar to the csv you uploaded and include today’s date. Copy that first argument, the green text between the two single apostrophes.

In FME:

  • Paste this into the FME workspace as the parameter ‘upload_config’.

On platform:

  • In the browser with the Automate This script, click "Done."
  • Click the “X” in the upper right of the Data Preview window, to return to the revision source.
  • Click the ellipsis (…) next to the Update button and select "Delete This Draft" (this because only the config is needed)

Run Workspace in FME

  • Connect the final transformer of the workspace to the FeatureWriter in the new bookmark.
  • Disable any existing Socrata writer.
  • Run workspace. Workspace will write a .csv then execute Python. The log will show the job progress until the message "Job progress: successful." is received
  • The translation will be successful and finish.

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.