Create Dataset on Platform
For this guide, the platform UI will be used to build the shell of a dataset that FME will write data to. Configure the dataset's schema, metadata, etc. and make sure to include the computed column that will store geocoded data as a point.
Set Up FME Workspace
Note: The attached file at the end of the article contains the FME transformers contained in a Bookmark.
- Set Python Interpreter (Preferences > Translation > Python Interpreter: Python 3.5+)
- Add the Socrata-py library to FME:
- Download the Socrata-py .zip located here and unzip the file (it will appear as socrata-master)
- Navigate to the python folder within FME, likely similar to ‘/program files/FME/2018.1/python’
- Grab the socrata folder within socrata-master and add the folder into the python directory
- Set the FME workspace to read from Python 3.5+. (Navigation Pane > Workspace Parameters > Scripting > Python Compatibility = Python 3.5+)
- Add the Bookmark containing the FeatureWriter and AutomateThisPython transformers to the workspace.
- Set workspace parameters:
- domain: domain to publish data to (no https://)
- username: Data & Insights username
- password: Data & Insights password
- dataset_uid: The dataset’s 4x4 that will be updated/replaced
- upload_config: The configuration name for uploading data (See step: choosing upload config)
- csv_name: The name of the csv that will be created and uploaded to the data platform
- csv_folder: The folder the csv will be output for writing to the data platform
Note: The workspace parameters are set on the workbench itself. Editing the Python Script directly will cause the FME workflow to break.
Get Upload Config
Note: This will only need to be done once, since a saved config can be re-used.
- Open the FME workspace used for automation.
- Run the workspace, writing the final data to a .csv with the same columns as the payload that the Socrata Writer sends.
- Navigate to the dataset that will be updated in the Data & Insights platform.
- Open a new revision by clicking the Edit button.
- On Add Data, choose whether you plan to Update or Replace the data
- Upload the .csv created by the FME workspace
- Click the "Automate This" button at the bottom
- Choose whether the script should be used to update or replace data (this decision is what will be used during automation, so choose with that action in mind)
- See the line starting with “(ok, job) = socrata.using_config(…” the first argument in that line is green text and will have a name similar to the csv you uploaded and include today’s date. Copy that first argument, the green text between the two single apostrophes.
- Paste this into the FME workspace as the parameter ‘upload_config’.
- In the browser with the Automate This script, click "Done."
- Click the “X” in the upper right of the Data Preview window, to return to the revision source.
- Click the ellipsis (…) next to the Update button and select "Delete This Draft" (this because only the config is needed)
Run Workspace in FME
- Connect the final transformer of the workspace to the FeatureWriter in the new bookmark.
- Disable any existing Socrata Writer.
- Run workspace. Workspace will write a .csv then execute Python. The log will show the job progress until the message "Job progress: successful." is received
- The translation will be successful and finish.