Working with Point Data Using the FME Socrata Writer

• Introduction
• Publishing new datasets
• Updating existing datasets
  • Latitude/longitude point data
  • Latitude/longitude location data
  • Point data not stored as latitude/longitude (e.g. state plane)
  • Publishing address data to be geocoded
• Related articles

Introduction

This article describes ways to publish point data using the Safe FME Socrata Writer. Currently, the Socrata Writer only supports point (location) data. Data & Insights can also be used to geocode U.S. addresses to generate lat/long coordinates on existing datasets with the location datatype. 

NOTE: Before reading this guide you should be sure to read the guide on using the FME Socrata Writer.

Publishing new datasets

Starting on August 30, 2019, new datasets with location data can no longer be created through the FME Socrata Writer. Data & Insights now uses the point datatype for all new datasets, as we have deprecated the creation of location columns on new datasets. Now when creating datasets using the location datatype, you will receive an error like below:

Moving forward, datasets with location/point data cannot be created through the Socrata Writer and should instead be created directly on your Data & Insights domain or with a Publishing API. A simple workflow would be to first export your data to a CSV file and use it to create a new dataset in the UI.

Once the datasets are created in the UI, you can update existing or new datasets with the FME Socrata Writer as described in the rest of this article.

Updating existing datasets

Latitude/longitude point data

This section applies to existing datasets that use the Point datatype where your source data has latitude and location columns. You can check the datatype of your columns on a datasets Primer page. 

To publish to a single Point datatype column on Data & Insights, the values must be formatted as: “POINT (long lat)”. The easiest way to achieve this is to use an Attribute Creator transformer to add a new field that combines the existing Latitude and Longitude fields into a single Location field formatted correctly. An example of a correctly formatted value is “POINT (-87.61694 41.70737)”. 

If you are publishing to an existing dataset ensure you are publishing the values to a Location datatype column. If necessary, you can change the datatype of a column using the Data & Insights web interface (but FME cannot be used to change datatypes of existing Data & Insights datasets).

Latitude/longitude location data

This section applies to existing datasets that use the location datatype where your source data has latitude and location columns. You can check the datatype of your columns on a datasets Primer page. 

To publish to a single Location datatype column on Data & Insights, the values must be formatted as: “(latitude, longitude)”. The easiest way to achieve this is to use an Attribute Creator transformer to add a new field that combines the existing Latitude and Longitude fields into a single Location field formatted correctly. An example of a correctly formatted value is “(41.70737, -87.61694)”. 

If you are publishing to an existing dataset ensure you are publishing the values to a Location datatype column. If necessary, you can change the datatype of a column using the Data & Insights web interface (but FME cannot be used to change datatypes of existing Data & Insights datasets).

Point data not stored as latitude/longitude (e.g. state plane)

If your point data is not already represented as Latitude and Longitude (a.k.a. WGS 84 or EPSG:4326) then the data must be reprojected. If the data source you are reading is not a tabular data file (e.g. CSV or Excel spreadsheet) you should use an AttributeReprojector transformer to reproject the data to LL84.

Publishing address data to be geocoded

Please note that the below will only work when writing to existing datasets with the Location Datatype. Point datasets only accept lat and long values.

Data & Insights can be used to geocode address data to generate latitude/longitude values for valid U.S. addresses. You need to publish the address as a single Location datatype column on Data & Insights. If the address is not already within a single field (for example if you have separate columns for street, city, state, etc) you can use a StringConcatenator transformer to concatenate the address columns into a single Location field formatted correctly. An example of a correctly formatted value is “1201 E 12th St, Seattle, WA 98102”. Find more information on acceptable address formats and avoiding errors geocoding.

IMPORTANT: When publishing address data to be geocoded be sure to set the ‘Use Data & Insights Geocoding’ field to ‘Yes’ in the Socrata Writer. This value should be set to ‘No’ in cases where you are publishing Latitude and Longitude data directly to the Location column (see image below).

Related articles

• Socrata Safe FME resources
• Safe FME ETL tools & connector templates: includes templates for reprojecting, publishing data in Shapefiles, ArcGIS Online, and ArcGIS Server.
• Using geospatial data on the Data & Insights platform
• Geocoding with Data & Insights