When using the Socrata Writer to update an existing dataset that has computed columns, you may encounter a failure message that looks like this:
Polling the job status: FAILURE: Processing TEMP.csv.sdiff.gz failed: Fields not in dataset: ":@computed_region_4b54_tmc4", ":@computed_region_p8tp_5dkv", ":@computed_region_4a3i_ccfj", ":@computed_region_kxmf_bzkv"
Datasets that have location data that intersects a Spatial Lens Boundary will have computed columns. These columns are created and curated by the Data & Insights platform, not the user. They are used to perform the geographic join between the geocoded row and the underlying spatial lens polygon. The column names begin with the prefix
In order to address this issue in FME, there is now an FME template in the Safe FME Hub that will address this issue: https://hub.safe.com/templates/socrata_computed_columns.
As a workaround, this FME workspace will use a headless DataSync job to update the existing dataset rather than the Data & Insights Writer. You will need to have DataSync installed for this process. It can either be downloaded and installed, or you can use FME to find the version of DataSync you have installed within FME.