When you update your datasets on the Socrata platform, a number of behind-the-scenes processes occur that enable a better data consumption experience for end users via the API, as well as via a number of our products built utilizing the Socrata API. This article explains how this process generally works, and any associated delays in data freshness it may cause.
What happens when a dataset is created?
By default, tabular datasets are created and initially stored in what we call the Original Back End (OBE). Upon creation, a copy of the dataset is replicated to the New Back End (NBE). Each copy of the dataset will have a different UID.
Note: The services underlying the OBE and NBE support different data types. The NBE copy of the dataset may not be identical to the OBE version due to the associated schema changes. See the list of datatypes supported for each version here.
What happens when a dataset is updated?
In order to keep the New Back End copy of the dataset up to date, dataset updates performed on the Original Back End copy are automatically synced to the NBE copy. This sync process will generally begin within about an hour. So, updates made to datasets may not be immediately reflected in the NBE copy of the dataset.
How will this affect user experience?
The NBE is more performant and powerful and allows Socrata to build better and faster features, so it is the back end for most newer Socrata products. Before the syncing process completes, products and features dependent on the OBE dataset will reflect the new data, while the products and features dependent on the NBE dataset will not reflect the data updates until the syncing process has completed. See below for a listing of products/features that fall into each category:
Products using the Original Backend (no data update delay)
- The data table
- Original Visualizations
- SODA 2.0 API Endpoint
Product using the New Backend (~1hr data update delay)
- Native Visualizations in Perspectives
- The New Visualization Experience
- Data Lens
- All Public Finance Apps
- Citizen Connect
- SODA 2.1 API endpoint
How do I know if my data is up to date in both places?
You can check to see if your dataset is up to date in the NBE by accessing the API Docs for the dataset.
On the right hand of the page, you will see a section titled Synchronized Copy. This box will tell you when the dataset was last synchronized and if it is up to date.