Technical Update: Dataset Storage Upgrade
Update: Storage upgrade complete as of March 2020.
As announced in November, we have been performing and are continuing to perform a rolling upgrade of storage configurations for datasets on the Data & Insights platform. This change is part of an ongoing effort to expand the capability of the Data & Insights platform, improve response times, increase availability and simplify how external users, applications and tools interact with your data.
The process of upgrading existing assets entails migrating them to our New Back End, and causing all new assets to be created in that environment. In general, we expect to provide continuity of service for the API, data publishing, and user experience on Data & Insights during this transition. This change is among the largest technical migrations in the history of Data & Insights, however, and there are some noticeable changes in system behavior. In particular, some publishing workflows have been impacted by changes in data validation logic for upgraded datasets, and certain characteristics of dataset downloads and the column metadata API are altered.
The benefits of this upgrade are long-term and impact the full surface area of the Data & Insights platform. We are committed to helping all customers enjoy the benefits of this upgrade and continue to prioritize the resolution of any upgrade-related blockers.
Benefits of Storage Upgrade
There will be two notable, intended changes for migrated assets that will result in a faster, more user-friendly developer experience.
1. Significantly improved performance for all assets. Assets that go through the storage upgrade process see a 3-10x improvement in response times to queries that return data rows (including /api/resource calls and rows.json calls).
2. Datasets with upgraded storage configurations will no longer be required to go through the Data & Insights Dataset Refresh Process, so changes in source data will propagate much more rapidly through the system. Please note that the replication and migrations API will no longer return results for datasets that have been upgraded.
Changes to Existing Workflows
Some community users and external developers may have leveraged the replication and migrations API endpoints historically. While we expect all previously supported queries to work, we understand that certain workflows that rely on the replication API will be interrupted. Affected users should be able to simplify their workflows and simply use the dataset ID available in the browser.
Data Publisher Impact
Because this upgrade significantly changes how data flows through the Data & Insights system, some customer's publication workflows have been impacted and in some cases have experienced issues or failures. Many of the issues customers have experienced have been related to column type validation and data reconciliation on ingress, and because we do not log the raw data that customers ingress onto our platform, we had a limited ability to anticipate many of these issues.
To help more easily identify the kinds of data uploads that may run into column validation issues, detailed information about column validation is now available here. If you encounter issues with ETL jobs or dataset updates that have begun to fail after having previously and reliably succeeded, you may be encountering an issue of this nature. Please let us know via Support, and your issue will be fast-tracked for immediate investigation. We will make adjustments to the system to support existing workflows when possible, but in some cases that the most straightforward path to resolution may be a modification of your ETL process. If such a change is required, Tyler can provide coaching and in some cases consultative support to assist with your transition.
We believe that any issues of this nature will only require one-time changes, and we are committed to supporting these changes and facilitating a smooth transition.
Questions or Concerns?
We are committed to supporting our customers throughout this transition with proactive updates, up-to-date resources, and one-on-one issue resolution.
- For an up-to-date list of known issues affecting the platform, bookmark our Known Issues article.
- More information about column validation is now available in the knowledge base.
- You can also keep an eye on our Deprecation Roadmap for features being deprecated as a result of this storage upgrade.
- For technical support or coaching, please contact us at firstname.lastname@example.org with any questions.
We appreciate your patience with our dust as we work to build a better Data & Insights experience!