Table of Contents
- Metadata Templates Overview
- New State
- SoQL mechanics
- User facing changes
- Data Management Experience Metadata Entry (Changes for Publishers)
- Domain Metadata management (Changes for Admins)
- Getting Started
Metadata Templates Overview
Metadata Templates take the place of domain metadata configurations found here. Much like before, Templates are collections of fieldsets and custom fields. The addition is that fields have a SoQL expression associated with them which allows the site admin to define custom validation.
It's important to note that none of this work changes the way metadata is stored or used or rendered in the platform.
There was a need to express more complex validation, transformation, and dependencies between fields. This is why metadata templates were created. More information on how to use the new feature can be found in this article: Metadata Templates
New State
A metadata template looks similar to the domain configuration structure - it has fieldsets, which have a name and multiple fields within them. A template also has built-in fields, which are the fields we hardcode onto all assets. These are things like Name, Description, Attribution, License, etc.
SoQL and users
The feature at /admin/metadata_templates allows the user to build a template by adding fieldsets and fields.
It also provides a number of components that allow the user to do things without interacting with (or even knowing about) SoQL. For example, there is a component that renders as a checkbox, but when checked/unchecked modifies the expression to be either:
the_metadata_field (optional) or something like
case(
the_metadata_field is null, error('the field is required),
the_metadata_field = ' ', error('the field is required),
true, the_metadata_field
)
(required)
At the time of writing, these specialized components include:
-
Making a field a multiple-choice field and adding/deleting choices
-
Making a multiple-choice field a regular text field
-
Making an optional field required
-
Making a required field optional
SoQL mechanics
Each field has a SoQL expression associated. For example, to represent a field called animal which must be dog, cat, rabbit, we'd write a case expression where we return either a literal dog, cat, or rabbit value, or an error. An error value is rendered in the form as the user types.
Field Names
The user might want to call their metadata field, Animal but in SoQL that's not a valid identifier. For this reason, each field needs to carry a display_name and field_name. The Dataset Management API will automatically generate field names as the user builds their metadata form. Field names are only surfaced if the user chooses to use the editor. So, if the user adds a field called Animal, the Dataset Management API will generate animal as the field name.
Qualifiers
In the old experience, different fieldsets can have fields named the same thing. For example, you could have a fieldset called Foo and a fieldset called Bar, each with a field called Qux. This would be valid. But in this experience, we'd end up with two fields in our SoQL schema called qux which would collide. For this reason, we will qualify all custom fields with a qualifier derived from the field set name. So in the previous example, to actually refer to the qux of the Foo fieldset within the SoQL expression, you'd write @qux.foo, and to get the qux from Bar you'd do @bar.qux. This solves the collision issue.
Built-in fields such as Name, Description, Attribution, and Row Label are all unqualified.
Metadata Audit
There is a metadata audit feature that allows the user to use the asset manager to assess metadata validity on a collection of assets. We use the common asset manager component with an extra column that displays the number of validation errors for each asset.
The metadata audit feature will ignore federated assets that are in the catalog from other domains.
User facing changes
Data Management Experience Metadata Entry (Changes for Publishers)
When metadata templates are enabled, the Data Management Experience metadata entry form changes to use the template as the source of truth for which metadata fields show up, instead of using the domain configuration. The changes here a fairly minor but are worth calling out
UI changes related to Metadata Validation
Whereas before validation was limited to required fields or optional fields, now validation can take on any form you can express in a SoQL expression. Because each metadata validation expression can return an error, just like with data validation, it is displayed in the UI as whatever the user chooses to return. This means the little red error messages next to each field can say whatever you'd like. Take for example this expression:
case(
to_number(@some_fieldset.`simple_validation`) < 5,
@some_fieldset.`simple_validation`,
true, error('field needs to be < 5')
)
UI changes related to Metadata Transformation
Whereas before metadata upon publishing was whatever the user input, now metadata fields can be transformed, just like dataset data. Now, because the output could differ from the input, we need to show it somehow.
The change here is, whenever the output of a metadata expression differs from the input, we show an element that displays the output. There is also help text that describes what is happening.
Take the following expression:
case(
@some_fieldset.`simple_transformation` == 'a', 'the user entered a',
@some_fieldset.`simple_transformation` == 'b', 'the user entered b',
@some_fieldset.`simple_transformation` == 'c', 'the user entered c',
true, @some_fieldset.`simple_transformation`
)
For the inputs, a, b, and c, the result of this expression is not the same as the input. For other values, it's just whatever the user entered.
Domain Metadata management (Changes for Admins)
Categories UI
The categories UI was overhauled as part of this work, however, there aren't any real changes to how it functions.
Fieldset and Field Builder
The custom fieldset and field builder UI has changed. You access it in the same way from the Admin page by clicking the "Metadata" link. The UI is split into category management, fieldset management, and metadata audit.
Custom SoQL expressions
The SoQL expression editor associated with each field is accessible at the top right of the field manager. Admins are free to define whatever expression they see fit for the metadata field. The constraints are the same as data transformations, as in, it produces a single value. The "columns" available to reference are the metadata input fields they have defined.
It works in the same way as data transformations, with the same functions available. The documentation is available below the editor, just as it is in Data Management Experience.
Metadata audit
The metadata audit tool allows an admin to run metadata validation against existing assets. It uses the common catalog component to display and interact with the list of assets on the domain.
Getting Started
For tips on how to get started, visit “Metadata Templates” at Metadata Templates
Comments
Article is closed for comments.