Datasets

Datasets let a plugin publish structured data that other extensions can then read. This data can be static (usually through a JSON file) or can be fetched from a dynamic source such as an API.

Creating a dataset

You can create a dataset by creating a definition in the plugin manifest.json file.

Example:

{
  // [...]
  "datasets": [
    { "$ref": "datasets/my-dataset/manifest.json" },
    { "$ref": "datasets/my-dynamic-dataset/manifest.json" }
  ]
}

Each $ref path resolves relative to the plugin manifest. Paths that escape the plugin directory are rejected at load time.

Dataset definition

Similar to plugins, a dataset is defined by a manifest.json file.

Reference

Please reference https://overseer.studio/schemas/dataset/manifest.json for a complete definition of the file.

Static datasets

Static datasets ship a JSON array of records. Extensions can read, paginate, and optionally create, update, or delete records.

To create a static dataset, create a datasets/my-dataset/manifest.json file defines a "local" source.

{
  "id": "my-dataset",
  "label": "D&D 5e Monsters",
  "description": "Stat blocks for the SRD monsters.",
  "version": "1.0.0",
  "idField": "slug",
  "source": {
    "local": { "$ref": "./monsters.json" }
  }
}

Then create another file named monsters.json in the same directory. this file must be be a JSON array and each record must contain the idField property.

[
  { "slug": "goblin", "name": "Goblin", "cr": 0.25, "hp": 7 },
  { "slug": "orc", "name": "Orc", "cr": 0.5, "hp": 15 }
]

Dynamic datasets

Dynamic datasets expose named URL templates that resolve to remote API calls. Extensions invoke an action by name with parameters and Overseer will fetch and return the raw JSON response.

To create a dynamic dataset, create a datasets/my-dynamic-dataset/manifest.json file defines a "remote" source.

{
  "id": "my-dynamic-dataset",
  "label": "D&D 5e Monsters (API)",
  "description": "Queries the Open5e API.",
  "version": "1.0.0",
  "source": {
    "remote": {
      "search": "https://api.open5e.com/v1/monsters/?search={q}&cr_min={cr_min}&cr_max={cr_max}",
      "byType": "https://api.open5e.com/v1/monsters/?type={type}"
    }
  }
}

URL templates

The URLs you define in the dataset manifest.json file will have {paramName} placeholders substituted with values from the caller’s params argument. Only https:// URLs can be defined.

Using datasets

Extensions can invoke dataset actions via the SDK.

See the SDK for examples of invoking static and dynamic datasets.

Dataset ID

Each dataset’s ID is constructed by combining the plugin ID with the dataset manifest’s id:

@my-name/my-plugin:my-dataset

For a plugin with id: "@my-name/my-plugin" and a dataset with id: "my-dataset", the resulting dataset ID is @my-name/my-plugin:my-dataset.

Limits

Datasets above 50,000 records are truncated to that count with a warning in the logs. Datasets whose records file is larger than 50 MB are skipped entirely (also with a warning) to protect the main process from out-of-memory crashes. If your catalog is larger, split it into multiple datasets or expose it as a dynamic dataset.

User edits

When an extension calls create, update, or delete through the SDK, the change is persisted to a user-owned JSON file, not the records file you shipped. User edits live outside the plugin directory, so reinstalling or updating your plugin does not wipe a user’s homebrew records or overrides.

On read, user records are merged over bundled records. On ID collision, the user record wins. Deleting a user record that overrides a bundled one reverts the bundled version on the next read.