Concepts

Underlay has four core primitives. Everything else is built from these.

Collection

A collection (plural: collections) is a named, versioned body of structured data. It belongs to an account (a user or an organization) and is identified by :owner/:slug, e.g. knowledge-futures/pubpub-archive.

A collection can be public (browsable by anyone) or private (visible only to the owner and org members). Each collection has its own independent version history.

Version

A version is an immutable snapshot of a collection at a point in time. Each version contains:

• A JSON Schema describing the structure of the records
• A set of records (the actual data)
• References to files (binary assets)
• A metadata object that can contain readme, license, and other fields

Versions are identified by semver (e.g. v1.0.0, v1.1.0, v2.0.0). The semver is derived automatically from what changed:

• Schema changes → major bump
• Record or file changes → minor bump
• Metadata-only changes (readme, license, etc.) → patch bump

Each version also has a hash, a SHA-256 digest of the canonical representation of the schema, records, and file references. Two versions with the same hash have identical content.

Record

A record is a flat JSON object with an id, a type, and a data payload. Records are the rows of your data.

{
  "id": "pub-001",
  "type": "Publication",
  "data": {
    "title": "The Structure of Scientific Revolutions",
    "doi": "10.1234/example",
    "authors": ["author-001", "author-002"],
    "pdf": { "$file": "sha256:a1b2c3..." }
  }
}

Records are content-addressed: each record is identified by the SHA-256 hash of its canonical JSON ({"id":...,"type":...,"data":...}). This means:

• The same record appearing in multiple collections is stored only once.
• Pushing a new version only transfers records the server doesn't already have (via the negotiate protocol).
• Any record can be traced back to every collection and version that includes it (provenance).

Relationships between records are expressed as ID references (just strings). There are no joins, no foreign keys. An LLM or application can resolve references by reading the schema and records together.

Records are validated against the schema on push. If a record contains fields not defined in the schema, the push is rejected with a 422 listing the extra fields. Set strip_unknown_fields to accept stripping them automatically.

Binary data is referenced via {"$file": "sha256:..."}, a pointer to a content-addressed file in the registry. The wire format for records is JSONL, one record per line, independently hashable and streamable.

File

A file is a binary blob (PDF, image, dataset, anything) stored by its SHA-256 hash. Files are content-addressed: the same bytes always produce the same hash, so identical files are stored only once regardless of how many records reference them.

Files are uploaded before pushing a version. When you push, the registry verifies that every $file reference in your records points to an existing file.

Accounts

Underlay has two account types:

• Users: individual accounts with email/password login
• Organizations: group accounts with members who have roles (owner, admin, member)

Both can own collections. API keys are scoped to an account and optionally to specific collections, with permission levels read or write (admin is not grantable through the API). A collection-scoped key is confined to those collections and is refused on account and organization endpoints.

Privacy & Visibility

Underlay supports fine-grained privacy at three levels, allowing you to store sensitive data alongside public data in the same collection.

Collection-level

A collection can be public (listed in browse, readable by anyone) or private (visible only to the owner and org members).

Type-level

Mark an entire record type as private in the schema by adding "private": true to the type definition. All records of that type are hidden from public readers, and the type is stripped from the schema response.

Field-level

Mark individual fields within a type as private by adding "private": true to the field definition. The type remains visible, but those fields are stripped from records returned to public readers.

Record-level

Mark an individual record private by setting "private": true on its manifest entry in the negotiate request — not in the record body, which is hashed as {id, type, data} and would reject the extra key:

{"id": "pub-001", "type": "Publication", "hash": "abc123…", "private": true}

The record is dropped entirely from listings, manifests, diffs, exports and the NDJSON stream for non-owners; members of the owning org still see it.

Privacy is recorded per version and must be re-declared on every push. The flag lives on that version’s reference to the record, not on the record itself — so omitting it means public, and re-pushing a record without the flag republishes it. Read the current flags back from GET .../versions/:semver/manifest, which echoes private on the entries that carry it.

Redaction is forward-only: marking a record private in v2 hides it in v2 only. v1 is immutable and still serves it. Because file access resolves across every ready version, a file referenced publicly in v1 also stays downloadable after the referencing record is redacted in v2.

Private content is excluded from the public hash (used for verifying publicly-visible content) but included in the private hash (used by owners for full integrity verification).