FDS sites should be able to act as a reconciliation source for external tools. Recogito Studio already consumes any Reconciliation Service API endpoint as a tag source (via its reconciliation plugin), and FairCopy Editor can do the same. FairData has a reconciliation API; FairData Sites does not yet. A site should be able to expose what it publishes, at its own URL, without requiring consumers to know which backend serves the responses.
Model to follow. FairData's endpoint in core-data-connector implements the Reconciliation Service API 0.2 (the OpenRefine protocol): GET/POST /core_data/reconcile/projects/:id serves the manifest and answers batch queries, and /view/:record_id redirects to the record. It queries not the main search collections but a dedicated reconciliation Typesense collection built from per-model projections, and it is enabled per project by reconciliation credentials. See config/routes/reconcile.rb, app/controllers/core_data_connector/reconcile/, lib/typesense/reconcile.rb, and the README's Reconciliation API section in that repo.
Proposed shape for server-rendered sites. An /api/reconcile route in src/pages/api/, next to the existing model proxy routes: serve the manifest and answer batch queries against the site's configured Typesense index, using the search-only key that config.json already contains. Consumers call cross-origin, so the route needs CORS headers.
Static builds: a prebuilt index plus a delivery layer. A static bundle has no server, and reconciliation consumers are external applications, so client-side code cannot serve them. Instead:
- The static pipeline produces a reconciliation index from the record snapshot: id, name, aliases, type, and a short description for each record. The planned static search index is built the same way, and so is the live endpoint's collection — a purpose-built index, separate from search. The reconciliation index is an order of magnitude smaller than the search index and needs no facet reconstruction.
- The router in front of the static bundles implements the endpoint. The manifest is a static document. Queries run in the router against the index, with fuzzy name matching and a type filter. The router needs its own scoring function; its scores will not match Typesense's
text_match, and the spec requires only that a score be present.
- The router serves the endpoint at the durable URI (
{slug}.fairdata.site) even when a site's pages are hosted elsewhere and reached by redirect: it keeps the index, so it can still answer reconciliation queries.
Why build a site-level index rather than point consumers at the FairData endpoint: the index outlives the live backend, as a static bundle should; it matches exactly what the site publishes (multi-project sites, result_filtering exclusions), which a per-project backend endpoint cannot; and consumers configure an authority source by its URL, so the site URL is the natural identity for the service.
Commitment. For static sites we commit to durable DNS; this feature does not extend that commitment. The reconciliation index is part of the static bundle, like the other build artifacts, so anyone who hosts the bundle can run the endpoint (we should publish the router's reconcile module to make that practical). The live endpoint at {slug}.fairdata.site belongs to the delivery layer, not to the bundle: a bundle copied to another host serves pages but not the reconciliation API, and dhtools fds doctor should classify the feature accordingly.
Lifecycle.
- While a site is routed — hosted by us, or hosted elsewhere and reached by redirect — the router answers from the most recent index. Rebuilding the index on publish keeps it limited to what the site currently publishes; removed records drop out.
- When a site is retired and no longer routed, the endpoint returns
410 Gone at the manifest URL. We should not return matches whose /view URLs no longer resolve.
Sequence. We should build this after the other static-build features. This index and the static search index are projections of the same record snapshot, and the search validation effort, which comes first, will prove that projection pipeline.
The reconciliation API itself is documented only in the connector README (the studio-docs "reconciliation" page covers External Web Authorities, a different feature).
FDS sites should be able to act as a reconciliation source for external tools. Recogito Studio already consumes any Reconciliation Service API endpoint as a tag source (via its reconciliation plugin), and FairCopy Editor can do the same. FairData has a reconciliation API; FairData Sites does not yet. A site should be able to expose what it publishes, at its own URL, without requiring consumers to know which backend serves the responses.
Model to follow. FairData's endpoint in core-data-connector implements the Reconciliation Service API 0.2 (the OpenRefine protocol):
GET/POST /core_data/reconcile/projects/:idserves the manifest and answers batch queries, and/view/:record_idredirects to the record. It queries not the main search collections but a dedicated reconciliation Typesense collection built from per-model projections, and it is enabled per project by reconciliation credentials. Seeconfig/routes/reconcile.rb,app/controllers/core_data_connector/reconcile/,lib/typesense/reconcile.rb, and the README's Reconciliation API section in that repo.Proposed shape for server-rendered sites. An
/api/reconcileroute insrc/pages/api/, next to the existing model proxy routes: serve the manifest and answer batch queries against the site's configured Typesense index, using the search-only key thatconfig.jsonalready contains. Consumers call cross-origin, so the route needs CORS headers.Static builds: a prebuilt index plus a delivery layer. A static bundle has no server, and reconciliation consumers are external applications, so client-side code cannot serve them. Instead:
text_match, and the spec requires only that a score be present.{slug}.fairdata.site) even when a site's pages are hosted elsewhere and reached by redirect: it keeps the index, so it can still answer reconciliation queries.Why build a site-level index rather than point consumers at the FairData endpoint: the index outlives the live backend, as a static bundle should; it matches exactly what the site publishes (multi-project sites,
result_filteringexclusions), which a per-project backend endpoint cannot; and consumers configure an authority source by its URL, so the site URL is the natural identity for the service.Commitment. For static sites we commit to durable DNS; this feature does not extend that commitment. The reconciliation index is part of the static bundle, like the other build artifacts, so anyone who hosts the bundle can run the endpoint (we should publish the router's reconcile module to make that practical). The live endpoint at
{slug}.fairdata.sitebelongs to the delivery layer, not to the bundle: a bundle copied to another host serves pages but not the reconciliation API, anddhtools fds doctorshould classify the feature accordingly.Lifecycle.
410 Goneat the manifest URL. We should not return matches whose/viewURLs no longer resolve.Sequence. We should build this after the other static-build features. This index and the static search index are projections of the same record snapshot, and the search validation effort, which comes first, will prove that projection pipeline.
The reconciliation API itself is documented only in the connector README (the studio-docs "reconciliation" page covers External Web Authorities, a different feature).