Return each stored document's blob references from SyncQueries

docs/files-guide.md

@@ -100,6 +100,11 @@ a client older than files says nothing and is respected, but an application that
 forgets to answer for one entity type says "references nothing", and from the server the two are
 identical.
 
+A document read through `module.queries` carries the files it references in `StoredDocument.blobs`.
+A host that copies documents out and pushes them back later — a backup and its restore — sends that
+list as the `blobs` of each upsert. The files have to be registered in the collection before the push
+that names them, or the group is refused with `BLOB_MISSING`.
+
 ## Client
 
 Three things: say where the bytes live, say which files a document points at, and hand the engine a

docs/server-guide.md

@@ -293,7 +293,15 @@ public interface SyncQueries {
 ```
 
 This is the read side for host screens and reports. It reflects the same rows the clients
-synchronise, without going through the protocol.
+synchronise, without going through the protocol. Deleted entities are not returned.
+
+Each `StoredDocument` carries, besides the body, the `blobs` it currently references, ordered by
+identifier — the set the clients declared, which the module stores and never derives from the body.
+A page reads them for all of its documents in one extra statement, inside the same transaction as
+the bodies. If you copy documents out — a backup, an export you will import again — keep that list
+with each one and send it back as the `blobs` of the `PushOperation.Upsert` that restores it: a
+document pushed without it references nothing, and its files are collected once the retention window
+passes.
 
 ## Erasing a scope
 

docs/sync-files-design.md

@@ -1196,15 +1196,24 @@ cursor, because every page of one snapshot has to describe the same moment; this
 promise — it is an administrative view of a table that is still being written — so the position is
 the whole of it.
 
+`StoredDocument`, from `document` and `documents`, carries `blobs`: the entity's current rows in
+`sync.blob_refs`, ordered by blob identifier, empty when there are none. A page reads them for all of
+its documents in one statement, filtered by collection, entity type and the page's identifiers —
+the leading columns of `pk_blob_refs` — inside the transaction that reads the bodies. The host needs
+them because references are a declaration rather than something derivable from the body: a host
+that copies documents out and pushes them back has to declare them again, or the restored documents
+reference nothing and their files are collected after the retention window.
+
 ## 13. Where the tests are
 
 The mechanism here has three seams that nothing above them can check, and each has a suite of its
 own rather than being covered incidentally.
 
 `server-sdk` tests the module against real PostgreSQL and a fake bucket: the schema and its cascades
 (`BlobSchemaTest`), registration, verification and the once-per-file acknowledgement
-(`BlobServiceTest`), reference bookkeeping inside the counter lock (`BlobReferenceTest`), and the
-collector, the promotion sweep and purge (`BlobMaintenanceTest`).
+(`BlobServiceTest`), reference bookkeeping inside the counter lock (`BlobReferenceTest`), the
+references the host reads back with each document, and the statement count of a page
+(`DocumentBlobsTest`), and the collector, the promotion sweep and purge (`BlobMaintenanceTest`).
 
 `client-sdk` tests the two halves that are invisible from outside: the queries that decide what is
 work (`SyncBlobDaoTest`), reconciliation, which produces only rows (`BlobReconciliationTest`), and

docs/sync-server-design.md

@@ -106,6 +106,9 @@ interface SyncQueries {
 }
 ```
 
+A `StoredDocument` is the body, its version and when it was last written, and the blobs it currently
+references (see [sync-files-design.md §12.1](sync-files-design.md#121-what-the-host-can-read-back)).
+
 And the removals the host schedules or orders, which are not protocol operations at all:
 
 ```kotlin

server-sdk/src/main/kotlin/dev/voir/reflector/sync/server/StoredDocument.kt

@@ -1,5 +1,6 @@
 package dev.voir.reflector.sync.server
 
+import dev.voir.reflector.sync.protocol.BlobId
 import dev.voir.reflector.sync.protocol.EntityId
 import dev.voir.reflector.sync.protocol.EntityType
 import dev.voir.reflector.sync.protocol.EntityVersion
@@ -16,11 +17,18 @@ import kotlin.time.Instant
  * @property data Stored document. The module treats it as opaque JSON: it stores bodies but models
  *   nothing about them.
  * @property updatedAt When the document was last written.
+ * @property blobs Blobs the document currently references, ordered by identifier; empty when it
+ *   references none. This is the set the last write that declared one left behind — an upsert that
+ *   declared nothing keeps the previous set — because the module never derives references from the
+ *   document itself. A host that copies documents elsewhere and later pushes them back has to
+ *   declare this set with them: a document re-created without a declaration references nothing,
+ *   and the files it names are collected once the retention window passes.
  */
 public data class StoredDocument(
     public val entityType: EntityType,
     public val entityId: EntityId,
     public val version: EntityVersion,
     public val data: JsonObject,
     public val updatedAt: Instant,
+    public val blobs: List<BlobId> = emptyList(),
 )

server-sdk/src/main/kotlin/dev/voir/reflector/sync/server/SyncQueries.kt

@@ -16,7 +16,7 @@ import dev.voir.reflector.sync.protocol.ScopeId
  */
 public interface SyncQueries {
     /**
-     * Reads one stored document.
+     * Reads one stored document, together with the blobs it currently references.
      *
      * @param scope Authorised scope the collection belongs to.
      * @param collection Collection the entity belongs to.
@@ -32,7 +32,10 @@ public interface SyncQueries {
     ): StoredDocument?
 
     /**
-     * Reads a page of stored documents of one type.
+     * Reads a page of stored documents of one type, each with the blobs it currently references.
+     *
+     * Every document of the page and its references are read in one transaction, so a document's
+     * [StoredDocument.blobs] describe the same moment as its body.
      *
      * @param scope Authorised scope the collection belongs to.
      * @param collection Collection to read.

server-sdk/src/main/kotlin/dev/voir/reflector/sync/server/postgres/BlobReferences.kt

@@ -1,6 +1,9 @@
 package dev.voir.reflector.sync.server.postgres
 
 import dev.voir.reflector.sync.protocol.BlobId
+import dev.voir.reflector.sync.protocol.EntityId
+import dev.voir.reflector.sync.protocol.EntityType
+import org.jetbrains.exposed.v1.core.SortOrder
 import org.jetbrains.exposed.v1.core.and
 import org.jetbrains.exposed.v1.core.eq
 import org.jetbrains.exposed.v1.core.exists
@@ -17,7 +20,8 @@ import kotlin.time.Instant
 import kotlin.uuid.Uuid
 
 /**
- * What each entity's document points at, maintained inside the transaction that writes the batch.
+ * What each entity's document points at, maintained inside the transaction that writes the batch
+ * and read back for the host through [SyncQueries][dev.voir.reflector.sync.server.SyncQueries].
  *
  * References are state-based like everything else in this protocol: a client sends the whole set it
  * declares, and that set replaces what was stored. There is no "reference added" or "reference
@@ -60,6 +64,41 @@ internal class BlobReferences {
         return declared - known
     }
 
+    /**
+     * Reads the blobs each of the given entities of one type currently points at.
+     *
+     * One statement for the whole set, so that a page of documents costs one extra query rather
+     * than one per document. It is served by the primary key, whose leading columns are exactly the
+     * filter and whose trailing ones are the order.
+     *
+     * @param collectionRowId Row identifier of the collection.
+     * @param entityType Type every one of [entityIds] has.
+     * @param entityIds Entities to read the references of.
+     * @return Each entity's blobs ordered by identifier; an entity that points at nothing is absent.
+     */
+    fun of(
+        collectionRowId: Uuid,
+        entityType: EntityType,
+        entityIds: Collection<EntityId>,
+    ): Map<EntityId, List<BlobId>> {
+        if (entityIds.isEmpty()) {
+            return emptyMap()
+        }
+        return BlobRefsTable
+            .select(BlobRefsTable.entityId, BlobRefsTable.blobId)
+            .where {
+                (BlobRefsTable.collection eq collectionRowId) and
+                    (BlobRefsTable.entityType eq entityType.value) and
+                    (BlobRefsTable.entityId inList entityIds.map { it.value })
+            }.orderBy(BlobRefsTable.entityId to SortOrder.ASC, BlobRefsTable.blobId to SortOrder.ASC)
+            // Materialised first: `Query` has a `groupBy` of its own, which is SQL's.
+            .toList()
+            .groupBy(
+                keySelector = { EntityId(it[BlobRefsTable.entityId]) },
+                valueTransform = { BlobId(it[BlobRefsTable.blobId]) },
+            )
+    }
+
     /**
      * Replaces the references of the entities a batch wrote, and settles what became garbage.
      *

server-sdk/src/main/kotlin/dev/voir/reflector/sync/server/postgres/PostgresSyncQueries.kt

@@ -12,6 +12,7 @@ import dev.voir.reflector.sync.server.StoredBlob
 import dev.voir.reflector.sync.server.StoredDocument
 import dev.voir.reflector.sync.server.SyncConfig
 import dev.voir.reflector.sync.server.SyncQueries
+import org.jetbrains.exposed.v1.core.ResultRow
 import org.jetbrains.exposed.v1.core.SortOrder
 import org.jetbrains.exposed.v1.core.and
 import org.jetbrains.exposed.v1.core.eq
@@ -38,9 +39,15 @@ internal class PostgresSyncQueries(
     private val config: SyncConfig,
     private val clock: Clock,
 ) : SyncQueries {
+    /** Resolves a scope's collection to its row; a read never creates one. */
     private val collections = CollectionRows(config, clock)
+
+    /** Reads what the module knows about files. */
     private val blobs = BlobRows()
 
+    /** Reads which files each document points at. */
+    private val references = BlobReferences()
+
     override fun document(
         scope: ScopeId,
         collection: CollectionId,
@@ -49,15 +56,16 @@ internal class PostgresSyncQueries(
     ): StoredDocument? =
         transaction(database) {
             val row = collections.find(scope, collection) ?: return@transaction null
-            EntitiesTable
-                .selectAll()
-                .where {
-                    (EntitiesTable.collection eq row.id) and
-                        (EntitiesTable.entityType eq type.value) and
-                        (EntitiesTable.entityId eq id.value) and
-                        (EntitiesTable.isDeleted eq false)
-                }.singleOrNull()
-                ?.let(::toDocument)
+            val entity =
+                EntitiesTable
+                    .selectAll()
+                    .where {
+                        (EntitiesTable.collection eq row.id) and
+                            (EntitiesTable.entityType eq type.value) and
+                            (EntitiesTable.entityId eq id.value) and
+                            (EntitiesTable.isDeleted eq false)
+                    }.singleOrNull() ?: return@transaction null
+            toDocument(entity, references.of(row.id, type, listOf(id))[id].orEmpty())
         }
 
     override fun documents(
@@ -89,7 +97,11 @@ internal class PostgresSyncQueries(
                     .toList()
 
             val hasMore = rows.size > capped
-            val documents = rows.take(capped).map(::toDocument)
+            val found = rows.take(capped)
+            // One read for the references of the whole page, in the same transaction, so that each
+            // document's set belongs to the same moment as its body.
+            val blobsOf = references.of(row.id, type, found.map { EntityId(it[EntitiesTable.entityId]) })
+            val documents = found.map { toDocument(it, blobsOf[EntityId(it[EntitiesTable.entityId])].orEmpty()) }
             DocumentPage(
                 documents = documents,
                 nextPage =
@@ -144,12 +156,23 @@ internal class PostgresSyncQueries(
             )
         }
 
-    private fun toDocument(row: org.jetbrains.exposed.v1.core.ResultRow): StoredDocument =
+    /**
+     * Maps a live entity row to what the host is handed.
+     *
+     * @param row Row of [EntitiesTable] for an entity that is not deleted.
+     * @param blobs Blobs the entity currently references, already ordered by identifier.
+     * @return Stored document.
+     */
+    private fun toDocument(
+        row: ResultRow,
+        blobs: List<BlobId>,
+    ): StoredDocument =
         StoredDocument(
             entityType = EntityType(row[EntitiesTable.entityType]),
             entityId = EntityId(row[EntitiesTable.entityId]),
             version = SyncSequences.version(row[EntitiesTable.version]),
             data = checkNotNull(row[EntitiesTable.data]) { "a live entity must have a document" },
             updatedAt = row[EntitiesTable.updatedAt],
+            blobs = blobs,
         )
 }