diff --git a/CHANGELOG.md b/CHANGELOG.md index 183c2e0..4d3afab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -141,6 +141,22 @@ Each was removed before the first tag; none was in a tagged release. - Two approvals recorded at once both count: the request is re-read under a row lock before an approval is added. - A REST `cursor` the listing did not hand out answers `422` instead of the first page. +- An `Idempotency-Key` retry sent while the first request is still running answers `409` instead + of running the write a second time. (SP-04) +- An actor without `approve` saw an empty page of change requests when other people's newer + requests filled the page; their own are now filtered in the query. (SP-05) +- `sync-definitions` read each definition's row and category one query at a time; it now reads + them once per store. (SP-06) +- Every write (set, forget, lock, unlock, and so each row of an import) read all the rows of its + scope, twice, to check one; it now reads that row only. (SP-07) +- The JSON driver lost updates under concurrent writers. A write now holds an exclusive `flock` + on a sidecar `.lock` (local disks) for its read-modify-write, and is version-checked like + the database layer, so a stale writer gets `ConcurrentModification`. (SP-09) +- `laranail::settings.export` and `POST /stores/{store}/export` stream the document, 500 rows per + query, instead of building the whole store in memory; the output bytes are unchanged. + The `StreamExport` action is new; `export()` still returns the `ExportDocument`. (SP-10) +- Two overlapping `sync-definitions` runs no longer fail on each other's rows: the second waits up + to five seconds for a cache lock, then reports `busy` and changes nothing. (SP-11) ### Security @@ -159,3 +175,16 @@ Each was removed before the first tag; none was in a tagged release. and returned every store's entries. - A persisted layer may not redefine a layer the configuration file defines, saved or read back; it could otherwise move a configured store onto another connection, table or driver. +- A value stored encrypted is sensitive whatever its definition now says. Redefining or deleting a + runtime definition (`manage_definitions`) used to make the stored secret readable in plaintext + by any actor with `view`, over REST and PHP. Reads now also check the stored row: without + `view_sensitive`, `get()` throws `AccessDenied`, `explain()` and REST redact, listings omit it. + `Lookup::isEncrypted()` is new. (SP-01) +- The REST API refuses a request whose actor resolver returns the system actor (`401` + `actor-required`), which used to skip every check. (SP-02) +- An API token counts as its user in approvals: it can no longer approve its user's own change + request, and a user and their token are one approval towards `min_approvals`. Approvals + recorded under `token:` before this release are read as their user's. (SP-03) +- Idempotent replays are keyed by tenant as well as actor: in `column` or `database` tenancy, + `user:1` of one tenant reusing a key used by `user:1` of another received that tenant's stored + response. (SP-08) diff --git a/docs/tools/access-control.md b/docs/tools/access-control.md index aabd060..41cc551 100644 --- a/docs/tools/access-control.md +++ b/docs/tools/access-control.md @@ -107,7 +107,7 @@ With `locks.overridable` on, an actor allowed `override_lock` may `set` or `forg ## Sensitive settings -Mark a definition `sensitive()` (or `'sensitive' => true` in config); `encrypted()` implies it. Reading one needs `view_sensitive` as well as `view`, and `explain()` redacts its value: `Explanation::isRedacted()` is true, `Lookup::value()` throws, and the layer and steps stay visible. +Mark a definition `sensitive()` (or `'sensitive' => true` in config); `encrypted()` implies it. Reading one needs `view_sensitive` as well as `view`, and `explain()` redacts its value: `Explanation::isRedacted()` is true, `Lookup::value()` throws, and the layer and steps stay visible. A value stored encrypted stays sensitive after its definition is changed or removed: the stored row decides as well as the definition (`Lookup::isEncrypted()`). ## Step-up verification diff --git a/docs/tools/approvals.md b/docs/tools/approvals.md index 1735272..a3b4684 100644 --- a/docs/tools/approvals.md +++ b/docs/tools/approvals.md @@ -50,7 +50,7 @@ $settings->as($editor)->approvals()->cancel($id); // the requester only | Days before an open request expires | 7 | `approvals.expire_after` (null: never) | | Days a decided request is kept | 90 | `approvals.keep_days` (null: for ever) | -Approving and rejecting need `approve` on the setting. Listing needs `approve` to see everyone's requests; an actor without it sees only their own. Trusted code without an actor sees and decides everything. +An API token acts for its user: a user and their token are one person, so a token cannot approve its user's request, and the two count as one approval towards `min_approvals`. Approving and rejecting need `approve` on the setting. Listing needs `approve` to see everyone's requests; an actor without it sees only their own. Trusted code without an actor sees and decides everything. ## Applying diff --git a/docs/tools/database.md b/docs/tools/database.md index a5017d2..ceeee39 100644 --- a/docs/tools/database.md +++ b/docs/tools/database.md @@ -79,7 +79,7 @@ table has the `deleted_at` column, so the option can be turned on at any time. `laranail::settings.sync-definitions` (also run by `install` once the table exists, and by `doctor --fix`) copies code and config definitions in as read-only `is_system` rows, so UIs can -list them; system rows nothing defines any more are removed. +list them; system rows nothing defines any more are removed. One run at a time: a run that finds another holding the lock waits up to five seconds, then changes nothing and says so (the report's `busy`). Other rows are definitions edited at runtime. With `database` in `definitions.sources` they answer for keys no code or config definition covers, so a row can never loosen a coded setting, and a row diff --git a/docs/tools/drivers.md b/docs/tools/drivers.md index 39bf2ca..978e9f5 100644 --- a/docs/tools/drivers.md +++ b/docs/tools/drivers.md @@ -21,7 +21,7 @@ Described in [Database](database.md): one table per store, version-checked write ## The JSON driver -One file per store. Writes go to a temporary file that is then moved over the real one, which is an atomic rename on local disks. There is no cross-process lock, so two processes writing at the same moment can lose one update; the runtime layer is meant for the install window, not for heavy concurrent writing. A public disk is refused. +One file per store. Writes go to a temporary file that is then moved over the real one, which is an atomic rename on local disks. On a local disk each write holds an exclusive `flock` on a sidecar `.lock` for its read-modify-write, and is version-checked like the database layer: a writer whose base version is stale gets `ConcurrentModification` (a plain `set()` retries once), so no update is lost. Reads take no lock; the rename means a reader never sees half a file. A disk that is not local has no lock, so only the version check guards it; the runtime layer is still meant for the install window, not for heavy concurrent writing. A public disk is refused. ## The PHP driver diff --git a/docs/tools/import-export.md b/docs/tools/import-export.md index 194964d..17e4cb9 100644 --- a/docs/tools/import-export.md +++ b/docs/tools/import-export.md @@ -49,6 +49,8 @@ php artisan laranail::settings.import storage/settings.json --apply php artisan laranail::settings.import storage/settings.json --apply --trusted ``` +The console command and `POST /stores/{store}/export` stream the document, reading 500 rows at a time, so a large store is never held in memory; the bytes are the ones `export()->toJson()` gives. In PHP, the `StreamExport` action (`handle($request, $stream, $actor)`) writes to any stream; `export()` still returns the whole `ExportDocument`. + An import creates missing categories first, by slug, parents before children, then writes each setting through the normal write pipeline as the actor: validated and cast under the destination's definition, encrypted, with history and one audit entry per key. A setting that needs approval opens a change request instead (the report lists them). Everything runs in one transaction, so one failed write leaves the store as it was. Redacted values, owners the store does not accept and global values for a dedicated store are skipped, each with its reason in the report. Locks and system categories in a document are applied only by a trusted import (`--trusted` on the console); the PHP `import()` and the REST API never apply them. `import` needs step-up verification by default (`verification.abilities`). diff --git a/docs/tools/rest-api-reference.md b/docs/tools/rest-api-reference.md index 502948a..a100e5e 100644 --- a/docs/tools/rest-api-reference.md +++ b/docs/tools/rest-api-reference.md @@ -14,7 +14,7 @@ Paths are under `/{http.prefix}/v1`, by default `/api/settings/v1`; set `http.pr | Locales | `?locale=fr` on settings routes; the write routes also take it. | | Bodies | JSON. Every write may send `"reason"` (recorded in the audit trail and history). | | ETags | setting responses carry `ETag`. Send it back as `If-Match` on `PUT` or `DELETE`: a mismatch is `412`, and the write or delete applies only at that version, so a change made after the check is also `412`; `*` matches any version. | -| Idempotency | every non-`GET` route accepts `Idempotency-Key`; a retry with the same key, actor, host, method and path within `http.idempotency.ttl` (24 hours) is answered from cache with `Idempotent-Replayed: true`. The same key with a different body answers `422` (`http-422`) and applies nothing. | +| Idempotency | every non-`GET` route accepts `Idempotency-Key`; a retry with the same key, tenant, actor, host, method and path within `http.idempotency.ttl` (24 hours) is answered from cache with `Idempotent-Replayed: true`. The same key with a different body answers `422` (`http-422`) and applies nothing; the same key while its first request still runs answers `409` (`http-409`). | | Pagination | lists answer `{"data": [...], "meta": {"next_cursor": "…" or null}}`; pass `?cursor=` and `?limit=` (1 to 200, default `http.page_size`). Cursors are opaque; one the listing did not hand out answers `422` (`validation`). | | Visibility | values the actor may not view are left out of lists; a single setting answers `"redacted": true` without `value`. | | Approvals | a write to a setting that needs approval answers `202` with the change request instead of the setting. | diff --git a/docs/tools/rest-api.md b/docs/tools/rest-api.md index c44a567..6e07241 100644 --- a/docs/tools/rest-api.md +++ b/docs/tools/rest-api.md @@ -15,7 +15,7 @@ A JSON API over every capability of the PHP API, in this package and off by defa ], ``` -No route is registered until `enabled` is true. The routes are declared in `routes/api.php`, and each controller method takes the action it runs as a parameter, so under Octane no request's actor, tenant or stores reach the next one. Routes are named `laranail-settings.api.` (the owner form of a settings route adds `owners.`), run through `http.middleware` and the `laranail-settings` rate limiter, and every write runs through the idempotency middleware. The host's middleware authenticates; the configured actor resolver turns the request into an [actor](access-control.md) on the `rest` channel, so with the `sanctum` resolver a token's `laranail-settings:` abilities are a ceiling on what it may do. A request nobody is authenticated for is refused (`401`); the API never runs as trusted code. +No route is registered until `enabled` is true. The routes are declared in `routes/api.php`, and each controller method takes the action it runs as a parameter, so under Octane no request's actor, tenant or stores reach the next one. Routes are named `laranail-settings.api.` (the owner form of a settings route adds `owners.`), run through `http.middleware` and the `laranail-settings` rate limiter, and every write runs through the idempotency middleware. The host's middleware authenticates; the configured actor resolver turns the request into an [actor](access-control.md) on the `rest` channel, so with the `sanctum` resolver a token's `laranail-settings:` abilities are a ceiling on what it may do. A request nobody is authenticated for is refused (`401`), and so is one the resolver maps to the system actor: the API never runs as trusted code. ## Endpoints @@ -69,7 +69,7 @@ Idempotency-Key: 6c0d1e8e-… - **Lists** are `{"data": [...], "meta": {"next_cursor": …}}`; pass `?cursor=` for the next page and `?limit=` (1-200). A cursor the list did not hand out answers `422`. - **Locales**: `?locale=fr`. - **Values the actor may not view** are left out, not masked: a list skips the key, a single setting answers `"redacted": true` without `value`. -- **Idempotency.** A write with an `Idempotency-Key` header is answered from cache when retried (same actor, host, method, path and key) for `http.idempotency.ttl` seconds (24 hours), with `Idempotent-Replayed: true`. A retry with the same key and a different body is refused with `422` and applies nothing. The stored response is encrypted with the application key, since it can hold an encrypted setting's value; without an application key, or without an actor, nothing is stored. +- **Idempotency.** A write with an `Idempotency-Key` header is answered from cache when retried (same tenant, actor, host, method, path and key) for `http.idempotency.ttl` seconds (24 hours), with `Idempotent-Replayed: true`. A retry with the same key and a different body is refused with `422` and applies nothing; one sent while the first request with that key is still running is refused with `409`, so the write runs once (the idempotency cache store must support locks for this, as Laravel's `redis`, `database`, `file` and `array` stores do). The stored response is encrypted with the application key, since it can hold an encrypted setting's value; without an application key, or without an actor, nothing is stored. ## Errors diff --git a/resources/lang/en/settings.php b/resources/lang/en/settings.php index be4fe3b..f0acab9 100644 --- a/resources/lang/en/settings.php +++ b/resources/lang/en/settings.php @@ -76,6 +76,7 @@ 'cache_off' => 'Caching is off; nothing to reset.', 'definitions_synced' => 'Synced :synced definition(s); removed :removed stale system row(s).', 'definitions_skipped' => 'Store [:store] skipped: no active database layer, or the definitions table is off.', + 'definitions_busy' => 'Another sync-definitions run holds the lock; this one changed nothing. Run it again once that one finishes.', 'definitions_conflict' => '[:key] is defined in code and edited in the table; code wins, the row was left alone.', 'diff_none' => 'No differences (:unchanged setting(s) equal).', 'diff_summary' => ':added to add, :removed only in the store, :changed changed, :unknown not comparable (redacted).', diff --git a/src/Actions/Approvals/ApproveChange.php b/src/Actions/Approvals/ApproveChange.php index 84cad01..e5d29d9 100644 --- a/src/Actions/Approvals/ApproveChange.php +++ b/src/Actions/Approvals/ApproveChange.php @@ -78,7 +78,7 @@ public function handle(ChangeDecision $decision, ?Actor $actor = null): ChangeRe // Counted under a row lock, so two approvals recorded at once both count. [$approvals, $remaining] = $database->transaction(function () use ($database, $store, $request, $subject, $actor, $decision, $now): array { $current = $database->attempt($store->name, static fn (): ChangeRequest => SettingChangeRequest::locked($database->connectionName(), $store->name, $request->id)); - $approvals = array_values(array_unique([...$current->approvals, ChangeRequest::actorKey($actor ?? Actor::system())])); + $approvals = array_values(array_unique([...array_map(ChangeRequest::normaliseKey(...), $current->approvals), ChangeRequest::actorKey($actor ?? Actor::system())])); $remaining = max(0, $this->policy->minApprovals - count($approvals)); if ($remaining > 0) { diff --git a/src/Actions/Approvals/QueryChanges.php b/src/Actions/Approvals/QueryChanges.php index 6707ffb..e0cceac 100644 --- a/src/Actions/Approvals/QueryChanges.php +++ b/src/Actions/Approvals/QueryChanges.php @@ -34,21 +34,26 @@ public function handle(ChangeFilter $filter, ?Actor $actor = null, int $limit = $this->access->assertActorNotRequired(Ability::Approve, $subject); } + // An actor who may not approve sees only their own requests: filtered in the query, so a + // page is never filled with other people's requests and then emptied. + $own = $actor instanceof Actor && ! $actor->isSystem() && ! $this->access->allows($actor, Ability::Approve, $subject) ? $actor : null; $database = $store->changeRequestsDatabase(); - $requests = $database->attempt($store->name, static fn (): array => SettingChangeRequest::on($database->connectionName()) - ->matching($filter) - ->when($after !== null, static fn (Builder $query): Builder => $query->where('id', '<', $after)) - ->limit($limit) - ->get() - ->map(static fn (SettingChangeRequest $request): ChangeRequest => $request->toRequest()) - ->values() - ->all()); - $requests = array_values($requests); - - if (! $actor instanceof Actor || $actor->isSystem() || $this->access->allows($actor, Ability::Approve, $subject)) { - return $requests; - } - - return array_values(array_filter($requests, static fn (ChangeRequest $request): bool => $request->isRequestedBy($actor))); + $requests = $database->attempt($store->name, static function () use ($database, $filter, $own, $after, $limit): array { + $query = SettingChangeRequest::on($database->connectionName())->matching($filter); + + if ($own instanceof Actor) { + SettingChangeRequest::requestedBy($query, $own); + } + + return $query + ->when($after !== null, static fn (Builder $query): Builder => $query->where('id', '<', $after)) + ->limit($limit) + ->get() + ->map(static fn (SettingChangeRequest $request): ChangeRequest => $request->toRequest()) + ->values() + ->all(); + }); + + return array_values(array_filter($requests, static fn (ChangeRequest $request): bool => ! $own instanceof Actor || $request->isRequestedBy($own))); } } diff --git a/src/Actions/SyncDefinitions.php b/src/Actions/SyncDefinitions.php index cb317e5..06c07cc 100644 --- a/src/Actions/SyncDefinitions.php +++ b/src/Actions/SyncDefinitions.php @@ -4,16 +4,21 @@ namespace Simtabi\Laranail\Settings\Actions; +use Illuminate\Contracts\Cache\Lock; use Simtabi\Laranail\Settings\Data\Actor; use Illuminate\Contracts\Events\Dispatcher; use Simtabi\Laranail\Settings\Data\Subject; +use Illuminate\Contracts\Cache\LockProvider; use Simtabi\Laranail\Settings\Enums\Ability; use Simtabi\Laranail\Settings\Data\StoreName; use Simtabi\Laranail\Settings\Data\Definition; use Simtabi\Laranail\Settings\Enums\AuditAction; use Simtabi\Laranail\Settings\Enums\AuditOutcome; use Simtabi\Laranail\Settings\Support\ValueEncoder; +use Illuminate\Contracts\Cache\LockTimeoutException; use Simtabi\Laranail\Settings\Drivers\DatabaseDriver; +use Simtabi\Laranail\Settings\Models\SettingCategory; +use Illuminate\Contracts\Cache\Factory as CacheFactory; use Simtabi\Laranail\Settings\Events\DefinitionsSynced; use Simtabi\Laranail\Settings\Models\SettingDefinition; use Simtabi\Laranail\Settings\Services\Access\AccessGuard; @@ -25,7 +30,10 @@ * rows whose definition is gone. Runtime rows are never touched; one sharing a key with a code * definition is reported as a conflict. * - * @phpstan-type SyncReport array{synced: int, removed: int, conflicts: list, skipped: list} + * One run at a time, under a cache lock: a run that cannot take it within `$wait` seconds does + * nothing and reports `busy`, rather than failing on the rows the other run is inserting. + * + * @phpstan-type SyncReport array{synced: int, removed: int, conflicts: list, skipped: list, busy: bool} */ final readonly class SyncDefinitions { @@ -35,6 +43,8 @@ public function __construct( private ValueEncoder $encoder, private AccessGuard $access, private Dispatcher $events, + private ?CacheFactory $caches = null, + private int $wait = 5, ) {} /** @return SyncReport */ @@ -43,7 +53,24 @@ public function handle(?Actor $actor = null): array $subject = new Subject($this->stores->default()->name); $this->access->authorize($actor, Ability::ManageDefinitions, $subject); - $report = ['synced' => 0, 'removed' => 0, 'conflicts' => [], 'skipped' => []]; + $store = $this->caches?->store()->getStore(); + $lock = $store instanceof LockProvider ? $store->lock('laranail-settings:sync-definitions', 300) : null; + + if ($lock !== null && ! $this->acquire($lock)) { + return ['synced' => 0, 'removed' => 0, 'conflicts' => [], 'skipped' => [], 'busy' => true]; + } + + try { + return $this->syncAll($subject, $actor); + } finally { + $lock?->release(); + } + } + + /** @return SyncReport */ + private function syncAll(Subject $subject, ?Actor $actor): array + { + $report = ['synced' => 0, 'removed' => 0, 'conflicts' => [], 'skipped' => [], 'busy' => false]; $byStore = []; foreach ($this->definitions->all() as $definition) { @@ -89,10 +116,13 @@ private function sync(DatabaseDriver $database, StoreName $store, array $definit $synced = 0; $conflicts = []; $keys = []; + // Every row of the store in one query, not one per definition. + $rows = SettingDefinition::on($connection)->forStore($store)->get()->keyBy('key'); + $categories = SettingCategory::on($connection)->central()->forStore($store)->pluck('id', 'slug')->all(); foreach ($definitions as $definition) { $keys[] = $definition->key->value; - $row = SettingDefinition::on($connection)->forStore($store)->where('key', $definition->key->value)->first(); + $row = $rows->get($definition->key->value); if ($row !== null && ! $row->is_system) { $conflicts[] = $store->value . '.' . $definition->key->value; @@ -101,7 +131,7 @@ private function sync(DatabaseDriver $database, StoreName $store, array $definit } $row ??= SettingDefinition::on($connection)->make(['store' => $store->value, 'key' => $definition->key->value, 'is_system' => true]); - $row->fillFromDefinition($definition, $this->encodedDefault($definition))->save(); + $row->fillFromDefinition($definition, $this->encodedDefault($definition), $categories)->save(); $synced++; } @@ -110,6 +140,16 @@ private function sync(DatabaseDriver $database, StoreName $store, array $definit return ['synced' => $synced, 'removed' => $removed, 'conflicts' => $conflicts]; } + /** Take the lock, waiting up to `$wait` seconds for a run that holds it. */ + private function acquire(Lock $lock): bool + { + try { + return $this->wait > 0 ? $lock->block($this->wait) === true : $lock->get() === true; + } catch (LockTimeoutException) { + return false; + } + } + private function encodedDefault(Definition $definition): ?string { return $definition->hasDefault ? $this->encoder->encode($definition->key, $definition->default) : null; diff --git a/src/Actions/Transfer/StreamExport.php b/src/Actions/Transfer/StreamExport.php new file mode 100644 index 0000000..d9d4ac1 --- /dev/null +++ b/src/Actions/Transfer/StreamExport.php @@ -0,0 +1,49 @@ +store->name); + + $this->access->authorize($actor, Ability::Export, $subject); + + if ($request->mode === ExportMode::Plaintext) { + $this->access->authorize($actor, Ability::ViewSensitive, $subject); + } + + $counts = $this->snapshots->exportTo($request->store, $request->mode, $stream); + $this->access->auditor()->record(AuditAction::Export, AuditOutcome::Allowed, $subject, $actor, meta: ['mode' => $request->mode->value, 'settings' => $counts['settings'], 'categories' => $counts['categories']]); + + return $counts; + } +} diff --git a/src/Console/DoctorCommand.php b/src/Console/DoctorCommand.php index a92d253..90e7918 100644 --- a/src/Console/DoctorCommand.php +++ b/src/Console/DoctorCommand.php @@ -31,7 +31,9 @@ public function handle(DiagnoseSettings $diagnose, SyncDefinitions $sync): int if ($this->option('fix') === true) { try { $report = $sync->handle(Actor::system('laranail::settings.doctor')); - $this->info(__('laranail/settings::settings.console.definitions_synced', ['synced' => $report['synced'], 'removed' => $report['removed']])); + $report['busy'] + ? $this->warn(__('laranail/settings::settings.console.definitions_busy')) + : $this->info(__('laranail/settings::settings.console.definitions_synced', ['synced' => $report['synced'], 'removed' => $report['removed']])); } catch (SettingsException $exception) { $this->error($exception->getMessage()); diff --git a/src/Console/ExportCommand.php b/src/Console/ExportCommand.php index 6fc0676..4a6e47f 100644 --- a/src/Console/ExportCommand.php +++ b/src/Console/ExportCommand.php @@ -4,11 +4,14 @@ namespace Simtabi\Laranail\Settings\Console; -use Illuminate\Filesystem\Filesystem; +use Simtabi\Laranail\Settings\Data\Actor; +use Simtabi\Laranail\Settings\Data\StoreName; use Simtabi\Laranail\Settings\Enums\ExportMode; -use Simtabi\Laranail\Settings\Contracts\Settings; use Simtabi\Laranail\Console\Tools\Commands\Command; +use Simtabi\Laranail\Settings\Data\Transfer\ExportRequest; use Simtabi\Laranail\Settings\Exceptions\SettingsException; +use Simtabi\Laranail\Settings\Actions\Transfer\StreamExport; +use Simtabi\Laranail\Settings\Services\Stores\StoreRegistry; use Simtabi\Laranail\Console\Tools\Commands\Concerns\SupportsNamespacedNames; /** @@ -28,11 +31,9 @@ final class ExportCommand extends Command protected $description = 'Export a settings store as JSON'; - public function handle(Settings $settings, Filesystem $files): int + public function handle(StoreRegistry $stores, StreamExport $export): int { $store = $this->option('store'); - $handle = $settings->asSystem('laranail::settings.export'); - $handle = is_string($store) && $store !== '' ? $handle->store($store) : $handle; $name = is_string($store) && $store !== '' ? $store : null; $mode = match (true) { $this->option('decrypt') === true => ExportMode::Plaintext, @@ -41,7 +42,7 @@ public function handle(Settings $settings, Filesystem $files): int }; try { - $document = $handle->export($mode); + $target = $name === null ? $stores->default() : $stores->get(new StoreName($name)); } catch (SettingsException $exception) { $this->error($exception->getMessage()); @@ -50,29 +51,53 @@ public function handle(Settings $settings, Filesystem $files): int if ($mode === ExportMode::Plaintext) { $confirm = $this->option('confirm'); - $typed = is_string($confirm) ? $confirm : $this->ask(__('laranail/settings::settings.console.export_confirm', ['store' => $document->store])); + $typed = is_string($confirm) ? $confirm : $this->ask(__('laranail/settings::settings.console.export_confirm', ['store' => $target->name->value])); - if ($typed !== ($name ?? $document->store)) { + if ($typed !== ($name ?? $target->name->value)) { $this->error(__('laranail/settings::settings.console.export_mismatch')); return self::FAILURE; } } - $path = $this->argument('path'); + $argument = $this->argument('path'); + $path = is_string($argument) && $argument !== '' ? $argument : null; + $toFile = $path !== null; + // Streamed 500 rows at a time; standard output goes through a temporary stream, which + // spills to disk past 2 MB rather than growing in memory. + $stream = $path !== null ? @fopen($path, 'wb') : fopen('php://temp', 'w+b'); - if (is_string($path) && $path !== '') { - $files->put($path, $document->toJson()); - $this->info(trans_choice('laranail/settings::settings.console.exported', count($document->categories), ['settings' => count($document->settings), 'store' => $document->store, 'path' => $path])); + if ($stream === false) { + $this->error(sprintf('Cannot write [%s].', $path ?? 'php://temp')); - if ($mode === ExportMode::Ciphertext) { - $this->comment(__('laranail/settings::settings.console.export_ciphertext')); + return self::FAILURE; + } + + try { + $counts = $export->handle(new ExportRequest($target, $mode), $stream, Actor::system('laranail::settings.export')); + + if (! $toFile) { + rewind($stream); + + while (! feof($stream)) { + $this->output->write((string) fread($stream, 65536)); + } } + } catch (SettingsException $exception) { + $this->error($exception->getMessage()); - return self::SUCCESS; + return self::FAILURE; + } finally { + fclose($stream); } - $this->output->write($document->toJson()); + if ($toFile) { + $this->info(trans_choice('laranail/settings::settings.console.exported', $counts['categories'], ['settings' => $counts['settings'], 'store' => $target->name->value, 'path' => $path])); + + if ($mode === ExportMode::Ciphertext) { + $this->comment(__('laranail/settings::settings.console.export_ciphertext')); + } + } return self::SUCCESS; } diff --git a/src/Console/SyncDefinitionsCommand.php b/src/Console/SyncDefinitionsCommand.php index 131c0c1..726f01e 100644 --- a/src/Console/SyncDefinitionsCommand.php +++ b/src/Console/SyncDefinitionsCommand.php @@ -32,6 +32,12 @@ public function handle(SyncDefinitions $sync): int return self::FAILURE; } + if ($report['busy']) { + $this->warn(__('laranail/settings::settings.console.definitions_busy')); + + return self::SUCCESS; + } + foreach ($report['skipped'] as $store) { $this->line(__('laranail/settings::settings.console.definitions_skipped', ['store' => $store])); } diff --git a/src/Data/ChangeRequest.php b/src/Data/ChangeRequest.php index a579725..338352f 100644 --- a/src/Data/ChangeRequest.php +++ b/src/Data/ChangeRequest.php @@ -42,15 +42,24 @@ public function __construct( public ?CarbonImmutable $createdAt = null, ) {} - /** How an actor is written in `approvals`, and compared with the requester. */ + /** + * How an actor is written in `approvals`, and compared with the requester. A token acts for + * its user, so both are one person: one approval, and never an approver of their own request. + */ public static function actorKey(Actor $actor): string { - return $actor->type->value . ':' . ($actor->alias ?? '') . ':' . ($actor->id ?? ''); + return self::identity($actor->type, $actor->alias, $actor->id); + } + + /** An `approvals` entry as {@see actorKey()} writes it now; older entries named token actors apart. */ + public static function normaliseKey(string $key): string + { + return str_starts_with($key, ActorType::Token->value . ':') ? ActorType::User->value . substr($key, strlen(ActorType::Token->value)) : $key; } public function requesterKey(): string { - return $this->requestedByType->value . ':' . ($this->requestedByAlias ?? '') . ':' . ($this->requestedById ?? ''); + return self::identity($this->requestedByType, $this->requestedByAlias, $this->requestedById); } public function isRequestedBy(Actor $actor): bool @@ -96,4 +105,9 @@ public function toArray(): array 'created_at' => $this->createdAt?->toIso8601String(), ]; } + + private static function identity(ActorType $type, ?string $alias, ?string $id): string + { + return ($type === ActorType::Token ? ActorType::User : $type)->value . ':' . ($alias ?? '') . ':' . ($id ?? ''); + } } diff --git a/src/Data/Lookup.php b/src/Data/Lookup.php index 10a3850..cf3762b 100644 --- a/src/Data/Lookup.php +++ b/src/Data/Lookup.php @@ -15,11 +15,13 @@ private function __construct( private ?string $layer, private ?Scope $scope, private bool $redacted = false, + private bool $encrypted = false, ) {} - public static function found(mixed $value, string $layer, Scope $scope): self + /** @param bool $encrypted whether the value was stored encrypted, which makes it sensitive whatever its definition says */ + public static function found(mixed $value, string $layer, Scope $scope, bool $encrypted = false): self { - return new self(true, $value, $layer, $scope); + return new self(true, $value, $layer, $scope, encrypted: $encrypted); } public static function fromDefault(mixed $value): self @@ -35,7 +37,13 @@ public static function missing(): self /** The same answer with its value withheld; presence, layer and scope stay visible. */ public function redacted(): self { - return new self($this->found, null, $this->layer, $this->scope, redacted: true); + return new self($this->found, null, $this->layer, $this->scope, redacted: true, encrypted: $this->encrypted); + } + + /** Whether the value was stored encrypted: a secret, even when no definition says so any more. */ + public function isEncrypted(): bool + { + return $this->encrypted; } public function isRedacted(): bool diff --git a/src/Data/Transfer/ExportDocument.php b/src/Data/Transfer/ExportDocument.php index e37ce52..d593eff 100644 --- a/src/Data/Transfer/ExportDocument.php +++ b/src/Data/Transfer/ExportDocument.php @@ -17,6 +17,9 @@ public const int VERSION = 1; + /** How {@see toJson()} encodes, and the streamed export with it. */ + public const int JSON_FLAGS = JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRESERVE_ZERO_FRACTION | JSON_THROW_ON_ERROR; + /** * @param list $categories * @param list $settings @@ -99,7 +102,7 @@ public function toArray(): array public function toJson(): string { - return json_encode($this->toArray(), JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRESERVE_ZERO_FRACTION | JSON_THROW_ON_ERROR) . "\n"; + return json_encode($this->toArray(), self::JSON_FLAGS) . "\n"; } /** @return array by identity */ diff --git a/src/Drivers/DatabaseDriver.php b/src/Drivers/DatabaseDriver.php index d6acae1..867ebee 100644 --- a/src/Drivers/DatabaseDriver.php +++ b/src/Drivers/DatabaseDriver.php @@ -225,6 +225,50 @@ public function trashedVersion(StoreName $store, Scope $scope, Key $key, Locale return is_numeric($version) ? (int) $version : 0; } + /** + * Hand every live record of the store (the current tenant's) to `$each`, `$chunk` rows per + * query, in the order their `scope|key|locale` identities sort, so an export never holds the + * whole store. Returns how many records were handed over. + * + * @param Closure(Record): void $each + */ + public function eachRecord(StoreName $store, Closure $each, int $chunk = 500): int + { + return $this->attempt($store, function () use ($store, $each, $chunk): int { + // `a|` sorts after `a0|`, as the identity string does; ordering by the bare column would not. + [$scope, $key] = match ($this->connection()->getDriverName()) { + 'mysql', 'mariadb' => ["concat(`scope`, '|')", "concat(`key`, '|')"], + 'sqlsrv' => ["[scope] + '|'", "[key] + '|'"], + default => ['"scope" || \'|\'', '"key" || \'|\''], + }; + $count = 0; + + $rows = $this->live($store) + ->orderByRaw($scope) + ->orderByRaw($key) + ->orderBy('locale') + ->select(self::COLUMNS) + ->lazy($chunk); + + foreach ($rows as $row) { + $each($this->record((array) $row)); + $count++; + } + + return $count; + }); + } + + /** Read one live row, without the rest of its scope: a write's precondition, not a read path. */ + public function find(StoreName $store, Scope $scope, Key $key, Locale $locale): ?Record + { + return $this->attempt($store, function () use ($store, $scope, $key, $locale): ?Record { + $row = $this->identity($this->live($store), $scope, $key, $locale)->first(self::COLUMNS); + + return $row === null ? null : $this->record((array) $row); + }); + } + /** * Read one live row under a row lock, for retrying a write after a conflict. * diff --git a/src/Drivers/JsonDriver.php b/src/Drivers/JsonDriver.php index 374d347..32a38c2 100644 --- a/src/Drivers/JsonDriver.php +++ b/src/Drivers/JsonDriver.php @@ -11,18 +11,24 @@ use Simtabi\Laranail\Settings\Data\Scope; use Simtabi\Laranail\Settings\Data\Locale; use Simtabi\Laranail\Settings\Data\Record; +use Illuminate\Filesystem\FilesystemAdapter; use Simtabi\Laranail\Settings\Data\RecordSet; use Simtabi\Laranail\Settings\Data\StoreName; use Illuminate\Contracts\Filesystem\Filesystem; use Simtabi\Laranail\Settings\Contracts\Driver; +use League\Flysystem\Local\LocalFilesystemAdapter; use Simtabi\Laranail\Settings\Exceptions\DriverFailure; use Simtabi\Laranail\Settings\Support\RecordSerializer; +use Simtabi\Laranail\Settings\Exceptions\ConcurrentModification; /** * Stores records in a JSON file on a filesystem disk, one file per store. * - * Writes replace the file through an atomic rename. There is no cross-process lock, so concurrent - * writers can lose an update. + * Writes replace the file through an atomic rename, so a reader never sees half a file. On a local + * disk each read-modify-write holds an exclusive `flock` on a sidecar `.lock`, and a write + * is version-checked like the database layer's: a writer whose base version is stale gets + * {@see ConcurrentModification} instead of overwriting. Reads take no lock. A disk that is not + * local (S3 and the like) has no lock, so the version check is all that guards it. */ final readonly class JsonDriver implements Driver { @@ -43,18 +49,34 @@ public function load(StoreName $store, array $scopes): RecordSet )); } + /** + * Store a record one version above the version the writer read. + * + * @throws ConcurrentModification when the file holds another version than the writer read + */ public function put(StoreName $store, Record $record): void { - $records = $this->read($store); - $records[$record->identity()] = $record; - $this->write($store, $records); + $this->locked($store, function () use ($store, $record): void { + $records = $this->read($store); + $current = $records[$record->identity()] ?? null; + + // As the database layer: a record must be at the version below; a new one may start anywhere. + if ($current instanceof Record && $current->version !== $record->version - 1) { + throw new ConcurrentModification(sprintf('Setting [%s] in store [%s] for scope [%s] is at version %d in its file, not %d; re-read and retry.', $record->key, $store, $record->scope, $current->version, $record->version - 1)); + } + + $records[$record->identity()] = $record; + $this->write($store, $records); + }); } public function forget(StoreName $store, Scope $scope, Key $key, Locale $locale): void { - $records = $this->read($store); - unset($records[Record::identityOf($scope, $key, $locale)]); - $this->write($store, $records); + $this->locked($store, function () use ($store, $scope, $key, $locale): void { + $records = $this->read($store); + unset($records[Record::identityOf($scope, $key, $locale)]); + $this->write($store, $records); + }); } public function isWritable(): bool @@ -129,7 +151,7 @@ public function archiveKeeping(StoreName $store, CarbonImmutable $at, array $kep $records[$record->identity()] = $record; } - $this->write($store, $records); + $this->locked($store, fn () => $this->write($store, $records)); return $archive; } @@ -220,6 +242,45 @@ private function write(StoreName $store, array $records): void } } + /** + * Run a read-modify-write under an exclusive lock on the store's sidecar lock file, released + * whatever happens. A disk that is not local runs it unlocked. + * + * @param callable(): void $work + */ + private function locked(StoreName $store, callable $work): void + { + $lockPath = $this->localPath($this->path($store) . '.lock'); + + if ($lockPath === null) { + $work(); + + return; + } + + $directory = dirname($lockPath); + $handle = (is_dir($directory) || @mkdir($directory, 0o755, true)) ? @fopen($lockPath, 'c') : false; + + if ($handle === false || ! flock($handle, LOCK_EX)) { + throw new DriverFailure(sprintf('Settings file [%s] could not be locked for writing.', $this->path($store))); + } + + try { + $work(); + } finally { + flock($handle, LOCK_UN); + fclose($handle); + } + } + + /** The absolute path of a file on a local disk, or null when the disk is not local. */ + private function localPath(string $path): ?string + { + return $this->disk instanceof FilesystemAdapter && $this->disk->getAdapter() instanceof LocalFilesystemAdapter + ? $this->disk->path($path) + : null; + } + private function archivePath(StoreName $store, CarbonImmutable $at): string { return preg_replace('/\.json$/', '', $this->path($store)) . '.promoted-' . $at->format('Ymd_His_u') . '.json'; diff --git a/src/Http/Controllers/TransferController.php b/src/Http/Controllers/TransferController.php index 60cd143..faffc6f 100644 --- a/src/Http/Controllers/TransferController.php +++ b/src/Http/Controllers/TransferController.php @@ -4,19 +4,21 @@ namespace Simtabi\Laranail\Settings\Http\Controllers; +use Throwable; use Simtabi\Laranail\Settings\Http\Support\ApiActor; +use Symfony\Component\HttpFoundation\StreamedResponse; +use Simtabi\Laranail\Settings\Exceptions\DriverFailure; use Simtabi\Laranail\Settings\Actions\Transfer\DiffStore; use Simtabi\Laranail\Settings\Data\Transfer\ExportRequest; use Simtabi\Laranail\Settings\Data\Transfer\ImportRequest; -use Simtabi\Laranail\Settings\Actions\Transfer\ExportStore; use Simtabi\Laranail\Settings\Actions\Transfer\ImportStore; +use Simtabi\Laranail\Settings\Actions\Transfer\StreamExport; use Simtabi\Laranail\Settings\Http\Requests\DiffStoreRequest; use Simtabi\Laranail\Settings\Http\Requests\ExportStoreRequest; use Simtabi\Laranail\Settings\Http\Requests\ImportStoreRequest; use Simtabi\Laranail\Settings\Http\Resources\StoreDiffResource; use Simtabi\Laranail\Settings\Services\Resolution\SettingTargets; use Simtabi\Laranail\Settings\Http\Resources\ImportReportResource; -use Simtabi\Laranail\Settings\Http\Resources\ExportDocumentResource; /** * `/stores/{store}/export`, `/import` and `/diff`. An import is a dry run unless `apply` is true, @@ -24,11 +26,33 @@ */ final readonly class TransferController { - public function export(ExportStoreRequest $request, ExportStore $export, SettingTargets $targets, ApiActor $actors): ExportDocumentResource + /** + * The document under `data`, streamed 500 rows at a time. It is written to a temporary stream + * first (which spills to disk past 2 MB), so a refusal or failure is still a problem response. + */ + public function export(ExportStoreRequest $request, StreamExport $export, SettingTargets $targets, ApiActor $actors): StreamedResponse { - $document = $export->handle(new ExportRequest($targets->store((string) $request->route('store')), $request->mode()), $actors->of($request)); + $buffer = fopen('php://temp', 'w+b'); - return ExportDocumentResource::make($document); + if ($buffer === false) { + throw new DriverFailure('The export could not open a temporary stream.'); + } + + try { + fwrite($buffer, '{"data":'); + $export->handle(new ExportRequest($targets->store((string) $request->route('store')), $request->mode()), $buffer, $actors->of($request)); + fwrite($buffer, '}'); + } catch (Throwable $exception) { + fclose($buffer); + + throw $exception; + } + + return new StreamedResponse(static function () use ($buffer): void { + rewind($buffer); + fpassthru($buffer); + fclose($buffer); + }, 200, ['Content-Type' => 'application/json']); } public function import(ImportStoreRequest $request, ImportStore $import, SettingTargets $targets, ApiActor $actors): ImportReportResource diff --git a/src/Http/Middleware/IdempotentWrites.php b/src/Http/Middleware/IdempotentWrites.php index d8cc1b0..e0e4a3d 100644 --- a/src/Http/Middleware/IdempotentWrites.php +++ b/src/Http/Middleware/IdempotentWrites.php @@ -9,17 +9,21 @@ use Illuminate\Http\Request; use Simtabi\Laranail\Settings\Data\Actor; use Illuminate\Contracts\Cache\Repository; +use Illuminate\Contracts\Cache\LockProvider; use Symfony\Component\HttpFoundation\Response; use Illuminate\Http\Response as IlluminateResponse; use Simtabi\Laranail\Settings\Services\Access\AccessGuard; use Simtabi\Laranail\Settings\Exceptions\SettingsException; +use Simtabi\Laranail\Settings\Contracts\TenantDiscriminator; use Simtabi\Laranail\Settings\Support\LaravelValueEncrypter; +use Symfony\Component\HttpKernel\Exception\ConflictHttpException; use Symfony\Component\HttpKernel\Exception\UnprocessableEntityHttpException; /** - * Replays the stored response to a REST write retried with the same `Idempotency-Key`, actor, - * host, method and path. A retry with a different body is refused with 422. Server errors are - * not stored, so they can be retried, and nothing is stored for a request with no actor. + * Replays the stored response to a REST write retried with the same `Idempotency-Key`, tenant, + * actor, host, method and path. A retry with a different body is refused with 422, and one sent while + * the first is still running with 409. Server errors are not stored, so they can be retried, and + * nothing is stored for a request with no actor. * * The stored response is encrypted, because it can hold the value of an encrypted setting. */ @@ -31,6 +35,7 @@ public function __construct( private LaravelValueEncrypter $encrypter, private int $ttl = 86400, private string $prefix = 'laranail-settings', + private ?TenantDiscriminator $tenant = null, ) {} public function handle(Request $request, Closure $next): Response @@ -44,35 +49,74 @@ public function handle(Request $request, Closure $next): Response } $cacheKey = $this->prefix . ':idempotency:' . hash('sha256', implode('|', [ - $actor->type->value, $actor->alias ?? '', $actor->id ?? '', $request->getHost(), $request->method(), $request->path(), $key, + // The tenant too: `user:1` in two tenants on one host is two people. + $this->tenant?->current() ?? '', $actor->type->value, $actor->alias ?? '', $actor->id ?? '', $request->getHost(), $request->method(), $request->path(), $key, ])); $fingerprint = hash('sha256', $request->getQueryString() . '|' . $request->getContent()); - $stored = $this->read($this->cache->get($cacheKey)); + $replay = $this->replay($cacheKey, $fingerprint); + + if ($replay instanceof Response) { + return $replay; + } + + // One request per key at a time: a retry sent while the first runs would write twice. + $store = $this->cache->getStore(); + $lock = $store instanceof LockProvider ? $store->lock($cacheKey . ':lock', 60) : null; + + if ($lock !== null && ! $lock->get()) { + throw new ConflictHttpException('A request with this Idempotency-Key is still being processed; retry once it has answered.'); + } + + try { + // The first request may have finished between the read above and taking the lock. + $replay = $this->replay($cacheKey, $fingerprint); + + if ($replay instanceof Response) { + return $replay; + } - if ($stored !== null) { - if ($stored['fingerprint'] !== $fingerprint) { - throw new UnprocessableEntityHttpException('This Idempotency-Key was already used with a different request; send a new key.'); + /** @var Response $response */ + $response = $next($request); + + if ($response->getStatusCode() < 500) { + $this->store($cacheKey, $fingerprint, $response); } - return new IlluminateResponse($stored['body'], $stored['status'], $stored['headers'] + ['Idempotent-Replayed' => 'true']); + return $response; + } finally { + $lock?->release(); } + } + + /** The stored response for this key, or null; a different body under the same key is 422. */ + private function replay(string $cacheKey, string $fingerprint): ?Response + { + $stored = $this->read($this->cache->get($cacheKey)); - /** @var Response $response */ - $response = $next($request); + if ($stored === null) { + return null; + } - if ($response->getStatusCode() < 500) { - $this->store($cacheKey, $fingerprint, $response); + if ($stored['fingerprint'] !== $fingerprint) { + throw new UnprocessableEntityHttpException('This Idempotency-Key was already used with a different request; send a new key.'); } - return $response; + return new IlluminateResponse($stored['body'], $stored['status'], $stored['headers'] + ['Idempotent-Replayed' => 'true']); } private function store(string $cacheKey, string $fingerprint, Response $response): void { + // A streamed body (an export) cannot be kept, and a read needs no replay. + $body = $response->getContent(); + + if ($body === false) { + return; + } + $entry = json_encode([ 'fingerprint' => $fingerprint, 'status' => $response->getStatusCode(), - 'body' => (string) $response->getContent(), + 'body' => $body, 'headers' => array_filter([ 'Content-Type' => $response->headers->get('Content-Type'), 'ETag' => $response->headers->get('ETag'), diff --git a/src/Http/Support/ApiActor.php b/src/Http/Support/ApiActor.php index 982477e..61db7e0 100644 --- a/src/Http/Support/ApiActor.php +++ b/src/Http/Support/ApiActor.php @@ -23,6 +23,12 @@ public function __construct( public function of(Request $request, ?string $reason = null): Actor { $actor = $this->access->resolveActor($request) ?? throw new ActorRequired('The settings API needs an authenticated actor.'); + + // The system actor skips every check, so a resolver handing it to a request would open the API. + if ($actor->isSystem()) { + throw new ActorRequired('The settings API never acts as the system actor; resolve a user, token or service actor for the request.'); + } + $actor = $actor->withChannel(Channel::Rest); return $reason === null || $reason === '' ? $actor : $actor->withReason($reason); diff --git a/src/Models/SettingChangeRequest.php b/src/Models/SettingChangeRequest.php index fb799a4..6476bfd 100644 --- a/src/Models/SettingChangeRequest.php +++ b/src/Models/SettingChangeRequest.php @@ -121,6 +121,22 @@ public static function transition(?string $connection, ChangeRequest $request, a return self::on($connection)->whereKey($request->id)->where('status', $to->value)->exists(); } + /** + * Limit a query to one actor's requests; a token's are its user's, as {@see ChangeRequest::actorKey()} counts them. + * + * @param Builder $query + * + * @return Builder + */ + public static function requestedBy(Builder $query, Actor $actor): Builder + { + $types = in_array($actor->type, [ActorType::User, ActorType::Token], true) ? [ActorType::User->value, ActorType::Token->value] : [$actor->type->value]; + + return $query->whereIn('requested_by_type', $types) + ->when($actor->alias === null, static fn (Builder $query): Builder => $query->whereNull('requested_by_alias'), static fn (Builder $query): Builder => $query->where('requested_by_alias', $actor->alias)) + ->when($actor->id === null, static fn (Builder $query): Builder => $query->whereNull('requested_by_id'), static fn (Builder $query): Builder => $query->where('requested_by_id', $actor->id)); + } + #[Override] public function getTable(): string { diff --git a/src/Models/SettingDefinition.php b/src/Models/SettingDefinition.php index 64a5ce5..cfde2a3 100644 --- a/src/Models/SettingDefinition.php +++ b/src/Models/SettingDefinition.php @@ -60,14 +60,15 @@ public function category(): BelongsTo * of that slug, since definitions are shared by every tenant. * * @param ?string $default the encoded default; an encrypted definition's default is never stored + * @param array|null $categories the store's central category ids by slug, when the caller loaded them once; null looks the slug up */ - public function fillFromDefinition(Definition $definition, ?string $default): static + public function fillFromDefinition(Definition $definition, ?string $default, ?array $categories = null): static { - $category = $definition->category === null ? null : SettingCategory::on($this->getConnectionName()) - ->central() - ->forStore($this->store) - ->where('slug', $definition->category) - ->value('id'); + $category = match (true) { + $definition->category === null => null, + $categories !== null => $categories[$definition->category] ?? null, + default => $this->categoryId($definition->category), + }; return $this->fill([ 'slug' => $definition->key->slug(), @@ -138,4 +139,14 @@ protected function casts(): array 'is_system' => 'boolean', ]; } + + /** The id of the store's central category with this slug, or null. */ + private function categoryId(string $slug): mixed + { + return SettingCategory::on($this->getConnectionName()) + ->central() + ->forStore($this->store) + ->where('slug', $slug) + ->value('id'); + } } diff --git a/src/Providers/SettingsServiceProvider.php b/src/Providers/SettingsServiceProvider.php index 0d3ac1c..0f4846f 100644 --- a/src/Providers/SettingsServiceProvider.php +++ b/src/Providers/SettingsServiceProvider.php @@ -628,6 +628,7 @@ private function registerHttp(): void $app->make(LaravelValueEncrypter::class), $app->make(Repository::class)->integer('laranail.settings.http.idempotency.ttl', 86400), $app->make(Repository::class)->string('laranail.settings.cache.prefix', 'laranail-settings'), + $app->make(TenantDiscriminator::class), )); } diff --git a/src/Services/Resolution/Resolver.php b/src/Services/Resolution/Resolver.php index 4f14996..5219760 100644 --- a/src/Services/Resolution/Resolver.php +++ b/src/Services/Resolution/Resolver.php @@ -75,7 +75,7 @@ public function resolve(Store $store, Key $key, Locale $locale, ?OwnerRef $owner $record = $loaded[$stepStore->name->value][$layer->name]->find($scope, $key, $candidate); if ($record !== null && $record->hasValue()) { - return new Explanation(Lookup::found($this->pipeline->fromRecord($record, $definition), $layer->name, $scope), $steps); + return new Explanation(Lookup::found($this->pipeline->fromRecord($record, $definition), $layer->name, $scope, $record->encrypted), $steps); } } } diff --git a/src/Services/Stores/SettingWriter.php b/src/Services/Stores/SettingWriter.php index 2938aab..d45a7e6 100644 --- a/src/Services/Stores/SettingWriter.php +++ b/src/Services/Stores/SettingWriter.php @@ -121,7 +121,7 @@ public function isLocked(Store $store, Key $key, Locale $locale, ?OwnerRef $owne { $scope = $this->scope($owner); - return $store->writableLayer()->driver->load($store->name, [$scope])->find($scope, $key, $locale)?->locked === true; + return $this->current($store, $store->writableLayer(), $scope, $key, $locale)?->locked === true; } /** Lock one row against writes, creating a value-less row when none exists. */ @@ -167,7 +167,7 @@ private function writeOnce(Store $store, Key $key, Locale $locale, ?OwnerRef $ow $database = $this->database($layer); $existing = $relocked && $database instanceof DatabaseDriver ? $database->lockedFind($store->name, $scope, $key, $locale) - : $layer->driver->load($store->name, [$scope])->find($scope, $key, $locale); + : $this->current($store, $layer, $scope, $key, $locale); if ($expectedVersion !== null && ($existing->version ?? 0) !== $expectedVersion) { throw new ConcurrentModification(sprintf('Setting [%s] in store [%s] for scope [%s] is at version %d, not the expected %d; re-read and retry.', $key, $store->name, $scope, $existing->version ?? 0, $expectedVersion)); @@ -234,6 +234,19 @@ private function persist(Store $store, Layer $layer, HistoryOperation $operation }, $store->name); } + /** + * The writable layer's row for this identity. A database layer reads that one row, not the + * whole scope, so a write costs the same in a scope of ten keys or ten thousand. + */ + private function current(Store $store, Layer $layer, Scope $scope, Key $key, Locale $locale): ?Record + { + $database = $this->database($layer); + + return $database instanceof DatabaseDriver + ? $database->find($store->name, $scope, $key, $locale) + : $layer->driver->load($store->name, [$scope])->find($scope, $key, $locale); + } + private function database(Layer $layer): ?DatabaseDriver { return $layer->driver instanceof DatabaseBacked && $layer->driver->isActive() ? $layer->driver->database() : null; @@ -245,7 +258,7 @@ private function remove(Store $store, Key $key, Locale $locale, ?OwnerRef $owner $layer = $store->writableLayer(); $scope = $this->scope($owner); - $existing = $layer->driver->load($store->name, [$scope])->find($scope, $key, $locale); + $existing = $this->current($store, $layer, $scope, $key, $locale); $this->assertVersion($store, $scope, $key, $existing, $expectedVersion); if (! $existing instanceof Record) { @@ -290,7 +303,7 @@ private function storeLock(Store $store, Key $key, Locale $locale, ?OwnerRef $ow $layer = $store->writableLayer(); $scope = $this->scope($owner); - $found = $layer->driver->load($store->name, [$scope])->find($scope, $key, $locale); + $found = $this->current($store, $layer, $scope, $key, $locale); $existing = $found ?? new Record($scope, $key, $locale, null, version: $this->baseVersion($layer, $store, $scope, $key, $locale, null)); $record = new Record( diff --git a/src/Services/Transfer/StoreSnapshots.php b/src/Services/Transfer/StoreSnapshots.php index 90d0918..73257cf 100644 --- a/src/Services/Transfer/StoreSnapshots.php +++ b/src/Services/Transfer/StoreSnapshots.php @@ -13,6 +13,7 @@ use Simtabi\Laranail\Settings\Support\ValueEncoder; use Simtabi\Laranail\Settings\Models\SettingCategory; use Simtabi\Laranail\Settings\Data\Transfer\StoreDiff; +use Simtabi\Laranail\Settings\Exceptions\DriverFailure; use Simtabi\Laranail\Settings\Exceptions\StoreNotWritable; use Simtabi\Laranail\Settings\Data\Transfer\ExportDocument; use Simtabi\Laranail\Settings\Data\Transfer\ExportedSetting; @@ -64,6 +65,49 @@ public function export(Store $store, ExportMode $mode): ExportDocument ); } + /** + * Write the store's export to a stream, the same bytes {@see export()} and + * {@see ExportDocument::toJson()} produce, reading 500 rows at a time instead of holding the + * store in memory. + * + * @param resource $stream + * + * @return array{settings: int, categories: int} + * + * @throws StoreNotWritable when the store has no active database layer + */ + public function exportTo(Store $store, ExportMode $mode, $stream): array + { + $database = $store->requireDatabase('; exports read the database layer.'); + $categories = array_map(ExportedCategory::of(...), $this->categories($store)); + $head = new ExportDocument($store->name->value, $categories, [], $mode, CarbonImmutable::now()->toIso8601String())->toJson(); + $empty = "[]\n}\n"; + + if (! str_ends_with($head, $empty)) { + throw new DriverFailure('The export document no longer ends with its settings list.'); + } + + $write = static function (string $bytes) use ($stream): void { + if (fwrite($stream, $bytes) !== strlen($bytes)) { + throw new DriverFailure('The export could not be written to its stream.'); + } + }; + + $write(substr($head, 0, -strlen($empty))); + $written = 0; + + // Each entry as toJson() nests it: two levels in, so every line gains eight spaces. + $database->eachRecord($store->name, function (Record $record) use ($write, $mode, &$written): void { + $entry = json_encode($this->setting($record, $mode)->toArray(), ExportDocument::JSON_FLAGS); + $write(($written === 0 ? "[\n" : ",\n") . preg_replace('/^/m', ' ', $entry)); + $written++; + }); + + $write($written === 0 ? $empty : "\n ]\n}\n"); + + return ['settings' => $written, 'categories' => count($categories)]; + } + /** * Compare the store with a document. * diff --git a/src/SettingsManager.php b/src/SettingsManager.php index 787bfd6..edcbce2 100644 --- a/src/SettingsManager.php +++ b/src/SettingsManager.php @@ -197,7 +197,14 @@ public function lookup(string $key): Lookup $this->access->authorize($this->actingAs, Ability::ViewSensitive, $subject); } - return $this->resolve($store, $settingKey, $definition)->lookup; + $lookup = $this->resolve($store, $settingKey, $definition)->lookup; + + // A value stored encrypted stays a secret after its definition stops saying so. + if (! $subject->sensitive && $lookup->isEncrypted()) { + $this->access->authorize($this->actingAs, Ability::ViewSensitive, $subject); + } + + return $lookup; } public function has(string $key): bool @@ -296,7 +303,7 @@ public function explain(string $key): Explanation [$store, $settingKey, $definition, $subject] = $this->readTarget($key); $explanation = $this->resolve($store, $settingKey, $definition); - if ($subject->sensitive && $this->actingAs instanceof Actor && ! $this->access->allows($this->actingAs, Ability::ViewSensitive, $subject)) { + if (($subject->sensitive || $explanation->lookup->isEncrypted()) && $this->actingAs instanceof Actor && ! $this->access->allows($this->actingAs, Ability::ViewSensitive, $subject)) { return $explanation->redacted(); } diff --git a/tests/Feature/Database/ApprovalsTest.php b/tests/Feature/Database/ApprovalsTest.php index d7942df..e06d527 100644 --- a/tests/Feature/Database/ApprovalsTest.php +++ b/tests/Feature/Database/ApprovalsTest.php @@ -10,6 +10,8 @@ use Simtabi\Laranail\Settings\Data\Subject; use Simtabi\Laranail\Settings\Data\OwnerRef; use Simtabi\Laranail\Settings\Enums\Ability; +use Simtabi\Laranail\Settings\Data\StoreName; +use Simtabi\Laranail\Settings\Data\ChangeFilter; use Simtabi\Laranail\Settings\Enums\AuditAction; use Simtabi\Laranail\Settings\Data\ChangeRequest; use Simtabi\Laranail\Settings\Enums\AuditOutcome; @@ -24,6 +26,7 @@ use Simtabi\Laranail\Settings\Events\ChangeRequested; use Simtabi\Laranail\Settings\Exceptions\AccessDenied; use Simtabi\Laranail\Package\Tools\Testing\TestCredentials; +use Simtabi\Laranail\Settings\Actions\Approvals\QueryChanges; use Simtabi\Laranail\Settings\Exceptions\ApprovalUnavailable; use Simtabi\Laranail\Settings\Exceptions\ChangeRequestClosed; use Simtabi\Laranail\Settings\Exceptions\UnknownChangeRequest; @@ -130,6 +133,53 @@ function needsApproval(string $key = 'mail.host'): void ->and(settings()->get('mail.host'))->toBe('smtp.acme.test'); }); +it('counts a user and their API token as one person: no self-approval, no second vote (SP-03)', function (): void { + config()->set('laranail.settings.approvals.min_approvals', 2); + app()->forgetScopedInstances(); + needsApproval(); + settings()->as(user(1))->set('mail.host', 'smtp.acme.test'); + $id = settings()->approvals()->get()[0]->id; + + expect(fn (): ChangeRequest => settings()->as(Actor::user(user(1), abilities: ['*']))->approvals()->approve($id))->toThrow(AccessDenied::class, 'own change request'); + + settings()->as(user(2))->approvals()->approve($id); + $again = settings()->as(Actor::user(user(2), abilities: ['*']))->approvals()->approve($id); + + expect($again->status)->toBe(ChangeStatus::Approved) + ->and($again->approvals)->toBe(['user:' . user(2)->getMorphClass() . ':2']) + ->and(settings()->get('mail.host'))->toBeNull() + ->and(settings()->as(user(3))->approvals()->approve($id)->status)->toBe(ChangeStatus::Applied); +}); + +it('reads an approval recorded for a token before 0.1.1 as its user\'s (SP-03)', function (): void { + config()->set('laranail.settings.approvals.min_approvals', 2); + app()->forgetScopedInstances(); + needsApproval(); + settings()->as(user(1))->set('mail.host', 'smtp.acme.test'); + $id = settings()->approvals()->get()[0]->id; + DB::table('settings_change_requests')->where('id', $id)->update(['status' => 'approved', 'approvals' => json_encode(['token:' . user(2)->getMorphClass() . ':2'])]); + + expect(settings()->as(user(2))->approvals()->approve($id)->status)->toBe(ChangeStatus::Approved); +}); + +it('pages an actor\'s own change requests in the query, not after it (SP-05)', function (): void { + needsApproval('mail.host'); + needsApproval('mail.port'); + settings()->as(user(1))->set('mail.host', 'smtp.acme.test'); + + foreach ([2, 3, 4] as $id) { + settings()->as(user($id))->set('mail.port', $id); + } + + Gate::define(Ability::Approve->gateName(), static fn ($user): bool => $user->getAuthIdentifier() !== 1); + $own = app(QueryChanges::class)->handle(new ChangeFilter(new StoreName('settings')), Actor::user(user(1)), 3); + $asToken = app(QueryChanges::class)->handle(new ChangeFilter(new StoreName('settings')), Actor::user(user(1), abilities: ['*']), 3); + + expect(array_map(static fn (ChangeRequest $request): string => $request->key, $own))->toBe(['mail.host']) + ->and($asToken)->toHaveCount(1) + ->and(app(QueryChanges::class)->handle(new ChangeFilter(new StoreName('settings')), Actor::user(user(2)), 3))->toHaveCount(3); +}); + it('counts two approvals recorded at once, never losing one', function (): void { config()->set('laranail.settings.approvals.min_approvals', 3); app()->forgetScopedInstances(); diff --git a/tests/Feature/Database/AutoPromotionTest.php b/tests/Feature/Database/AutoPromotionTest.php index e872aac..bf4780b 100644 --- a/tests/Feature/Database/AutoPromotionTest.php +++ b/tests/Feature/Database/AutoPromotionTest.php @@ -359,7 +359,7 @@ function runtimeScopes(): array app()->terminate(); expect(DB::getQueryLog())->toBe([]) - ->and(Storage::disk('local')->files('laranail/settings'))->toHaveCount($inFile === [] ? 1 : 2); + ->and(array_filter(Storage::disk('local')->files('laranail/settings'), static fn (string $path): bool => ! str_ends_with($path, '.lock')))->toHaveCount($inFile === [] ? 1 : 2); })->with([ 'settings only' => [['include' => ['settings']], ['global'], ['user:1', 'team:2']], 'owners only' => [['include' => ['owners']], ['team:2', 'user:1'], ['global']], diff --git a/tests/Feature/Database/DatabaseLayerTest.php b/tests/Feature/Database/DatabaseLayerTest.php index 35fa634..348820d 100644 --- a/tests/Feature/Database/DatabaseLayerTest.php +++ b/tests/Feature/Database/DatabaseLayerTest.php @@ -202,3 +202,29 @@ function databaseDriver(bool $writable = true): DatabaseDriver expect(DB::table('app_settings')->value('key'))->toBe('site.name') ->and(Schema::hasTable('settings'))->toBeFalse(); }); + +it('reads only the row it writes, never the whole scope, on set, forget, lock and unlock (SP-07)', function (): void { + for ($index = 0; $index < 30; $index++) { + settings()->set('bulk.key' . $index, $index); + } + + $rows = 0; + $selects = 0; + DB::listen(static function ($query) use (&$rows, &$selects): void { + if (preg_match('/^select .* from ["`]?settings["`]? where/i', $query->sql) === 1) { + $selects++; + $rows += preg_match('/["`]key["`] = \\?/', $query->sql) === 1 ? 1 : 30; + } + }); + + settings()->set('bulk.key0', 'changed'); + settings()->lock('bulk.key1'); + settings()->unlock('bulk.key1'); + settings()->forget('bulk.key2'); + + // Each select names the key; one that loaded the global scope would count all 30 rows. + expect($selects)->toBeGreaterThan(0) + ->and($rows)->toBe($selects) + ->and(settings()->get('bulk.key0'))->toBe('changed') + ->and(settings()->has('bulk.key2'))->toBeFalse(); +}); diff --git a/tests/Feature/Database/DefinitionsTableTest.php b/tests/Feature/Database/DefinitionsTableTest.php index fc2f344..d981550 100644 --- a/tests/Feature/Database/DefinitionsTableTest.php +++ b/tests/Feature/Database/DefinitionsTableTest.php @@ -4,6 +4,7 @@ use Illuminate\Support\Str; use Illuminate\Support\Facades\DB; +use Illuminate\Support\Facades\Cache; use Illuminate\Support\Facades\Schema; use Simtabi\Laranail\Settings\Data\Actor; use Simtabi\Laranail\Settings\Enums\Ability; @@ -13,8 +14,10 @@ use Simtabi\Laranail\Settings\Exceptions\AccessDenied; use Simtabi\Laranail\Package\Tools\Testing\TestCredentials; use Simtabi\Laranail\Settings\Exceptions\InvalidDefinition; +use Simtabi\Laranail\Settings\Services\Stores\StoreRegistry; use Simtabi\Laranail\Settings\Exceptions\InvalidSettingValue; use Simtabi\Laranail\Settings\Exceptions\InvalidConfiguration; +use Simtabi\Laranail\Settings\Actions\Configuration\SaveDefinition; use Simtabi\Laranail\Settings\Services\Definitions\DefinitionRegistry; /** Insert a definition row edited at runtime (not is_system). */ @@ -171,3 +174,79 @@ function runtimeDefinition(string $key, array $columns = []): void $this->artisan('laranail::settings.sync-definitions')->assertFailed(); $this->artisan('laranail::settings.doctor', ['--fix' => true])->assertFailed(); }); + +it('syncs N definitions with the same number of reads as one (SP-06)', function (): void { + useDatabase(['definitions.sources' => ['code', 'config', 'database']]); + settings()->categories()->create(new NewCategory('mail', 'Mail')); + + $queries = static function (int $count): array { + for ($index = 0; $index < $count; $index++) { + app(DefinitionRegistry::class)->define('sync.key' . $index)->ofType('string')->inCategory('mail')->register(); + } + + DB::flushQueryLog(); + DB::enableQueryLog(); + app(SyncDefinitions::class)->handle(); + $selects = count(array_filter(DB::getQueryLog(), static fn (array $query): bool => str_starts_with(strtolower($query['query']), 'select'))); + DB::flushQueryLog(); + app(SyncDefinitions::class)->handle(); + $resync = count(DB::getQueryLog()); + DB::disableQueryLog(); + + return [$selects, $resync]; + }; + + [$selectsFor5, $resyncFor5] = $queries(5); + [$selectsFor20, $resyncFor20] = $queries(20); + + // A runtime definition saved on its own still finds its category. + app(SaveDefinition::class)->handle(app(StoreRegistry::class)->default(), 'sync.runtime', ['type' => 'string', 'category' => 'mail']); + + expect($selectsFor20)->toBe($selectsFor5) + ->and(DB::table('settings_definitions')->where('key', 'sync.runtime')->value('category_id'))->toBe(DB::table('settings_categories')->where('slug', 'mail')->value('id')) + ->and($resyncFor20)->toBe($resyncFor5) + ->and(DB::table('settings_definitions')->where('key', 'like', 'sync.key%')->whereNotNull('category_id')->count())->toBe(20); +}); + +it('skips a sync that overlaps a running one instead of failing on its rows (SP-11)', function (): void { + useDatabase(['definitions.sources' => ['code', 'config', 'database']]); + app(DefinitionRegistry::class)->define('overlap.a')->ofType('string')->register(); + app(DefinitionRegistry::class)->define('overlap.b')->ofType('string')->register(); + $second = null; + + // The second run starts while the first is inserting its rows. + DB::listen(static function ($query) use (&$second): void { + if ($second === null && str_starts_with(strtolower($query->sql), 'insert into') && str_contains($query->sql, 'settings_definitions')) { + $second = false; + + try { + $second = app()->makeWith(SyncDefinitions::class, ['wait' => 0])->handle(); + } catch (Throwable $exception) { + $second = $exception; + } + } + }); + + $first = app(SyncDefinitions::class)->handle(); + + expect($second)->toBeArray() + ->and($second['busy'] ?? null)->toBeTrue() + ->and($second['synced'] ?? null)->toBe(0) + ->and($first['busy'])->toBeFalse() + ->and($first['synced'])->toBeGreaterThanOrEqual(2) + ->and(DB::table('settings_definitions')->whereIn('key', ['overlap.a', 'overlap.b'])->count())->toBe(2); + + // The lock is released: the next run syncs again. + expect(app()->makeWith(SyncDefinitions::class, ['wait' => 0])->handle()['busy'])->toBeFalse(); + + Cache::lock('laranail-settings:sync-definitions', 300)->get(); + $this->app->when(SyncDefinitions::class)->needs('$wait')->give(0); + $this->artisan('laranail::settings.sync-definitions')->expectsOutputToContain('Another sync-definitions run holds the lock')->assertSuccessful(); +}); + +it('gives up on the sync lock after waiting, and reports busy (SP-11)', function (): void { + useDatabase(['definitions.sources' => ['code', 'config', 'database']]); + Cache::lock('laranail-settings:sync-definitions', 300)->get(); + + expect(app()->makeWith(SyncDefinitions::class, ['wait' => 1])->handle()['busy'])->toBeTrue(); +}); diff --git a/tests/Feature/Database/RestApiTest.php b/tests/Feature/Database/RestApiTest.php index 712c613..7255fb2 100644 --- a/tests/Feature/Database/RestApiTest.php +++ b/tests/Feature/Database/RestApiTest.php @@ -2,6 +2,8 @@ declare(strict_types=1); +use Illuminate\Http\Request; +use Illuminate\Http\Response; use Illuminate\Routing\Router; use Illuminate\Support\Facades\DB; use Illuminate\Support\Facades\Gate; @@ -9,6 +11,7 @@ use Illuminate\Support\Facades\Event; use Illuminate\Support\Facades\Schema; use Illuminate\Support\Facades\Storage; +use Illuminate\Cache\Events\CacheMissed; use Illuminate\Database\Schema\Blueprint; use Simtabi\Laranail\Settings\Data\Actor; use Simtabi\Laranail\Settings\Data\OwnerRef; @@ -20,13 +23,18 @@ use Simtabi\Laranail\Settings\Http\Support\Problems; use Simtabi\Laranail\Settings\Exceptions\AccessDenied; use Simtabi\Laranail\Settings\Tests\Fixtures\TestUser; +use Symfony\Component\HttpFoundation\StreamedResponse; use Simtabi\Laranail\Settings\Tests\Support\ApiSchemas; use Simtabi\Laranail\Settings\Tests\Support\JsonSchema; use Simtabi\Laranail\Package\Tools\Testing\TestCredentials; use Simtabi\Laranail\Settings\Tests\Fixtures\SettingsOwner; +use Simtabi\Laranail\Settings\Contracts\TenantDiscriminator; use Simtabi\Laranail\Settings\Services\Values\ValuePipeline; use Simtabi\Laranail\Settings\Exceptions\InvalidSettingValue; use Simtabi\Laranail\Settings\Exceptions\InvalidConfiguration; +use Simtabi\Laranail\Settings\Http\Middleware\IdempotentWrites; +use Simtabi\Laranail\Settings\Tenancy\ClosureTenantDiscriminator; +use Symfony\Component\HttpKernel\Exception\ConflictHttpException; use Simtabi\Laranail\Settings\Services\Definitions\DefinitionRegistry; /* @@ -52,12 +60,16 @@ function api(string $method, string $uri, array $data = [], array $headers = []) $response = test()->json($method, '/api/settings/v1/' . $uri, $data, $headers); $route = $response->baseRequest->route(); - if (is_object($route) && $response->status() !== 204) { - $schema = ApiSchemas::response((string) $route->getName(), $response->status()); + $status = $response->baseResponse->getStatusCode(); - expect($schema)->not->toBeNull(sprintf('%s %s answered %d, which no response shape describes', $method, $uri, $response->status())); + if (is_object($route) && $status !== 204) { + $schema = ApiSchemas::response((string) $route->getName(), $status); - $errors = new JsonSchema(ApiSchemas::document())->errors(json_decode((string) $response->getContent(), true), $schema); + expect($schema)->not->toBeNull(sprintf('%s %s answered %d, which no response shape describes', $method, $uri, $status)); + + // An export is streamed. + $content = $response->baseResponse instanceof StreamedResponse ? $response->streamedContent() : (string) $response->getContent(); + $errors = new JsonSchema(ApiSchemas::document())->errors(json_decode($content, true), $schema); expect($errors)->toBe([], sprintf('%s %s does not match its schema', $method, $uri)); } @@ -267,6 +279,148 @@ function api(string $method, string $uri, array $data = [], array $headers = []) ->and((string) file_get_contents((string) array_key_first($files)))->not->toContain($secret); }); +it('answers a retry sent while the first request still runs with 409, and writes once (SP-04)', function (): void { + $runs = 0; + $inner = null; + $request = static function (): Request { + $request = Request::create('/api/settings/v1/stores/settings/settings/site.name', 'PUT', content: '{"value":"Acme"}'); + $request->headers->set('Idempotency-Key', 'in-flight'); + + return $request; + }; + $middleware = app(IdempotentWrites::class); + + // The retry arrives while the first request is still inside the action. + $middleware->handle($request(), function () use ($middleware, $request, &$runs, &$inner): Response { + $runs++; + + try { + $middleware->handle($request(), static function () use (&$runs): Response { + $runs++; + + return new Response('second', 200); + }); + } catch (ConflictHttpException $conflict) { + $inner = $conflict->getStatusCode(); + } + + return new Response('first', 200); + }); + + $replay = $middleware->handle($request(), static fn (): Response => new Response('third', 200)); + + expect($runs)->toBe(1) + ->and($inner)->toBe(409) + ->and($replay->getContent())->toBe('first') + ->and($replay->headers->get('Idempotent-Replayed'))->toBe('true'); +}); + +it('replays a response stored while it waited for the key, rather than writing again (SP-04)', function (): void { + $runs = 0; + $request = static function (): Request { + $request = Request::create('/api/settings/v1/stores/settings/settings/site.name', 'PUT', content: '{"value":"Acme"}'); + $request->headers->set('Idempotency-Key', 'finished-meanwhile'); + + return $request; + }; + $middleware = app(IdempotentWrites::class); + $raced = false; + + // The first request finishes between this one's cache read and its taking the lock. + Event::listen(CacheMissed::class, static function () use ($middleware, $request, &$runs, &$raced): void { + if (! $raced) { + $raced = true; + $middleware->handle($request(), static function () use (&$runs): Response { + $runs++; + + return new Response('first', 200); + }); + } + }); + + $response = $middleware->handle($request(), static function () use (&$runs): Response { + $runs++; + + return new Response('second', 200); + }); + + expect($raced)->toBeTrue() + ->and($runs)->toBe(1) + ->and($response->getContent())->toBe('first') + ->and($response->headers->get('Idempotent-Replayed'))->toBe('true'); +}); + +it('keys a replay by the tenant, so the same user id in two tenants never shares a response (SP-08)', function (): void { + useDatabase(['stores.settings.owners' => ['user'], 'tenancy.mode' => 'column', 'verification.driver' => 'none']); + $tenant = (object) ['id' => 'acme']; + app()->forgetScopedInstances(); + app()->instance(TenantDiscriminator::class, new ClosureTenantDiscriminator(static fn (): ?string => $tenant->id)); + allowGates(...Ability::cases()); + $secret = TestCredentials::secret(); + + api('PUT', 'stores/settings/settings/site.name', ['value' => $secret], ['Idempotency-Key' => 'same-key'])->assertOk(); + + $tenant->id = 'globex'; + $other = test()->json('PUT', '/api/settings/v1/stores/settings/settings/site.name', ['value' => $secret], ['Idempotency-Key' => 'same-key'])->assertOk(); + + expect($other->headers->get('Idempotent-Replayed'))->toBeNull() + ->and($other->json('data.version'))->toBe(1); + + $tenant->id = 'acme'; + $replay = test()->json('PUT', '/api/settings/v1/stores/settings/settings/site.name', ['value' => $secret], ['Idempotency-Key' => 'same-key'])->assertOk(); + + expect($replay->headers->get('Idempotent-Replayed'))->toBe('true'); +}); + +it('answers a refused export with a problem, not a half-written stream (SP-10)', function (): void { + Gate::define(Ability::Export->gateName(), static fn (): bool => false); + + api('POST', 'stores/settings/export')->assertForbidden()->assertJsonPath('status', 403); +}); + +it('never runs a REST request as the system actor, whatever the resolver returns (SP-02)', function (): void { + settings()->configuration()->resolveActorsUsing(static fn (): Actor => Actor::system('service account')); + + foreach (Ability::cases() as $ability) { + Gate::define($ability->gateName(), static fn (): bool => false); + } + + test()->json('PUT', '/api/settings/v1/stores/settings/settings/site.name', ['value' => 'Acme']) + ->assertUnauthorized() + ->assertJsonPath('type', 'https://opensource.simtabi.com/documentation/laranail/settings/tools/exceptions#actor-required'); + test()->json('GET', '/api/settings/v1/stores/settings/settings')->assertUnauthorized(); + + expect(settings()->get('site.name'))->toBeNull(); +}); + +it('keeps a value stored encrypted secret after its definition stops saying so (SP-01)', function (): void { + $secret = TestCredentials::secret(); + api('PUT', 'stores/settings/definitions/mail.password', ['attributes' => ['type' => 'string', 'encrypted' => true]])->assertOk(); + settings()->set('mail.password', $secret); + Gate::define(Ability::ViewSensitive->gateName(), static fn (): bool => false); + + // manage_definitions alone: redefine the key as plain, then delete the definition. + api('PUT', 'stores/settings/definitions/mail.password', ['attributes' => ['type' => 'string']])->assertOk(); + api('GET', 'stores/settings/settings/mail.password')->assertOk()->assertJsonPath('data.redacted', true)->assertJsonMissingPath('data.value'); + api('DELETE', 'stores/settings/definitions/mail.password')->assertNoContent(); + + $show = api('GET', 'stores/settings/settings/mail.password')->assertOk()->assertJsonPath('data.redacted', true)->assertJsonMissingPath('data.value'); + $list = api('GET', 'stores/settings/settings')->assertOk(); + $explain = api('GET', 'stores/settings/settings/mail.password/explain')->assertOk(); + + expect($show->getContent())->not->toContain($secret) + ->and($list->getContent())->not->toContain($secret) + ->and($explain->getContent())->not->toContain($secret) + ->and(fn (): mixed => settings()->as(user(1))->get('mail.password'))->toThrow(AccessDenied::class, 'view_sensitive') + ->and(settings()->as(user(1))->all())->not->toHaveKey('mail.password') + // Trusted code and an actor allowed view_sensitive still read it. + ->and(settings()->get('mail.password'))->toBe($secret); + + Gate::define(Ability::ViewSensitive->gateName(), static fn (): bool => true); + + expect(settings()->as(user(1))->get('mail.password'))->toBe($secret); +}); + it('applies values.max_bytes to REST writes, imports and promotion alike', function (): void { config()->set('laranail.settings.values.max_bytes', 64); app()->forgetInstance(ValuePipeline::class); diff --git a/tests/Feature/Database/TransferTest.php b/tests/Feature/Database/TransferTest.php index 936c309..5dce695 100644 --- a/tests/Feature/Database/TransferTest.php +++ b/tests/Feature/Database/TransferTest.php @@ -2,7 +2,9 @@ declare(strict_types=1); +use Illuminate\Support\Str; use Illuminate\Support\Facades\DB; +use Illuminate\Support\Facades\Gate; use Illuminate\Support\Facades\Schema; use Illuminate\Support\Facades\Artisan; use Illuminate\Database\Schema\Blueprint; @@ -17,9 +19,12 @@ use Simtabi\Laranail\Settings\Exceptions\AccessDenied; use Simtabi\Laranail\Settings\Exceptions\InvalidImport; use Simtabi\Laranail\Settings\Data\Transfer\ImportReport; +use Simtabi\Laranail\Settings\Data\Transfer\ExportRequest; use Simtabi\Laranail\Settings\Services\Transfer\Importers; use Simtabi\Laranail\Package\Tools\Testing\TestCredentials; use Simtabi\Laranail\Settings\Data\Transfer\ExportDocument; +use Simtabi\Laranail\Settings\Actions\Transfer\StreamExport; +use Simtabi\Laranail\Settings\Services\Stores\StoreRegistry; use Simtabi\Laranail\Settings\Data\Transfer\ExportedCategory; use Simtabi\Laranail\Settings\Exceptions\InvalidConfiguration; use Simtabi\Laranail\Settings\Services\Definitions\DefinitionRegistry; @@ -222,3 +227,67 @@ function seedStore(): string ->and(fn (): StoreDiff => settings()->as(user())->compare($document))->toThrow(AccessDenied::class, 'export') ->and(settings()->as(Actor::system())->compare($document)->isEmpty())->toBeTrue(); }); + +it('streams the export as the same bytes the document gives, in identity order (SP-10)', function (ExportMode $mode): void { + $this->travelTo(now()->startOfSecond()); + settings()->categories()->create(new NewCategory('mail', 'Mail')); + app(DefinitionRegistry::class)->define('mail.password')->encrypted()->register(); + settings()->set('mail.password', TestCredentials::secret()); + // `user:1` / `user:10` and `mail` / `mail.host` sort differently as columns and as identities. + settings()->set('mail', ['nested' => ['a' => 1.0, 'b' => 'naïve/✓'], 'empty' => []]); + settings()->set('mail.host', 'smtp.acme.test'); + settings()->for(new OwnerRef('user', 10))->set('ui.theme', 'dark'); + settings()->for(new OwnerRef('user', 1))->set('ui.theme', 'light'); + settings()->for(new OwnerRef('user', 1))->set('ui.null', null); + $stream = fopen('php://temp', 'w+b'); + + $counts = app(StreamExport::class)->handle(new ExportRequest(app(StoreRegistry::class)->default(), $mode), $stream); + rewind($stream); + + expect(stream_get_contents($stream))->toBe(settings()->export($mode)->toJson()) + ->and($counts)->toBe(['settings' => 6, 'categories' => 1]); +})->with([ExportMode::Redacted, ExportMode::Ciphertext, ExportMode::Plaintext]); + +it('streams an empty store as the same bytes too (SP-10)', function (): void { + $this->travelTo(now()->startOfSecond()); + $stream = fopen('php://temp', 'w+b'); + app(StreamExport::class)->handle(new ExportRequest(app(StoreRegistry::class)->default()), $stream); + rewind($stream); + + expect(stream_get_contents($stream))->toBe(settings()->export()->toJson()); +}); + +it('reads the store 500 rows at a time when it streams an export (SP-10)', function (): void { + $now = now()->toDateTimeString(); + DB::table('settings')->insert(array_map(static fn (int $index): array => [ + 'id' => (string) Str::ulid(), 'uid' => (string) Str::ulid(), 'scope' => 'global', 'key' => 'bulk.k' . $index, 'locale' => '', + 'value' => (string) $index, 'version' => 1, 'autoload' => true, 'sort_order' => 0, 'group' => 'bulk', 'created_at' => $now, 'updated_at' => $now, + ], range(1, 1100))); + + DB::flushQueryLog(); + DB::enableQueryLog(); + $path = sys_get_temp_dir() . '/laranail-settings-stream-' . bin2hex(random_bytes(4)) . '.json'; + expect(Artisan::call('laranail::settings.export', ['path' => $path]))->toBe(0); + $chunks = array_values(array_filter(DB::getQueryLog(), static fn (array $query): bool => preg_match('/from ["`]settings["`] .*limit 500/i', $query['query']) === 1)); + DB::disableQueryLog(); + + expect($chunks)->toHaveCount(3) + ->and(ExportDocument::fromJson((string) file_get_contents($path))->settings)->toHaveCount(1100); +}); + +it('reports what stops a streamed export: an unknown store, an unwritable path, no database layer, a refusal (SP-10)', function (): void { + expect(Artisan::call('laranail::settings.export', ['--store' => 'nope']))->toBe(1) + ->and(Artisan::call('laranail::settings.export', ['path' => sys_get_temp_dir() . '/no-such-directory-' . bin2hex(random_bytes(4)) . '/x.json']))->toBe(1); + + config()->set('laranail.settings.stores.filed', ['layers' => ['runtime']]); + app()->forgetScopedInstances(); + + expect(Artisan::call('laranail::settings.export', ['--store' => 'filed']))->toBe(1) + ->and(Artisan::output())->toContain('no active database layer'); + + $stream = fopen('php://temp', 'w+b'); + Gate::define(Ability::Export->gateName(), static fn (): bool => false); + + expect(fn () => app(StreamExport::class)->handle(new ExportRequest(app(StoreRegistry::class)->default()), $stream, Actor::user(user())))->toThrow(AccessDenied::class) + ->and(fstat($stream)['size'] ?? null)->toBe(0); +}); diff --git a/tests/Feature/DriverContractTest.php b/tests/Feature/DriverContractTest.php index 427d630..1f19764 100644 --- a/tests/Feature/DriverContractTest.php +++ b/tests/Feature/DriverContractTest.php @@ -7,13 +7,19 @@ use Simtabi\Laranail\Settings\Data\Locale; use Simtabi\Laranail\Settings\Data\Record; use Simtabi\Laranail\Settings\Data\OwnerRef; +use Simtabi\Laranail\Settings\Data\RecordSet; use Simtabi\Laranail\Settings\Data\StoreName; +use Illuminate\Contracts\Filesystem\Filesystem; use Simtabi\Laranail\Settings\Contracts\Driver; use Simtabi\Laranail\Settings\Data\LayerConfig; use Simtabi\Laranail\Settings\Enums\DriverType; +use Simtabi\Laranail\Settings\Drivers\JsonDriver; +use Simtabi\Laranail\Settings\Exceptions\DriverFailure; +use Simtabi\Laranail\Settings\Support\RecordSerializer; use Simtabi\Laranail\Settings\Exceptions\StoreNotWritable; use Simtabi\Laranail\Settings\Services\Drivers\DriverFactory; use Simtabi\Laranail\Settings\Exceptions\InvalidConfiguration; +use Simtabi\Laranail\Settings\Exceptions\ConcurrentModification; /** * One suite over every built-in driver. The driver list is DriverType::cases(), not a @@ -105,3 +111,130 @@ function driverFor(DriverType $type): Driver ->and(fn () => $factory->extend('json', static fn (LayerConfig $layer): Driver => driverFor(DriverType::Array))) ->toThrow(InvalidConfiguration::class, 'is built in'); }); + +/** A JSON layer on a fresh local disk of its own, for the concurrency tests. */ +function freshJsonDriver(): Driver +{ + $root = sys_get_temp_dir() . '/laranail-json-' . bin2hex(random_bytes(6)); + config()->set('filesystems.disks.json-concurrency', ['driver' => 'local', 'root' => $root]); + + return app(DriverFactory::class)->make(new LayerConfig('j', 'json', true, ['disk' => 'json-concurrency', 'path' => 'settings/{store}.json'])); +} + +it('refuses a JSON write whose base version is stale, so no update is lost (SP-09)', function (): void { + $driver = freshJsonDriver(); + $store = new StoreName('settings'); + $key = new Key('site.name'); + + $driver->put($store, new Record(Scope::global(), $key, Locale::none(), '"first"', version: 1)); + // Two writers both read version 1; the first lands version 2. + $driver->put($store, new Record(Scope::global(), $key, Locale::none(), '"writer a"', version: 2)); + + expect(fn () => $driver->put($store, new Record(Scope::global(), $key, Locale::none(), '"writer b"', version: 2)))->toThrow(ConcurrentModification::class, 'version 2 in its file, not 1') + ->and($driver->load($store, [Scope::global()])->find(Scope::global(), $key, Locale::none())?->payload)->toBe('"writer a"'); + + // The lock was released by the refused write: the next one goes through. + $driver->put($store, new Record(Scope::global(), $key, Locale::none(), '"writer b again"', version: 3)); + + expect($driver->load($store, [Scope::global()])->find(Scope::global(), $key, Locale::none())?->version)->toBe(3); +}); + +it('loses no record when two processes write one JSON file at once (SP-09)', function (): void { + if (! function_exists('pcntl_fork')) { + $this->markTestSkipped('pcntl is not available.'); + } + + $driver = freshJsonDriver(); + $store = new StoreName('settings'); + $driver->put($store, new Record(Scope::global(), new Key('seed'), Locale::none(), '0', version: 1)); + $writes = 40; + $pid = pcntl_fork(); + + if ($pid === 0) { + for ($index = 0; $index < $writes; $index++) { + $driver->put($store, new Record(Scope::global(), new Key('child.k' . $index), Locale::none(), '1', version: 1)); + } + + // End the child without running the parent's shutdown and test teardown. + posix_kill(getmypid(), SIGKILL); + } + + for ($index = 0; $index < $writes; $index++) { + $driver->put($store, new Record(Scope::global(), new Key('parent.k' . $index), Locale::none(), '1', version: 1)); + } + + pcntl_waitpid($pid, $status); + + expect($driver->load($store, [Scope::global()])->count())->toBe(2 * $writes + 1); +}); + +it('reports a JSON lock it cannot take, and still version-checks a disk with no local path (SP-09)', function (): void { + $root = sys_get_temp_dir() . '/laranail-json-' . bin2hex(random_bytes(6)); + config()->set('filesystems.disks.json-lockless', ['driver' => 'local', 'root' => $root]); + $driver = app(DriverFactory::class)->make(new LayerConfig('j', 'json', true, ['disk' => 'json-lockless', 'path' => 'settings/{store}.json'])); + // A directory where the lock file belongs: it cannot be opened, so the write is refused. + mkdir($root . '/settings/settings.json.lock', 0o755, true); + + expect(fn () => $driver->put(new StoreName('settings'), new Record(Scope::global(), new Key('a'), Locale::none(), '1', version: 1)))->toThrow(DriverFailure::class, 'could not be locked'); + + // A disk that is not local: no lock, but the version check still refuses a stale write. + $files = []; + $disk = Mockery::mock(Filesystem::class); + $disk->allows('exists')->andReturnUsing(static function (string $path) use (&$files): bool { + return isset($files[$path]); + }); + $disk->allows('get')->andReturnUsing(static function (string $path) use (&$files): string { + return $files[$path]; + }); + $disk->allows('put')->andReturnUsing(static function (string $path, string $contents) use (&$files): bool { + $files[$path] = $contents; + + return true; + }); + $disk->allows('move')->andReturnUsing(static function (string $from, string $to) use (&$files): bool { + $files[$to] = $files[$from]; + unset($files[$from]); + + return true; + }); + $remote = new JsonDriver($disk, 'settings/{store}.json', true, app(RecordSerializer::class)); + $remote->put(new StoreName('settings'), new Record(Scope::global(), new Key('a'), Locale::none(), '1', version: 1)); + $remote->put(new StoreName('settings'), new Record(Scope::global(), new Key('a'), Locale::none(), '2', version: 2)); + + expect(fn () => $remote->put(new StoreName('settings'), new Record(Scope::global(), new Key('a'), Locale::none(), '3', version: 2)))->toThrow(ConcurrentModification::class) + ->and(array_keys($files))->toBe(['settings/settings.json']); +}); + +it('reports every JSON disk failure as DriverFailure, never as the disk\'s own exception', function (): void { + $broken = Mockery::mock(Filesystem::class); + $broken->allows('exists')->andThrow(new RuntimeException('disk gone')); + $broken->allows('copy')->andThrow(new RuntimeException('disk gone')); + $driver = new JsonDriver($broken, 'settings/{store}.json', true, app(RecordSerializer::class)); + $store = new StoreName('settings'); + + expect(fn (): bool => $driver->hasFile($store))->toThrow(DriverFailure::class, 'could not be read') + ->and(fn (): array => $driver->recordsIn('../etc/passwd'))->toThrow(DriverFailure::class, 'must not contain') + ->and(fn (): array => $driver->recordsIn('settings/other.json'))->toThrow(DriverFailure::class, 'could not be read') + ->and(fn (): ?string => $driver->archive($store, now()->toImmutable()))->toThrow(DriverFailure::class, 'could not be archived') + ->and(fn (): string => $driver->archiveKeeping($store, now()->toImmutable(), []))->toThrow(DriverFailure::class, 'could not be archived'); + + $refusing = Mockery::mock(Filesystem::class); + $refusing->allows('exists')->andReturn(true); + $refusing->allows('get')->andReturn('{"records": ["not a record"]}'); + $refusing->allows('copy')->andReturn(false); + $refusing->allows('move')->andReturn(false); + $refusing->allows('put')->andThrow(new RuntimeException('read-only')); + $driver = new JsonDriver($refusing, 'settings/{store}.json', true, app(RecordSerializer::class)); + + expect(fn (): RecordSet => $driver->load($store, [Scope::global()]))->toThrow(DriverFailure::class, 'malformed record') + ->and(fn (): ?string => $driver->archive($store, now()->toImmutable()))->toThrow(DriverFailure::class, 'could not be archived') + ->and(fn (): string => $driver->archiveKeeping($store, now()->toImmutable(), []))->toThrow(DriverFailure::class, 'could not be archived') + ->and(fn () => $driver->forget($store, Scope::global(), new Key('a'), Locale::none()))->toThrow(DriverFailure::class, 'malformed record'); + + $readOnly = Mockery::mock(Filesystem::class); + $readOnly->allows('exists')->andReturn(false); + $readOnly->allows('put')->andThrow(new RuntimeException('read-only')); + $driver = new JsonDriver($readOnly, 'settings/{store}.json', true, app(RecordSerializer::class)); + + expect(fn () => $driver->forget($store, Scope::global(), new Key('a'), Locale::none()))->toThrow(DriverFailure::class, 'could not be written'); +});