diff --git a/AGENTS.md b/AGENTS.md index 9b7eb8a4..552e54fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,7 +63,7 @@ Depmesh is configured to log significant operation steps via `task` tool. Special workflows to use: -- `@/workflows/polish.donna.md` — format, fix architecture, lint, and test errors. Run it after making changes to the codebase at the moments when the project is expected to be in a working state: between significant implementation steps, before reporting completion of a task, etc. Run this workflow instead of running individual operations, unless you are explicitly needed to run a specific operation for some reason. +- `@/workflows/polish.donna.md` — format, fix architecture, lint, and test errors. Run it after making changes to the codebase at the moments when the project is expected to be in a working state: between significant implementation steps, before reporting completion of a task, etc. Do not run it when all changes made for the current task are confined to files under `specs/`; review `depmesh` dependencies and perform targeted specification checks instead. Run this workflow instead of running individual operations, unless you are explicitly needed to run a specific operation for some reason. Do not run `donna -p llm new-session` unless the developer explicitly asks to reset or start a fresh Donna session. @@ -90,6 +90,7 @@ The queue is an isolated Taskwarrior database of relation-pair checks. Each queu `depmesh` relation from a changed or manually selected file to one related artifact, plus the current SHA-256 checksums of both files, the relation id, the check status, and an optional markdown report. Pair keys include the relation and both file checksums, so old records remain as history while changed file content creates a fresh unchecked pair. +Reconciliation immediately marks older checksum versions of the same oriented relation pair as `outdated`. The checker loop is the `run-cycle` command. It discovers files changed relative to `main`, queries all configured `depmesh` relations for those files, reconciles the current relation pairs into the queue, then handles at most one @@ -97,10 +98,24 @@ unchecked pair. If a current pair is already marked `inconsistent`, the loop pri child checker. Otherwise it runs one read-only child Codex checker for the first unchecked pair, stores the result, and exits with a code that tells the Donna workflow whether to stop for a fix, continue the loop, or finish successfully. +The `enqueue-changed` command performs the changed-file discovery, `depmesh` queries, and queue reconciliation portion +of `run-cycle`, then exits successfully without processing unchecked pairs or spawning child Codex checkers. + +The `sync-queue` command discovers Git-changed files and non-outdated manually tracked changed-side files, reconciles +their current `depmesh` relations, and marks stale checksum versions or relations no longer returned by `depmesh` as +`outdated`. It does not process unchecked pairs or spawn child Codex checkers. + +`list-pairs` shows queue history by default. Pass `--current` to show only records whose stored checksums match the +current files and whose oriented relation is still returned by `depmesh`; this current-only view does not mutate the +queue. + Main commands: - `python ./bin/inconsistency-check.py enqueue @/path/to/file` — manually add one file and all configured depmesh relation pairs for that file to the isolated queue. - `python ./bin/inconsistency-check.py enqueue @/first @/second` — enqueue multiple files. +- `python ./bin/inconsistency-check.py enqueue-changed` — enqueue all relation pairs for files changed relative to `main` without processing unchecked pairs or spawning child checkers. +- `python ./bin/inconsistency-check.py sync-queue` — reconcile Git-changed and manually tracked files, then mark stale checksums and removed relations outdated without processing unchecked pairs. +- `python ./bin/inconsistency-check.py list-pairs --current` — show only current-checksum records for relations still returned by `depmesh`. - `python ./bin/inconsistency-check.py progress --file @/path/to/file` — show queued records where the file is either the changed side or the related side. - `python ./bin/inconsistency-check.py mark-consistent --changed @/changed --related @/related --relation ` — explicitly mark the current-checksum relation pair as consistent. - `python ./bin/inconsistency-check.py mark-inconsistent --changed @/changed --related @/related --relation --report ""` — explicitly mark the current-checksum relation pair as inconsistent. @@ -133,6 +148,12 @@ Use `rg` for text and file searches unless a structural code query is needed. `ast-grep` has a higher priority than `rg` whenever a structural code query is needed. +### Specification reading + +Grep-like tools, including `rg`, MAY be used to discover relevant specification files. Search results MUST be treated only as discovery hints. + +Before relying on, interpreting, reviewing, or changing a specification file, an agent MUST read that file in full from beginning to end. Agents MUST NOT use `sed`, `head`, `tail`, line-range readers, grep context output, or any other partial-file reading method to read specification content. If a whole-file read is truncated, the agent MUST repeat it with sufficient output capacity and MUST NOT proceed until the complete file has been read. + ### `taskwarrior` `task` — Taskwarrior — is the project journal for significant agent-side work. diff --git a/bin/inconsistency-check.py b/bin/inconsistency-check.py index dcb605a9..fdd91faa 100755 --- a/bin/inconsistency-check.py +++ b/bin/inconsistency-check.py @@ -32,12 +32,14 @@ import time import tomllib from collections import Counter +from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass from datetime import UTC, datetime from enum import IntEnum from pathlib import Path, PurePosixPath +from threading import Barrier, Lock from types import SimpleNamespace -from typing import Any, Iterable +from typing import Any, Callable, Iterable PROJECT_ROOT = Path(__file__).resolve().parents[1] @@ -58,8 +60,9 @@ class ConsistencyConfig: runtime_dir: Path comparison_base_refs: tuple[str, ...] allowed_file_relations: tuple[str, ...] - jobs: int - journal_cmd: tuple[str, ...] + agent_jobs: int + discovery_jobs: int + journal_cmd: tuple[str, ...] | None agent_cmd: tuple[str, ...] agent_timeout_seconds: int prompt_template: str @@ -145,6 +148,7 @@ class ListPairsOptions: multi_line: bool = False include_report: bool = False include_all_fields: bool = False + current_only: bool = False statuses: tuple[str, ...] = () include_count: bool = True @@ -156,6 +160,44 @@ class CurrentPair: record: CheckRecord +@dataclass(frozen=True) +class QueueSyncResult: + tracked_files: tuple[str, ...] + current_pairs: tuple[CurrentPair, ...] + checked_records: int + marked_outdated_records: int + + +GraphComponent = tuple[str, ...] + + +@dataclass(frozen=True) +class DependencyGraphs: + traversal_vertices: tuple[str, ...] + traversal_edges: tuple[tuple[str, str], ...] + scheduling_components: tuple[GraphComponent, ...] + scheduling_edges: tuple[tuple[GraphComponent, GraphComponent], ...] + cycles: tuple[tuple[str, ...], ...] + ignored_self_edges: int + + +@dataclass(frozen=True) +class DependencyState: + changed_files: tuple[str, ...] + direct_pairs: tuple[RelationPair, ...] + graphs: DependencyGraphs + + +@dataclass(frozen=True) +class FrontierSelection: + component_statuses: tuple[tuple[GraphComponent, str], ...] + frontier_components: tuple[GraphComponent, ...] + resolved_files: tuple[str, ...] + pending_files: tuple[str, ...] + blocked_files: tuple[str, ...] + deferred_inconsistencies: int + + @dataclass(frozen=True) class PairSelection: inconsistent: CurrentPair | None @@ -333,13 +375,21 @@ def validate_config(config: dict[str, Any], *, mode: str) -> ConsistencyConfig: runtime_dir = normalize_runtime_dir(require_string(config, "runtime_dir", context="consistency.toml")) comparison_base_refs = require_string_list(config, "comparison_base_refs", context="consistency.toml") allowed_file_relations = require_string_list(config, "allowed_file_relations", context="consistency.toml") - jobs = require_int(config, "jobs", context="consistency.toml") + agent_jobs = require_int(config, "agent_jobs", context="consistency.toml") + discovery_jobs = require_int(config, "discovery_jobs", context="consistency.toml") - if jobs <= 0: - raise CheckerFailureError("consistency.toml: jobs must be positive") + if agent_jobs <= 0: + raise CheckerFailureError("consistency.toml: agent_jobs must be positive") + + if discovery_jobs <= 0: + raise CheckerFailureError("consistency.toml: discovery_jobs must be positive") journal = require_table(config, "journal", context="consistency.toml") - journal_cmd = require_string_list(journal, "cmd", context="consistency.toml journal") + journal_cmd = ( + None + if journal.get("cmd") is None + else require_string_list(journal, "cmd", context="consistency.toml journal") + ) agent = require_table(config, "agent", context="consistency.toml") agent_cmd = require_string_list(agent, "cmd", context="consistency.toml agent") agent_timeout_seconds = require_int(agent, "timeout_seconds", context="consistency.toml agent") @@ -390,7 +440,8 @@ def validate_config(config: dict[str, Any], *, mode: str) -> ConsistencyConfig: runtime_dir=runtime_dir, comparison_base_refs=comparison_base_refs, allowed_file_relations=allowed_file_relations, - jobs=jobs, + agent_jobs=agent_jobs, + discovery_jobs=discovery_jobs, journal_cmd=journal_cmd, agent_cmd=agent_cmd, agent_timeout_seconds=agent_timeout_seconds, @@ -838,7 +889,7 @@ def parse_depmesh_dependencies( return pairs -def query_depmesh_pairs(changed_files: list[str]) -> list[RelationPair]: +def resolved_allowed_relations() -> tuple[DepmeshRelation, ...]: config = get_config() depmesh_relations = {relation.relation_id: relation for relation in load_depmesh_relations()} missing_depmesh_relations = sorted(set(config.allowed_file_relations) - set(depmesh_relations)) @@ -849,26 +900,67 @@ def query_depmesh_pairs(changed_files: list[str]) -> list[RelationPair]: + ", ".join(missing_depmesh_relations) ) - relations = [ + return tuple( DepmeshRelation( relation_id=relation_id, description=config.relations[relation_id].description, ) for relation_id in config.allowed_file_relations - ] - pair_map: dict[tuple[str, str, str], RelationPair] = {} + ) - for changed_path in sorted(changed_files): - for relation in relations: - records = run_automation_jsonl( - ["depmesh", "-p", "automation", "dependencies", "--relation", relation.relation_id, changed_path], - failure_context=f"querying depmesh relation {relation.relation_id} for {changed_path}", + +def query_depmesh_dependency_records( + changed_path: str, + relation: DepmeshRelation, +) -> list[dict[str, Any]]: + return run_automation_jsonl( + ["depmesh", "-p", "automation", "dependencies", "--relation", relation.relation_id, changed_path], + failure_context=f"querying depmesh relation {relation.relation_id} for {changed_path}", + ) + + +def query_artifacts_pairs( + changed_files: Iterable[str], + relations: Iterable[DepmeshRelation], + *, + discovery_jobs: int, + query_records: Callable[[str, DepmeshRelation], list[dict[str, Any]]] = query_depmesh_dependency_records, +) -> dict[str, list[RelationPair]]: + if discovery_jobs <= 0: + raise CheckerFailureError("discovery_jobs must be positive") + + changed_paths = tuple(sorted(set(changed_files))) + ordered_relations = tuple(sorted(relations, key=lambda item: item.relation_id)) + queries = tuple( + (changed_path, relation) + for changed_path in changed_paths + for relation in ordered_relations + ) + pairs_by_artifact: dict[str, list[RelationPair]] = {changed_path: [] for changed_path in changed_paths} + + if not queries: + return pairs_by_artifact + + worker_count = min(discovery_jobs, len(queries)) + + with ThreadPoolExecutor(max_workers=worker_count) as executor: + records_by_query = { + (changed_path, relation): executor.submit( + query_records, + changed_path, + relation, ) + for changed_path, relation in queries + } + + for changed_path, relation in queries: + records = records_by_query[(changed_path, relation)].result() pairs = parse_depmesh_dependencies(records, changed_path=changed_path, relation=relation) log_project_journal( "step", f"depmesh query for {changed_path} relation {relation.relation_id} found {len(pairs)} related files", ) + pairs_by_artifact[changed_path].extend(pairs) for pair in pairs: log_project_journal( @@ -878,11 +970,240 @@ def query_depmesh_pairs(changed_files: list[str]) -> list[RelationPair]: f"{pair.changed_path} -> {pair.related_path} [{pair.relation}]" ), ) - pair_map[(pair.changed_path, pair.relation, pair.related_path)] = pair + + return { + changed_path: sorted( + { + (pair.changed_path, pair.relation, pair.related_path): pair + for pair in pairs + }.values(), + key=lambda pair: (pair.changed_path, pair.relation, pair.related_path), + ) + for changed_path, pairs in sorted(pairs_by_artifact.items()) + } + + +def query_depmesh_pairs(changed_files: list[str]) -> list[RelationPair]: + relations = resolved_allowed_relations() + pair_map: dict[tuple[str, str, str], RelationPair] = {} + pairs_by_artifact = query_artifacts_pairs( + changed_files, + relations, + discovery_jobs=get_config().discovery_jobs, + ) + + for changed_path in sorted(changed_files): + for pair in pairs_by_artifact[changed_path]: + pair_map[(pair.changed_path, pair.relation, pair.related_path)] = pair return [pair_map[key] for key in sorted(pair_map)] +def strongly_connected_components( + vertices: Iterable[str], + edges: Iterable[tuple[str, str]], +) -> tuple[tuple[GraphComponent, ...], dict[str, GraphComponent]]: + ordered_vertices = tuple(sorted(set(vertices))) + adjacency = {vertex: [] for vertex in ordered_vertices} + + for source, target in sorted(set(edges)): + adjacency[source].append(target) + + index = 0 + indices: dict[str, int] = {} + lowlinks: dict[str, int] = {} + stack: list[str] = [] + on_stack: set[str] = set() + components: list[GraphComponent] = [] + + def visit(vertex: str) -> None: + nonlocal index + indices[vertex] = index + lowlinks[vertex] = index + index += 1 + stack.append(vertex) + on_stack.add(vertex) + + for target in adjacency[vertex]: + if target not in indices: + visit(target) + lowlinks[vertex] = min(lowlinks[vertex], lowlinks[target]) + elif target in on_stack: + lowlinks[vertex] = min(lowlinks[vertex], indices[target]) + + if lowlinks[vertex] != indices[vertex]: + return + + members: list[str] = [] + + while True: + member = stack.pop() + on_stack.remove(member) + members.append(member) + + if member == vertex: + break + + components.append(tuple(sorted(members))) + + for vertex in ordered_vertices: + if vertex not in indices: + visit(vertex) + + ordered_components = tuple(sorted(components)) + component_by_artifact = { + artifact: component + for component in ordered_components + for artifact in component + } + + return ordered_components, component_by_artifact + + +def build_dependency_graphs( + changed_files: Iterable[str], + dependency_adjacency: dict[str, Iterable[str]], +) -> DependencyGraphs: + changed_set = set(changed_files) + vertices = set(changed_set) + traversal_edges: set[tuple[str, str]] = set() + ignored_self_edges = 0 + + for queried_path in sorted(dependency_adjacency): + vertices.add(queried_path) + + for related_path in sorted(set(dependency_adjacency[queried_path])): + vertices.add(related_path) + + if related_path == queried_path: + ignored_self_edges += 1 + continue + + traversal_edges.add((related_path, queried_path)) + + components, component_by_artifact = strongly_connected_components(vertices, traversal_edges) + component_edges: set[tuple[GraphComponent, GraphComponent]] = set() + + for source, target in traversal_edges: + source_component = component_by_artifact[source] + target_component = component_by_artifact[target] + + if source_component != target_component: + component_edges.add((source_component, target_component)) + + changed_components = tuple( + sorted( + tuple(sorted(changed_set.intersection(component))) + for component in components + if changed_set.intersection(component) + ) + ) + changed_component_by_full_component = { + component: tuple(sorted(changed_set.intersection(component))) + for component in components + if changed_set.intersection(component) + } + component_successors: dict[GraphComponent, set[GraphComponent]] = { + component: set() for component in components + } + + for source_component, target_component in component_edges: + component_successors[source_component].add(target_component) + + scheduling_edges: set[tuple[GraphComponent, GraphComponent]] = set() + + for full_source, scheduling_source in sorted(changed_component_by_full_component.items()): + pending = sorted(component_successors[full_source]) + visited: set[GraphComponent] = set() + + while pending: + component = pending.pop(0) + + if component in visited: + continue + + visited.add(component) + scheduling_target = changed_component_by_full_component.get(component) + + if scheduling_target is not None: + scheduling_edges.add((scheduling_source, scheduling_target)) + continue + + pending.extend(sorted(component_successors[component])) + pending.sort() + + cycles = tuple(component for component in components if len(component) > 1) + + return DependencyGraphs( + traversal_vertices=tuple(sorted(vertices)), + traversal_edges=tuple(sorted(traversal_edges)), + scheduling_components=changed_components, + scheduling_edges=tuple(sorted(scheduling_edges)), + cycles=cycles, + ignored_self_edges=ignored_self_edges, + ) + + +def discover_dependency_state(changed_files: Iterable[str]) -> DependencyState: + changed_paths = tuple(sorted(set(changed_files))) + relations = resolved_allowed_relations() + pending = list(changed_paths) + visited: set[str] = set() + dependency_adjacency: dict[str, tuple[str, ...]] = {} + direct_pair_map: dict[tuple[str, str, str], RelationPair] = {} + log_project_journal("step", "dependency graph construction started") + + while pending: + wave = tuple(sorted(set(pending) - visited)) + pending = [] + + if not wave: + break + + visited.update(wave) + pairs_by_artifact = query_artifacts_pairs( + wave, + relations, + discovery_jobs=get_config().discovery_jobs, + ) + next_wave: set[str] = set() + + for queried_path in wave: + pairs = pairs_by_artifact[queried_path] + dependencies = tuple(sorted({pair.related_path for pair in pairs})) + dependency_adjacency[queried_path] = dependencies + + if queried_path in changed_paths: + for pair in pairs: + direct_pair_map[(pair.changed_path, pair.relation, pair.related_path)] = pair + + next_wave.update(path for path in dependencies if path not in visited) + + pending = sorted(next_wave) + + graphs = build_dependency_graphs(changed_paths, dependency_adjacency) + log_project_journal( + "step", + ( + "dependency graph construction completed " + f"traversal-vertices:{len(graphs.traversal_vertices)} " + f"traversal-edges:{len(graphs.traversal_edges)} " + f"scheduling-vertices:{len(graphs.scheduling_components)} " + f"scheduling-edges:{len(graphs.scheduling_edges)} " + f"ignored-self-edges:{graphs.ignored_self_edges}" + ), + ) + + for cycle in graphs.cycles: + log_project_journal("thought", f"collapsed dependency cycle: {', '.join(cycle)}") + + return DependencyState( + changed_files=changed_paths, + direct_pairs=tuple(direct_pair_map[key] for key in sorted(direct_pair_map)), + graphs=graphs, + ) + + def render_expression_template(template: str, context: dict[str, Any]) -> str: parts: list[str] = [] index = 0 @@ -942,13 +1263,18 @@ def single_line(value: str) -> str: def log_project_journal(kind: str, message: str) -> None: """Log project-level script events without touching relation-pair records.""" + journal_cmd = get_config().journal_cmd + + if journal_cmd is None: + return + clean_kind = single_line(kind) if not clean_kind or clean_kind != kind: raise CheckerFailureError(f"invalid journal kind: {kind!r}") command = render_command_argv( - get_config().journal_cmd, + journal_cmd, { "kind": clean_kind, "message": single_line(message), @@ -1063,6 +1389,99 @@ def raw_record_to_check_record(record: dict[str, Any]) -> CheckRecord: ) +def load_allowed_check_records() -> list[CheckRecord]: + allowed_relations = set(get_config().allowed_file_relations) + + return [ + raw_record_to_check_record(record) + for record in load_taskwarrior_records() + if record.get("pair_key") and record.get("relation") in allowed_relations + ] + + +def load_allowed_check_records_read_only() -> list[CheckRecord]: + paths = runtime_paths() + + if not paths.taskrc_path.is_file() or not paths.task_data_dir.is_dir(): + return [] + + result = run_command( + task_command_args("export"), + check=True, + failure_context="exporting isolated inconsistency-check Taskwarrior records read-only", + ) + + try: + records = json.loads(result.stdout or "[]") + except json.JSONDecodeError as error: + raise CheckerFailureError(f"invalid Taskwarrior export JSON: {error}") from error + + if not isinstance(records, list) or any(not isinstance(record, dict) for record in records): + raise CheckerFailureError("Taskwarrior export did not return a JSON object list") + + allowed_relations = set(get_config().allowed_file_relations) + + return [ + raw_record_to_check_record(record) + for record in records + if record.get("pair_key") and record.get("relation") in allowed_relations + ] + + +def virtual_unchecked_record(identity: PairIdentity) -> CheckRecord: + return CheckRecord( + uuid="", + pair_key=identity.pair_key, + file_pair=identity.file_pair, + changed_path=identity.changed_path, + related_path=identity.related_path, + relation=identity.relation, + checksum_changed=identity.checksum_changed, + checksum_related=identity.checksum_related, + check_status="unchecked", + report="", + checked_at="", + ) + + +def current_pairs_read_only( + pairs: Iterable[RelationPair], + records: Iterable[CheckRecord], +) -> list[CurrentPair]: + records_by_key: dict[str, CheckRecord] = {} + + for record in records: + if record.pair_key in records_by_key: + raise CheckerFailureError("isolated Taskwarrior DB has duplicate pair_key records") + + records_by_key[record.pair_key] = record + + current_pairs: list[CurrentPair] = [] + + for pair in sorted(pairs, key=lambda item: (item.changed_path, item.relation, item.related_path)): + try: + identity = build_pair_identity(pair) + except MissingArtifactError: + continue + + record = records_by_key.get(identity.pair_key) + matching_record = ( + record is not None + and record.checksum_changed == identity.checksum_changed + and record.checksum_related == identity.checksum_related + and normalized_check_status(record) != "outdated" + ) + current_pairs.append( + CurrentPair( + pair=pair, + identity=identity, + record=record if matching_record and record is not None else virtual_unchecked_record(identity), + ) + ) + + return current_pairs + + def find_raw_record_by_pair_key(records: list[dict[str, Any]], pair_key: str) -> dict[str, Any] | None: matches = [record for record in records if record.get("pair_key") == pair_key] @@ -1234,6 +1653,31 @@ def mark_current_pair_outdated_if_needed(current_pair: CurrentPair) -> CurrentPa return CurrentPair(pair=current_pair.pair, identity=current_pair.identity, record=record) +def mark_superseded_pair_versions(identity: PairIdentity, records: Iterable[CheckRecord]) -> int: + marked_count = 0 + + for record in records: + same_oriented_pair = ( + record.changed_path == identity.changed_path + and record.related_path == identity.related_path + and record.relation == identity.relation + ) + + if not same_oriented_pair or record.pair_key == identity.pair_key: + continue + + if normalized_check_status(record) == "outdated": + continue + + mark_record_outdated( + record, + f"superseded by current relation-pair checksums: {identity.pair_key}", + ) + marked_count += 1 + + return marked_count + + def relation_description_for(relation_id: str) -> str: descriptions = { configured_relation_id: relation.description @@ -1286,7 +1730,9 @@ def set_relation_pair_check_status( def reconcile_queue(pairs: list[RelationPair]) -> list[CurrentPair]: current_pairs: list[CurrentPair] = [] + existing_records = load_allowed_check_records() skipped_missing = 0 + superseded_records = 0 for pair in sorted(pairs, key=lambda item: (item.changed_path, item.relation, item.related_path)): try: @@ -1303,11 +1749,18 @@ def reconcile_queue(pairs: list[RelationPair]) -> list[CurrentPair]: continue record = upsert_unchecked_record(identity) + superseded_records += mark_superseded_pair_versions(identity, existing_records) current_pairs.append(CurrentPair(pair=pair, identity=identity, record=record)) if skipped_missing: log_project_journal("step", f"queue reconciliation skipped {skipped_missing} missing-file pair records") + if superseded_records: + log_project_journal( + "step", + f"queue reconciliation marked {superseded_records} superseded pair records outdated", + ) + log_project_journal("step", f"queue reconciliation produced {len(current_pairs)} current pair records") return current_pairs @@ -1383,92 +1836,321 @@ def print_summary(changed_files: list[str], current_pairs: list[CurrentPair]) -> print(f"- {changed_file}") -def replace_current_pair(current_pairs: list[CurrentPair], updated_pair: CurrentPair) -> list[CurrentPair]: - return [ - updated_pair if current_pair.identity.pair_key == updated_pair.identity.pair_key else current_pair - for current_pair in current_pairs - ] +def artifact_statuses( + changed_files: Iterable[str], + current_pairs: Iterable[CurrentPair], +) -> dict[str, str]: + statuses_by_artifact: dict[str, list[str]] = {path: [] for path in changed_files} + for current_pair in current_pairs: + if current_pair.pair.changed_path not in statuses_by_artifact: + continue -def remove_current_pair(current_pairs: list[CurrentPair], removed_pair: CurrentPair) -> list[CurrentPair]: - return [ - current_pair - for current_pair in current_pairs - if current_pair.identity.pair_key != removed_pair.identity.pair_key - ] + status = normalized_check_status(current_pair.record) + if status == "outdated": + status = "unchecked" -def first_inconsistent_pair(current_pairs: list[CurrentPair]) -> CurrentPair | None: - for current_pair in sorted(current_pairs, key=current_pair_sort_key): - if normalized_check_status(current_pair.record) == "inconsistent": - return current_pair + statuses_by_artifact[current_pair.pair.changed_path].append(status) - return None + result: dict[str, str] = {} + for artifact, statuses in statuses_by_artifact.items(): + if "inconsistent" in statuses: + result[artifact] = "inconsistent" + elif "unchecked" in statuses: + result[artifact] = "unchecked" + elif statuses: + result[artifact] = "resolved" + else: + result[artifact] = "resolved-without-pairs" -def has_unchecked_pair(current_pairs: list[CurrentPair]) -> bool: - return any(normalized_check_status(current_pair.record) == "unchecked" for current_pair in current_pairs) + return result -def relation_specific_criteria(relation_id: str) -> list[str]: - config = get_config() - relation_config = config.relations.get(relation_id) +def select_frontier( + graphs: DependencyGraphs, + current_pairs: Iterable[CurrentPair], +) -> FrontierSelection: + current_pairs = list(current_pairs) + changed_files = tuple(sorted(path for component in graphs.scheduling_components for path in component)) + statuses_by_artifact = artifact_statuses(changed_files, current_pairs) + component_statuses: dict[GraphComponent, str] = {} - if relation_config is None: - valid_relations = ", ".join(sorted(config.relations)) or "(none)" - raise CheckerFailureError(f"unknown configured relation {relation_id!r}; valid relations: {valid_relations}") + for component in graphs.scheduling_components: + member_statuses = [statuses_by_artifact[path] for path in component] - return [*relation_config.criteria, *config.common_criteria] + if "inconsistent" in member_statuses: + component_statuses[component] = "inconsistent" + elif "unchecked" in member_statuses: + component_statuses[component] = "unchecked" + else: + component_statuses[component] = "resolved" + predecessors: dict[GraphComponent, set[GraphComponent]] = { + component: set() for component in graphs.scheduling_components + } -def fenced_content(label: str, content: str) -> str: - fence = "```" + for source, target in graphs.scheduling_edges: + predecessors[target].add(source) - while fence in content: - fence += "`" + def all_predecessors(component: GraphComponent) -> set[GraphComponent]: + result: set[GraphComponent] = set() + pending = sorted(predecessors[component]) - return f"{label}\n{fence}\n{content}\n{fence}" + while pending: + predecessor = pending.pop(0) + if predecessor in result: + continue -def sha256_file(path: str) -> str: - filesystem_path = PROJECT_ROOT / path + result.add(predecessor) + pending.extend(sorted(predecessors[predecessor])) + pending.sort() - try: - content = filesystem_path.read_bytes() - except OSError as error: - raise CheckerFailureError(f"could not read file for sha256_file({path!r}): {error}") from error + return result - return checksum_bytes(content) + pending_components = { + component + for component, status in component_statuses.items() + if status in {"unchecked", "inconsistent"} + } + frontier_components = tuple( + sorted( + component + for component in pending_components + if all(component_statuses[predecessor] == "resolved" for predecessor in all_predecessors(component)) + ) + ) + if pending_components and not frontier_components: + pending_text = ", ".join(path for component in sorted(pending_components) for path in component) + raise CheckerFailureError(f"pending dependency components have no selectable frontier: {pending_text}") -def identity_hash(identity: PairIdentity) -> str: - return hashlib.sha256(identity.pair_key.encode("utf-8")).hexdigest() + frontier_files = {path for component in frontier_components for path in component} + resolved_files = tuple( + sorted(path for path, status in statuses_by_artifact.items() if status.startswith("resolved")) + ) + pending_files = tuple( + sorted(path for path, status in statuses_by_artifact.items() if status in {"unchecked", "inconsistent"}) + ) + blocked_files = tuple(sorted(set(pending_files) - frontier_files)) + deferred_inconsistencies = sum( + 1 + for current_pair in current_pairs + if current_pair.pair.changed_path not in frontier_files + and normalized_check_status(current_pair.record) == "inconsistent" + ) + return FrontierSelection( + component_statuses=tuple(sorted(component_statuses.items())), + frontier_components=frontier_components, + resolved_files=resolved_files, + pending_files=pending_files, + blocked_files=blocked_files, + deferred_inconsistencies=deferred_inconsistencies, + ) -def validate_prompt_snapshots(current_pair: CurrentPair) -> tuple[FileSnapshot, FileSnapshot]: - changed, related = read_pair_snapshots(current_pair.pair) - if changed.checksum != current_pair.identity.checksum_changed: - raise OutdatedPairError(f"changed file checksum drifted before child check: {changed.artifact_path}") +def frontier_files(selection: FrontierSelection) -> tuple[str, ...]: + return tuple(sorted({path for component in selection.frontier_components for path in component})) - if related.checksum != current_pair.identity.checksum_related: - raise OutdatedPairError(f"related file checksum drifted before child check: {related.artifact_path}") - return changed, related +def build_frontier_report(changed_files: Iterable[str], selection: FrontierSelection) -> str: + changed_files = tuple(changed_files) + selected_files = frontier_files(selection) + lines = [ + "Current frontier", + f"changed files: {len(changed_files)}", + f"frontier files: {len(selected_files)}", + ] + lines.extend(f"- {selected_file}" for selected_file in selected_files) + return "\n".join(lines) -def build_child_prompt(current_pair: CurrentPair) -> str: - changed, related = validate_prompt_snapshots(current_pair) - base_ref = resolve_comparison_base() - merge_base = merge_base_with_head(base_ref) - criteria = "\n".join(f"- {criterion}" for criterion in relation_specific_criteria(current_pair.pair.relation)) - context = { - "pair": SimpleNamespace( - relation=current_pair.pair.relation, - relation_description=current_pair.pair.relation_description, - ), - "changed": SimpleNamespace( - path=changed.root_path, + +def ordered_frontier_pairs( + selection: FrontierSelection, + current_pairs: Iterable[CurrentPair], +) -> list[CurrentPair]: + component_index = { + path: index + for index, component in enumerate(selection.frontier_components) + for path in component + } + + return sorted( + ( + current_pair + for current_pair in current_pairs + if current_pair.pair.changed_path in component_index + ), + key=lambda current_pair: ( + component_index[current_pair.pair.changed_path], + *current_pair_sort_key(current_pair), + ), + ) + + +def first_frontier_pair_with_status( + frontier_pairs: Iterable[CurrentPair], + status: str, + *, + excluded_pair_keys: set[str] | None = None, +) -> CurrentPair | None: + excluded_pair_keys = excluded_pair_keys or set() + + for current_pair in frontier_pairs: + if current_pair.identity.pair_key in excluded_pair_keys: + continue + + if normalized_check_status(current_pair.record) == status: + return current_pair + + return None + + +def next_frontier_candidate( + frontier_pairs: Iterable[CurrentPair], + running_pair_keys: set[str], + *, + agent_jobs: int, + stop_launching: bool, +) -> CurrentPair | None: + if stop_launching or len(running_pair_keys) >= agent_jobs: + return None + + return first_frontier_pair_with_status( + frontier_pairs, + "unchecked", + excluded_pair_keys=running_pair_keys, + ) + + +def completed_frontier_exit_code( + frontier_pairs: Iterable[CurrentPair], + *, + launched_child: bool, + stale_found: bool, +) -> ExitCode: + if stale_found: + return ExitCode.CONTINUE_CYCLE + + if first_frontier_pair_with_status(frontier_pairs, "inconsistent") is not None: + return ExitCode.INCONSISTENCY_FOUND + + if launched_child: + return ExitCode.CONTINUE_CYCLE + + raise CheckerFailureError("pending frontier completed without a child launch or workflow outcome") + + +def print_frontier_summary( + changed_files: Iterable[str], + current_pairs: list[CurrentPair], + selection: FrontierSelection, + *, + fresh_cycle_required: bool, +) -> None: + counts = status_counts(current_pairs) + print("Consistency frontier summary") + print(f"changed artifacts: {len(tuple(changed_files))}") + print(f"resolved artifacts: {len(selection.resolved_files)}") + print(f"pending artifacts: {len(selection.pending_files)}") + print(f"blocked artifacts: {len(selection.blocked_files)}") + print(f"active frontier artifacts: {len(frontier_files(selection))}") + print(f"consistent pairs: {counts.get('consistent', 0)}") + print(f"inconsistent pairs: {counts.get('inconsistent', 0)}") + print(f"unchecked pairs: {counts.get('unchecked', 0)}") + print(f"fresh cycle required: {'yes' if fresh_cycle_required else 'no'}") + + +def replace_current_pair(current_pairs: list[CurrentPair], updated_pair: CurrentPair) -> list[CurrentPair]: + return [ + updated_pair if current_pair.identity.pair_key == updated_pair.identity.pair_key else current_pair + for current_pair in current_pairs + ] + + +def remove_current_pair(current_pairs: list[CurrentPair], removed_pair: CurrentPair) -> list[CurrentPair]: + return [ + current_pair + for current_pair in current_pairs + if current_pair.identity.pair_key != removed_pair.identity.pair_key + ] + + +def first_inconsistent_pair(current_pairs: list[CurrentPair]) -> CurrentPair | None: + for current_pair in sorted(current_pairs, key=current_pair_sort_key): + if normalized_check_status(current_pair.record) == "inconsistent": + return current_pair + + return None + + +def has_unchecked_pair(current_pairs: list[CurrentPair]) -> bool: + return any(normalized_check_status(current_pair.record) == "unchecked" for current_pair in current_pairs) + + +def relation_specific_criteria(relation_id: str) -> list[str]: + config = get_config() + relation_config = config.relations.get(relation_id) + + if relation_config is None: + valid_relations = ", ".join(sorted(config.relations)) or "(none)" + raise CheckerFailureError(f"unknown configured relation {relation_id!r}; valid relations: {valid_relations}") + + return [*relation_config.criteria, *config.common_criteria] + + +def fenced_content(label: str, content: str) -> str: + fence = "```" + + while fence in content: + fence += "`" + + return f"{label}\n{fence}\n{content}\n{fence}" + + +def sha256_file(path: str) -> str: + filesystem_path = PROJECT_ROOT / path + + try: + content = filesystem_path.read_bytes() + except OSError as error: + raise CheckerFailureError(f"could not read file for sha256_file({path!r}): {error}") from error + + return checksum_bytes(content) + + +def identity_hash(identity: PairIdentity) -> str: + return hashlib.sha256(identity.pair_key.encode("utf-8")).hexdigest() + + +def validate_prompt_snapshots(current_pair: CurrentPair) -> tuple[FileSnapshot, FileSnapshot]: + changed, related = read_pair_snapshots(current_pair.pair) + + if changed.checksum != current_pair.identity.checksum_changed: + raise OutdatedPairError(f"changed file checksum drifted before child check: {changed.artifact_path}") + + if related.checksum != current_pair.identity.checksum_related: + raise OutdatedPairError(f"related file checksum drifted before child check: {related.artifact_path}") + + return changed, related + + +def build_child_prompt(current_pair: CurrentPair) -> str: + changed, related = validate_prompt_snapshots(current_pair) + base_ref = resolve_comparison_base() + merge_base = merge_base_with_head(base_ref) + criteria = "\n".join(f"- {criterion}" for criterion in relation_specific_criteria(current_pair.pair.relation)) + context = { + "pair": SimpleNamespace( + relation=current_pair.pair.relation, + relation_description=current_pair.pair.relation_description, + ), + "changed": SimpleNamespace( + path=changed.root_path, artifact_path=changed.artifact_path, root_path=changed.root_path, checksum=changed.checksum, @@ -1720,17 +2402,6 @@ def update_record_from_child_output(current_pair: CurrentPair, output: str) -> C return CurrentPair(pair=current_pair.pair, identity=current_pair.identity, record=record) -def first_unchecked_pair(current_pairs: list[CurrentPair], excluded_pair_keys: set[str]) -> CurrentPair | None: - for current_pair in sorted(current_pairs, key=current_pair_sort_key): - if current_pair.identity.pair_key in excluded_pair_keys: - continue - - if normalized_check_status(current_pair.record) == "unchecked": - return current_pair - - return None - - def wait_for_finished_children(running: dict[str, RunningChildCheck]) -> list[RunningChildCheck]: while True: finished = [child for child in running.values() if child.process.poll() is not None] @@ -1761,31 +2432,57 @@ def terminate_running_children(running: dict[str, RunningChildCheck]) -> None: kill_running_child(child) -def process_current_pairs( +def process_current_frontier( changed_files: list[str], current_pairs: list[CurrentPair], + graphs: DependencyGraphs, *, command_name: str, - jobs: int, + agent_jobs: int, ) -> ExitCode: + selection = select_frontier(graphs, current_pairs) + selected_files = frontier_files(selection) + frontier_pairs = ordered_frontier_pairs(selection, current_pairs) + unchecked_job_count = sum( + normalized_check_status(current_pair.record) == "unchecked" for current_pair in frontier_pairs + ) + log_project_journal( + "step", + f"{command_name} selected frontier artifacts: {', '.join(selected_files) or '(none)'}", + ) + log_project_journal("step", f"{command_name} frontier pair-job count:{unchecked_job_count}") + log_project_journal( + "step", + f"{command_name} deferred non-frontier inconsistencies:{selection.deferred_inconsistencies}", + ) + + inconsistent_pair = first_frontier_pair_with_status(frontier_pairs, "inconsistent") + + if inconsistent_pair is not None: + print_inconsistent_pair(inconsistent_pair) + log_project_journal("step", f"{command_name} outcome: current frontier inconsistency found") + return ExitCode.INCONSISTENCY_FOUND + + if not selection.frontier_components: + print_frontier_summary(changed_files, current_pairs, selection, fresh_cycle_required=False) + log_project_journal("step", f"{command_name} outcome: success after fresh no-work cycle") + return ExitCode.SUCCESS + running: dict[str, RunningChildCheck] = {} inconsistency_found = False - log_project_journal("step", f"{command_name} processing with jobs:{jobs}") + stale_found = False + launched_child = False + log_project_journal("step", f"{command_name} processing with agent-jobs:{agent_jobs}") try: while True: - inconsistent_pair = first_inconsistent_pair(current_pairs) - - if inconsistent_pair is not None: - inconsistency_found = True - - if not running: - print_inconsistent_pair(inconsistent_pair) - log_project_journal("step", f"{command_name} outcome: current inconsistency found") - return ExitCode.INCONSISTENCY_FOUND - - while not inconsistency_found and len(running) < jobs: - unchecked_pair = first_unchecked_pair(current_pairs, set(running)) + while True: + unchecked_pair = next_frontier_candidate( + frontier_pairs, + set(running), + agent_jobs=agent_jobs, + stop_launching=inconsistency_found or stale_found, + ) if unchecked_pair is None: break @@ -1794,7 +2491,16 @@ def process_current_pairs( if normalized_check_status(checked_pair.record) == "outdated": current_pairs = replace_current_pair(current_pairs, checked_pair) - continue + frontier_pairs = replace_current_pair(frontier_pairs, checked_pair) + stale_found = True + log_project_journal( + "step", + ( + f"{command_name} stale frontier detected before launch: " + f"{pair_journal_subject(checked_pair.identity)}" + ), + ) + break try: prepared = prepare_child_check(checked_pair) @@ -1807,35 +2513,89 @@ def process_current_pairs( current_pairs, CurrentPair(pair=checked_pair.pair, identity=checked_pair.identity, record=updated_record), ) - continue + frontier_pairs = replace_current_pair( + frontier_pairs, + CurrentPair(pair=checked_pair.pair, identity=checked_pair.identity, record=updated_record), + ) + stale_found = True + log_project_journal("step", f"{command_name} stale frontier detected before launch") + break except OutdatedPairError as error: updated_record = mark_record_outdated_during_processing(checked_pair.record, error.reason) current_pairs = replace_current_pair( current_pairs, CurrentPair(pair=checked_pair.pair, identity=checked_pair.identity, record=updated_record), ) - continue + frontier_pairs = replace_current_pair( + frontier_pairs, + CurrentPair(pair=checked_pair.pair, identity=checked_pair.identity, record=updated_record), + ) + stale_found = True + log_project_journal("step", f"{command_name} stale frontier detected before launch") + break running[checked_pair.identity.pair_key] = start_child_checker(prepared) + launched_child = True if not running: - inconsistent_pair = first_inconsistent_pair(current_pairs) + outcome = completed_frontier_exit_code( + frontier_pairs, + launched_child=launched_child, + stale_found=stale_found, + ) + + if outcome == ExitCode.CONTINUE_CYCLE and stale_found: + current_selection = select_frontier(graphs, current_pairs) + print_frontier_summary(changed_files, current_pairs, current_selection, fresh_cycle_required=True) + log_project_journal("step", f"{command_name} outcome: stale frontier requires rediscovery") + return outcome + + if outcome == ExitCode.INCONSISTENCY_FOUND: + inconsistent_pair = first_frontier_pair_with_status(frontier_pairs, "inconsistent") + + if inconsistent_pair is None: + raise CheckerFailureError("frontier inconsistency outcome has no inconsistent pair") - if inconsistent_pair is not None: print_inconsistent_pair(inconsistent_pair) - log_project_journal("step", f"{command_name} outcome: child found inconsistency") - return ExitCode.INCONSISTENCY_FOUND + log_project_journal("step", f"{command_name} outcome: frontier child found inconsistency") + return outcome + + if outcome == ExitCode.CONTINUE_CYCLE: + current_selection = select_frontier(graphs, current_pairs) + print_frontier_summary(changed_files, current_pairs, current_selection, fresh_cycle_required=True) + log_project_journal("step", f"{command_name} frontier completed; fresh cycle required") + log_project_journal("step", f"{command_name} outcome: continue after successful frontier") + return outcome - print_summary(changed_files, current_pairs) - log_project_journal("step", f"{command_name} outcome: success") - return ExitCode.SUCCESS + raise CheckerFailureError(f"{command_name} produced unsupported frontier outcome: {outcome}") for finished_child in wait_for_finished_children(running): pair_key = finished_child.prepared.current_pair.identity.pair_key running.pop(pair_key) child_output = finish_child_checker(finished_child) - updated_pair = update_record_from_child_output(finished_child.prepared.current_pair, child_output) + finished_pair = finished_child.prepared.current_pair + stale_reason = record_outdated_reason(finished_pair.record) + + if stale_reason is not None: + updated_record = mark_record_outdated_during_processing(finished_pair.record, stale_reason) + updated_pair = CurrentPair( + pair=finished_pair.pair, + identity=finished_pair.identity, + record=updated_record, + ) + stale_found = True + log_project_journal( + "step", + ( + f"{command_name} stale frontier detected after child completion: " + f"{pair_journal_subject(finished_pair.identity)}" + ), + ) + else: + updated_pair = update_record_from_child_output(finished_pair, child_output) + current_pairs = replace_current_pair(current_pairs, updated_pair) + frontier_pairs = replace_current_pair(frontier_pairs, updated_pair) if normalized_check_status(updated_pair.record) == "inconsistent": inconsistency_found = True @@ -1844,23 +2604,191 @@ def process_current_pairs( raise -def effective_jobs(args: argparse.Namespace) -> int: - jobs = args.jobs if getattr(args, "jobs", None) is not None else get_config().jobs +def effective_agent_jobs(args: argparse.Namespace) -> int: + agent_jobs = ( + args.agent_jobs + if getattr(args, "agent_jobs", None) is not None + else get_config().agent_jobs + ) + + if agent_jobs <= 0: + raise CheckerFailureError("agent_jobs must be positive") + + return agent_jobs + + +def reconcile_changed_files() -> tuple[list[str], list[CurrentPair], DependencyGraphs]: + changed_files = discover_changed_files() + dependency_state = discover_dependency_state(changed_files) + current_pairs = reconcile_queue(list(dependency_state.direct_pairs)) + + return changed_files, current_pairs, dependency_state.graphs + + +def reconcile_direct_changed_files() -> tuple[list[str], list[CurrentPair]]: + changed_files = discover_changed_files() + current_pairs = reconcile_queue(query_depmesh_pairs(changed_files)) + + return changed_files, current_pairs + - if jobs <= 0: - raise CheckerFailureError("jobs must be positive") +def existing_file_artifacts(paths: Iterable[str]) -> list[str]: + return sorted( + path + for path in set(paths) + if artifact_to_filesystem_path(path).is_file() + ) + + +def current_pair_keys_for_records(records: Iterable[CheckRecord]) -> set[str]: + records = list(records) + changed_paths = existing_file_artifacts(record.changed_path for record in records) + + if not changed_paths: + return set() + + relation_pairs = query_depmesh_pairs(changed_paths) + current_pair_keys: set[str] = set() + + for pair in relation_pairs: + try: + current_pair_keys.add(build_pair_identity(pair).pair_key) + except MissingArtifactError: + continue + + return current_pair_keys + + +def filter_current_records(records: Iterable[CheckRecord], current_pair_keys: set[str]) -> list[CheckRecord]: + return [record for record in records if record.pair_key in current_pair_keys] - return jobs + +def mark_records_outside_current_pairs( + records: Iterable[CheckRecord], + current_pair_keys: set[str], +) -> tuple[int, int]: + checked_count = 0 + marked_count = 0 + + for record in records: + if normalized_check_status(record) == "outdated": + continue + + checked_count += 1 + + if record.pair_key in current_pair_keys: + continue + + reason = record_outdated_reason(record) + + if reason is None: + reason = "relation pair is no longer returned by current depmesh relations" + + mark_record_outdated(record, reason) + marked_count += 1 + + return checked_count, marked_count + + +def synchronize_queue_state() -> QueueSyncResult: + changed_files = discover_changed_files() + queued_records = load_allowed_check_records() + active_queued_paths = { + record.changed_path + for record in queued_records + if normalized_check_status(record) != "outdated" + } + tracked_files = existing_file_artifacts([*changed_files, *active_queued_paths]) + relation_pairs = query_depmesh_pairs(tracked_files) if tracked_files else [] + current_pairs = reconcile_queue(relation_pairs) + current_pair_keys = {current_pair.identity.pair_key for current_pair in current_pairs} + checked_records, marked_outdated_records = mark_records_outside_current_pairs( + load_allowed_check_records(), + current_pair_keys, + ) + + return QueueSyncResult( + tracked_files=tuple(tracked_files), + current_pairs=tuple(current_pairs), + checked_records=checked_records, + marked_outdated_records=marked_outdated_records, + ) + + +def print_sync_summary(result: QueueSyncResult) -> None: + counts = status_counts(list(result.current_pairs)) + print("Queue synchronization summary") + print(f"tracked files: {len(result.tracked_files)}") + print(f"current relation pairs: {len(result.current_pairs)}") + print(f"consistent pairs: {counts.get('consistent', 0)}") + print(f"inconsistent pairs: {counts.get('inconsistent', 0)}") + print(f"unchecked pairs: {counts.get('unchecked', 0)}") + print(f"records checked for currentness: {result.checked_records}") + print(f"records marked outdated: {result.marked_outdated_records}") + + if result.tracked_files: + print("tracked file list:") + + for tracked_file in result.tracked_files: + print(f"- {tracked_file}") + + +def sync_queue() -> ExitCode: + ensure_runtime_state() + log_project_journal("step", "sync-queue command started") + result = synchronize_queue_state() + print_sync_summary(result) + log_project_journal( + "step", + ( + "sync-queue command completed " + f"current:{len(result.current_pairs)} marked-outdated:{result.marked_outdated_records}" + ), + ) + + return ExitCode.SUCCESS + + +def enqueue_changed() -> ExitCode: + ensure_runtime_state() + log_project_journal("step", "enqueue-changed command started") + changed_files, current_pairs = reconcile_direct_changed_files() + print_summary(changed_files, current_pairs) + log_project_journal("step", "enqueue-changed outcome: queue reconciled without processing") + + return ExitCode.SUCCESS def run_cycle(args: argparse.Namespace) -> ExitCode: ensure_runtime_state() log_project_journal("step", "run-cycle command started") + changed_files, current_pairs, graphs = reconcile_changed_files() + + return process_current_frontier( + changed_files, + current_pairs, + graphs, + command_name="run-cycle", + agent_jobs=effective_agent_jobs(args), + ) + + +def show_frontier() -> ExitCode: + log_project_journal("step", "frontier diagnostic command started") changed_files = discover_changed_files() - relation_pairs = query_depmesh_pairs(changed_files) - current_pairs = reconcile_queue(relation_pairs) + dependency_state = discover_dependency_state(changed_files) + current_pairs = current_pairs_read_only( + dependency_state.direct_pairs, + load_allowed_check_records_read_only(), + ) + selection = select_frontier(dependency_state.graphs, current_pairs) + selected_files = frontier_files(selection) + print(build_frontier_report(changed_files, selection)) + + log_project_journal("step", f"frontier diagnostic selected-file count:{len(selected_files)}") + log_project_journal("step", "frontier diagnostic outcome: success") - return process_current_pairs(changed_files, current_pairs, command_name="run-cycle", jobs=effective_jobs(args)) + return ExitCode.SUCCESS def record_to_current_pair(record: CheckRecord, relation_descriptions: dict[str, str]) -> CurrentPair: @@ -1880,10 +2808,7 @@ def load_queued_current_pairs() -> list[CurrentPair]: relation_descriptions = { relation_id: relation.description for relation_id, relation in config.relations.items() } - records = [ - mark_record_outdated_if_needed(raw_record_to_check_record(record)) - for record in load_taskwarrior_records() - ] + records = [mark_record_outdated_if_needed(record) for record in load_allowed_check_records()] current_pairs = [ record_to_current_pair(record, relation_descriptions) for record in records @@ -1897,23 +2822,22 @@ def load_queued_current_pairs() -> list[CurrentPair]: def process_queue(args: argparse.Namespace) -> ExitCode: ensure_runtime_state() log_project_journal("step", "process-queue command started") - current_pairs = load_queued_current_pairs() - changed_files = sorted({current_pair.pair.changed_path for current_pair in current_pairs}) - - return process_current_pairs(changed_files, current_pairs, command_name="process-queue", jobs=effective_jobs(args)) + changed_files, current_pairs, graphs = reconcile_changed_files() + + return process_current_frontier( + changed_files, + current_pairs, + graphs, + command_name="process-queue", + agent_jobs=effective_agent_jobs(args), + ) def mark_outdated_records() -> tuple[int, int]: checked_count = 0 marked_count = 0 - allowed_relations = set(get_config().allowed_file_relations) - - for raw_record in load_taskwarrior_records(): - if not raw_record.get("pair_key") or raw_record.get("relation") not in allowed_relations: - continue - + for record in load_allowed_check_records(): checked_count += 1 - record = raw_record_to_check_record(raw_record) reason = record_outdated_reason(record) if reason is None: @@ -1995,14 +2919,17 @@ def report_progress(path: str) -> ExitCode: return ExitCode.SUCCESS -def load_list_pair_records(statuses: Iterable[str] = ()) -> list[CheckRecord]: +def load_list_pair_records( + statuses: Iterable[str] = (), + *, + current_only: bool = False, +) -> list[CheckRecord]: status_filter = set(statuses) - allowed_relations = set(get_config().allowed_file_relations) - records = [ - raw_record_to_check_record(record) - for record in load_taskwarrior_records() - if record.get("pair_key") and record.get("relation") in allowed_relations - ] + records = load_allowed_check_records() + + if current_only: + records = filter_current_records(records, current_pair_keys_for_records(records)) + if status_filter: records = [record for record in records if (record.check_status or "unknown") in status_filter] @@ -2061,7 +2988,7 @@ def format_list_pair_record_multi_line(record: CheckRecord, options: ListPairsOp def build_list_pairs_report(options: ListPairsOptions | None = None) -> str: options = options or ListPairsOptions() - records = load_list_pair_records(options.statuses) + records = load_list_pair_records(options.statuses, current_only=options.current_only) lines: list[str] = [] for index, record in enumerate(records): @@ -2088,12 +3015,16 @@ def list_pairs(args: argparse.Namespace) -> ExitCode: multi_line=bool(args.multi_line), include_report=bool(args.report), include_all_fields=bool(args.all), + current_only=bool(args.current), statuses=tuple(args.statuses or ()), include_count=not bool(args.no_count), ) log_project_journal( "step", - f"list-pairs command started statuses:{','.join(options.statuses) or 'all'}", + ( + f"list-pairs command started statuses:{','.join(options.statuses) or 'all'} " + f"current-only:{options.current_only}" + ), ) report = build_list_pairs_report(options) @@ -2249,6 +3180,404 @@ def self_check_child_output(check_status: str, report: str) -> str: return json.dumps(payload) +def synthetic_current_pair( + changed_path: str, + *, + status: str, + relation: str = "synthetic-relation", + related_path: str | None = None, +) -> CurrentPair: + related_path = related_path or f"@/validation/{changed_path.removeprefix('@/')}" + checksum_changed = f"checksum:{changed_path}" + checksum_related = f"checksum:{related_path}" + file_pair = f"<{changed_path}|{checksum_changed}>:<{related_path}|{checksum_related}>" + identity = PairIdentity( + pair_key=f"{relation}|{file_pair}", + file_pair=file_pair, + changed_path=changed_path, + related_path=related_path, + relation=relation, + checksum_changed=checksum_changed, + checksum_related=checksum_related, + ) + pair = RelationPair( + changed_path=changed_path, + related_path=related_path, + relation=relation, + relation_description="Synthetic self-check relation", + ) + record = CheckRecord( + uuid="synthetic", + pair_key=identity.pair_key, + file_pair=file_pair, + changed_path=changed_path, + related_path=related_path, + relation=relation, + checksum_changed=checksum_changed, + checksum_related=checksum_related, + check_status=status, + report="## Synthetic inconsistency" if status == "inconsistent" else "", + checked_at="", + ) + + return CurrentPair(pair=pair, identity=identity, record=record) + + +def run_dependency_scheduler_self_checks() -> None: + root = "@/changed/root" + middle = "@/changed/middle" + leaf = "@/changed/leaf" + chain_graph = build_dependency_graphs( + [leaf, root, middle], + {leaf: [middle], root: [], middle: [root]}, + ) + chain_pairs = [ + synthetic_current_pair(path, status="unchecked") + for path in [leaf, root, middle] + ] + chain_selection = select_frontier(chain_graph, chain_pairs) + assert_self_check(frontier_files(chain_selection) == (root,), "simple chain must select only its root") + + advanced_pairs = [ + synthetic_current_pair(root, status="consistent"), + synthetic_current_pair(middle, status="unchecked"), + synthetic_current_pair(leaf, status="unchecked"), + ] + assert_self_check( + frontier_files(select_frontier(chain_graph, advanced_pairs)) == (middle,), + "a resolved root must advance the next cycle to the middle", + ) + + independent = "@/changed/independent" + independent_graph = build_dependency_graphs([root, independent], {root: [], independent: []}) + independent_pairs = [ + synthetic_current_pair(root, status="unchecked"), + synthetic_current_pair(independent, status="unchecked"), + ] + assert_self_check( + frontier_files(select_frontier(independent_graph, independent_pairs)) == tuple(sorted((root, independent))), + "independent roots must share a frontier", + ) + + left = "@/changed/left" + right = "@/changed/right" + descendant = "@/changed/descendant" + diamond_graph = build_dependency_graphs( + [descendant, left, right], + {descendant: [right, left], left: [], right: []}, + ) + diamond_pairs = [ + synthetic_current_pair(left, status="consistent"), + synthetic_current_pair(right, status="unchecked"), + synthetic_current_pair(descendant, status="unchecked"), + ] + assert_self_check( + frontier_files(select_frontier(diamond_graph, diamond_pairs)) == (right,), + "a diamond descendant must wait for both parents", + ) + + intermediary = "@/unchanged/intermediary" + intermediary_graph = build_dependency_graphs( + [root, leaf], + {root: [], intermediary: [root], leaf: [intermediary]}, + ) + assert_self_check( + intermediary_graph.scheduling_edges == (((root,), (leaf,)),), + "unchanged intermediaries must contract into derived scheduling edges", + ) + assert_self_check( + intermediary not in {path for component in intermediary_graph.scheduling_components for path in component}, + "unchanged intermediaries must not become scheduling work", + ) + + unchanged_upstream_graph = build_dependency_graphs( + [leaf], + {intermediary: [], leaf: [intermediary]}, + ) + assert_self_check( + frontier_files( + select_frontier( + unchanged_upstream_graph, + [synthetic_current_pair(leaf, status="unchecked")], + ) + ) + == (leaf,), + "an unchanged upstream dependency must not block the first changed descendant", + ) + + self_edge_graph = build_dependency_graphs([root], {root: [root]}) + assert_self_check(self_edge_graph.ignored_self_edges == 1, "self-edges must be counted and ignored") + assert_self_check( + frontier_files(select_frontier(self_edge_graph, [synthetic_current_pair(root, status="unchecked")])) + == (root,), + "a self-edge must not deadlock frontier selection", + ) + + cycle_a = "@/changed/cycle-a" + cycle_b = "@/changed/cycle-b" + cycle_graph = build_dependency_graphs( + [cycle_b, cycle_a], + {cycle_a: [cycle_b], cycle_b: [cycle_a]}, + ) + cycle_selection = select_frontier( + cycle_graph, + [ + synthetic_current_pair(cycle_b, status="unchecked"), + synthetic_current_pair(cycle_a, status="unchecked"), + ], + ) + assert_self_check( + cycle_graph.scheduling_components == ((cycle_a, cycle_b),), + "changed cycle members must collapse into one scheduling component", + ) + assert_self_check( + frontier_files(cycle_selection) == (cycle_a, cycle_b), + "cycle members must be scheduled atomically and reported lexically", + ) + cycle_report = build_frontier_report([cycle_b, cycle_a], cycle_selection) + assert_self_check( + cycle_report.splitlines()[-2:] == [f"- {cycle_a}", f"- {cycle_b}"], + "frontier output must flatten cycle members into unique lexical paths", + ) + + deferred_pairs = [ + synthetic_current_pair(root, status="unchecked"), + synthetic_current_pair(middle, status="consistent"), + synthetic_current_pair(leaf, status="inconsistent"), + ] + deferred_selection = select_frontier(chain_graph, deferred_pairs) + assert_self_check( + frontier_files(deferred_selection) == (root,) and deferred_selection.deferred_inconsistencies == 1, + ( + "a descendant inconsistency must be deferred behind its pending predecessor " + f"frontier={frontier_files(deferred_selection)} " + f"deferred={deferred_selection.deferred_inconsistencies}" + ), + ) + + frontier_inconsistent_pairs = [ + synthetic_current_pair(root, status="inconsistent"), + synthetic_current_pair(middle, status="unchecked"), + synthetic_current_pair(leaf, status="unchecked"), + ] + frontier_inconsistent_selection = select_frontier(chain_graph, frontier_inconsistent_pairs) + frontier_pair_set = [ + pair + for pair in frontier_inconsistent_pairs + if pair.pair.changed_path in set(frontier_files(frontier_inconsistent_selection)) + ] + assert_self_check( + first_inconsistent_pair(frontier_pair_set) is not None, + "an existing frontier inconsistency must be found before a child launch", + ) + + cycle_order_a = "@/changed/a-cycle" + cycle_order_z = "@/changed/z-cycle" + independent_order_b = "@/changed/b-independent" + component_order_graph = build_dependency_graphs( + [cycle_order_a, cycle_order_z, independent_order_b], + { + cycle_order_a: [cycle_order_z], + cycle_order_z: [cycle_order_a], + independent_order_b: [], + }, + ) + component_order_pairs = [ + synthetic_current_pair(cycle_order_a, status="consistent"), + synthetic_current_pair(cycle_order_z, status="inconsistent"), + synthetic_current_pair(independent_order_b, status="inconsistent"), + ] + component_order_selection = select_frontier(component_order_graph, component_order_pairs) + ordered_pairs = ordered_frontier_pairs(component_order_selection, component_order_pairs) + first_component_inconsistency = first_frontier_pair_with_status(ordered_pairs, "inconsistent") + assert_self_check( + first_component_inconsistency is not None + and first_component_inconsistency.pair.changed_path == cycle_order_z, + "frontier inconsistency order must prioritize component order before artifact path", + ) + + concurrency_candidates = sorted(independent_pairs, key=current_pair_sort_key) + first_batch = concurrency_candidates[: get_config().agent_jobs] + assert_self_check( + len(first_batch) <= get_config().agent_jobs, + "frontier concurrency must not exceed the resolved agent_jobs value", + ) + assert_self_check( + next_frontier_candidate( + concurrency_candidates, + {concurrency_candidates[0].identity.pair_key}, + agent_jobs=1, + stop_launching=False, + ) + is None, + "the scheduler must not launch above its resolved concurrency limit", + ) + assert_self_check( + next_frontier_candidate( + concurrency_candidates, + set(), + agent_jobs=get_config().agent_jobs, + stop_launching=True, + ) + is None, + "an inconsistency or stale result must stop later frontier launches", + ) + assert_self_check( + root in frontier_files(chain_selection) and middle not in frontier_files(chain_selection), + "a descendant must not cross the invocation frontier boundary", + ) + + no_pair_graph = build_dependency_graphs([root, leaf], {root: [], leaf: [root]}) + no_pair_selection = select_frontier( + no_pair_graph, + [synthetic_current_pair(leaf, status="unchecked")], + ) + assert_self_check( + frontier_files(no_pair_selection) == (leaf,), + "a changed artifact without validation pairs must be resolved for scheduling", + ) + all_resolved_selection = select_frontier( + chain_graph, + [synthetic_current_pair(path, status="consistent") for path in [leaf, middle, root]], + ) + assert_self_check( + not frontier_files(all_resolved_selection), + "a fresh all-resolved pass must have no frontier", + ) + assert_self_check( + build_frontier_report([leaf, middle, root], all_resolved_selection).endswith("frontier files: 0"), + "an empty frontier report must succeed without path entries", + ) + deferred_report = build_frontier_report([leaf, middle, root], deferred_selection) + assert_self_check( + f"- {leaf}" not in deferred_report, + "frontier output must omit blocked descendants", + ) + completed_pairs = [synthetic_current_pair(root, status="consistent")] + assert_self_check( + completed_frontier_exit_code(completed_pairs, launched_child=True, stale_found=False) + == ExitCode.CONTINUE_CYCLE, + "a successful frontier must require one fresh cycle", + ) + assert_self_check( + completed_frontier_exit_code( + [synthetic_current_pair(root, status="inconsistent")], + launched_child=True, + stale_found=False, + ) + == ExitCode.INCONSISTENCY_FOUND, + "a valid frontier inconsistency must return the repair exit code", + ) + assert_self_check( + completed_frontier_exit_code( + [synthetic_current_pair(root, status="inconsistent")], + launched_child=True, + stale_found=True, + ) + == ExitCode.CONTINUE_CYCLE, + "stale frontier work must force rediscovery before reporting a concurrent inconsistency", + ) + + shuffled_graph = build_dependency_graphs( + [middle, leaf, root], + {middle: [root], leaf: [middle], root: []}, + ) + shuffled_pairs = list(reversed(chain_pairs)) + assert_self_check(shuffled_graph == chain_graph, "shuffled graph inputs must be deterministic") + assert_self_check( + frontier_files(select_frontier(shuffled_graph, shuffled_pairs)) == frontier_files(chain_selection), + "shuffled queue inputs must select the same frontier", + ) + + configured_relation_ids = get_config().allowed_file_relations + assert_self_check(configured_relation_ids, "the resolved allowed relation list must not be empty") + assert_self_check( + all(relation_id in get_config().relations for relation_id in configured_relation_ids), + "every mode-resolved allowed relation must have config-defined criteria", + ) + synthetic_relations = ["synthetic-alpha", "synthetic-beta"] + synthetic_relation_pairs = [ + synthetic_current_pair(root, status="unchecked", relation=relation_id, related_path=intermediary) + for relation_id in synthetic_relations + ] + synthetic_adjacency = { + root: sorted({pair.pair.related_path for pair in synthetic_relation_pairs}), + intermediary: [], + } + assert_self_check( + build_dependency_graphs([root], synthetic_adjacency).traversal_edges == ((intermediary, root),), + "synthetic configured relation ids must contribute uniformly to one deduplicated graph edge", + ) + + discovery_relation = DepmeshRelation( + relation_id="synthetic-discovery", + description="Synthetic parallel discovery relation", + ) + discovery_paths = ("@/discovery/a", "@/discovery/b") + discovery_barrier = Barrier(len(discovery_paths)) + discovery_lock = Lock() + active_discovery_queries = 0 + maximum_discovery_queries = 0 + + def synthetic_discovery_query( + changed_path: str, + relation: DepmeshRelation, + ) -> list[dict[str, Any]]: + nonlocal active_discovery_queries, maximum_discovery_queries + + with discovery_lock: + active_discovery_queries += 1 + maximum_discovery_queries = max(maximum_discovery_queries, active_discovery_queries) + + try: + discovery_barrier.wait(timeout=2) + finally: + with discovery_lock: + active_discovery_queries -= 1 + + return [ + { + "type": "dependency", + "relation": relation.relation_id, + "dependency": f"@/dependency/{changed_path.removeprefix('@/').replace('/', '-')}", + } + ] + + parallel_discovery_pairs = query_artifacts_pairs( + reversed(discovery_paths), + [discovery_relation], + discovery_jobs=len(discovery_paths), + query_records=synthetic_discovery_query, + ) + assert_self_check( + maximum_discovery_queries == len(discovery_paths), + "depmesh discovery must execute concurrently up to discovery_jobs", + ) + + def deterministic_discovery_query( + changed_path: str, + relation: DepmeshRelation, + ) -> list[dict[str, Any]]: + return [ + { + "type": "dependency", + "relation": relation.relation_id, + "dependency": f"@/dependency/{changed_path.removeprefix('@/').replace('/', '-')}", + } + ] + + sequential_discovery_pairs = query_artifacts_pairs( + discovery_paths, + [discovery_relation], + discovery_jobs=1, + query_records=deterministic_discovery_query, + ) + assert_self_check( + parallel_discovery_pairs == sequential_discovery_pairs, + "parallel depmesh discovery must preserve deterministic pair results", + ) + + def run_self_check() -> ExitCode: ensure_runtime_state() log_project_journal("step", "self-check command started") @@ -2259,19 +3588,21 @@ def run_self_check() -> ExitCode: active_config = get_config() assert_self_check( - active_config.allowed_file_relations == ("governed_by",), - "allowed relations must load from config", + bool(active_config.allowed_file_relations), + "allowed relations must load from config without a source-code relation allowlist", ) - assert_self_check(active_config.jobs > 0, "jobs must load from config") + assert_self_check(active_config.agent_jobs > 0, "agent_jobs must load from config") + assert_self_check(active_config.discovery_jobs > 0, "discovery_jobs must load from config") + allowed_relation = active_config.allowed_file_relations[0] assert_self_check( - relation_specific_criteria("governed_by")[0].startswith("The implementation"), - "relation criteria must load from config", + bool(relation_specific_criteria(allowed_relation)), + "configured relation criteria must load from config", ) + run_dependency_scheduler_self_checks() changed_files = discover_changed_files() assert_self_check(all(path.startswith("@/") for path in changed_files), "changed files must be artifact ids") paths = runtime_paths() - allowed_relation = get_config().allowed_file_relations[0] changed_path = runtime_artifact_path("self-check", "changed.txt") related_path = runtime_artifact_path("self-check", "related.txt") second_related_path = runtime_artifact_path("self-check", "second-related.txt") @@ -2317,6 +3648,43 @@ def run_self_check() -> ExitCode: all(current_pair.record.check_status == "unchecked" for current_pair in current_pairs), "new current pairs must be unchecked", ) + records_before_read_only = load_taskwarrior_records() + child_runtime_files_before = tuple( + sorted( + path.relative_to(paths.runtime_dir).as_posix() + for directory in [paths.agent_output_dir, paths.prompt_dir, paths.schema_dir] + for path in directory.rglob("*") + if path.is_file() + ) + ) + read_only_pairs = current_pairs_read_only([pair, second_pair], [current_pairs[0].record]) + records_after_read_only = load_taskwarrior_records() + child_runtime_files_after = tuple( + sorted( + path.relative_to(paths.runtime_dir).as_posix() + for directory in [paths.agent_output_dir, paths.prompt_dir, paths.schema_dir] + for path in directory.rglob("*") + if path.is_file() + ) + ) + virtual_pair = next(item for item in read_only_pairs if item.identity.pair_key == second_identity.pair_key) + assert_self_check( + virtual_pair.record.check_status == "unchecked" and not virtual_pair.record.uuid, + "a current pair missing from the queue must be virtual unchecked", + ) + assert_self_check( + records_before_read_only == records_after_read_only, + "read-only current-pair derivation must not mutate queue records", + ) + assert_self_check( + child_runtime_files_before == child_runtime_files_after, + "read-only frontier derivation must not create child-runtime artifacts", + ) + read_only_graph = build_dependency_graphs([changed_path], {changed_path: []}) + assert_self_check( + frontier_files(select_frontier(read_only_graph, read_only_pairs)) == (changed_path,), + "read-only frontier selection must match reconciled unchecked status", + ) prepared_self_check = prepare_child_check(current_pairs[0]) serialized_schema = json.loads(prepared_self_check.schema_path.read_text(encoding="utf-8")) rendered_agent_cmd = render_command_argv( @@ -2376,6 +3744,10 @@ def run_self_check() -> ExitCode: "malformed output must produce a markdown issue section", ) changed_file.write_text("changed self-check content v2\n", encoding="utf-8") + assert_self_check( + record_outdated_reason(current_pairs[0].record) is not None, + "a checksum change after a child snapshot must make its result stale", + ) checked_count, marked_count = mark_outdated_records() outdated_raw_record = find_raw_record_by_pair_key(load_taskwarrior_records(), identity.pair_key) assert_self_check(outdated_raw_record is not None, "outdated source pair record must still exist") @@ -2395,6 +3767,13 @@ def run_self_check() -> ExitCode: reset_self_check_record(changed_identity) changed_current_pair = reconcile_queue([pair])[0] assert_self_check(changed_current_pair.record.check_status == "unchecked", "changed checksum must force unchecked") + superseded_raw_record = find_raw_record_by_pair_key(load_taskwarrior_records(), identity.pair_key) + assert_self_check(superseded_raw_record is not None, "superseded checksum record must remain as history") + superseded_record = raw_record_to_check_record(superseded_raw_record) + assert_self_check( + superseded_record.check_status == "outdated", + "reconciliation must eagerly mark an older checksum version outdated", + ) explicitly_consistent_pair = set_relation_pair_check_status( pair, check_status="consistent", @@ -2466,6 +3845,26 @@ def run_self_check() -> ExitCode: == "records: 0", "list-pairs status filter must filter records", ) + current_filtered_records = filter_current_records( + load_allowed_check_records(), + {changed_identity.pair_key}, + ) + assert_self_check( + [record.pair_key for record in current_filtered_records] == [changed_identity.pair_key], + "current-only filtering must retain only current pair keys", + ) + checked_current_records, marked_removed_records = mark_records_outside_current_pairs( + [explicitly_inconsistent_pair.record], + set(), + ) + assert_self_check(checked_current_records == 1, "queue synchronization must check active pair records") + assert_self_check(marked_removed_records == 1, "queue synchronization must mark removed relations outdated") + removed_raw_record = find_raw_record_by_pair_key(load_taskwarrior_records(), changed_identity.pair_key) + assert_self_check(removed_raw_record is not None, "removed relation record must remain as history") + assert_self_check( + raw_record_to_check_record(removed_raw_record).check_status == "outdated", + "removed relation record must be outdated", + ) pair_records = load_taskwarrior_records() project_journal = json.loads( run_command( @@ -2483,8 +3882,9 @@ def run_self_check() -> ExitCode: "pair record must not be written to project journal DB", ) assert_self_check( - any(record.get("description") == "self-check command started" for record in project_journal), - "script operations must log to the project journal", + active_config.journal_cmd is None + or any(record.get("description") == "self-check command started" for record in project_journal), + "script operations must log to the project journal when journaling is configured", ) assert_self_check( all("journal" not in record.get("tags", []) for record in pair_records if record.get("pair_key")), @@ -2528,11 +3928,13 @@ def add_pair_status_arguments(parser: argparse.ArgumentParser) -> None: ) -def add_jobs_argument(parser: argparse.ArgumentParser) -> None: +def add_agent_jobs_argument(parser: argparse.ArgumentParser) -> None: parser.add_argument( + "--agent-jobs", "--jobs", + dest="agent_jobs", type=int, - help="number of child agent checks to keep running; defaults to consistency.toml jobs", + help="number of child agent checks to keep running; defaults to consistency.toml agent_jobs", ) @@ -2544,17 +3946,32 @@ def parse_args() -> argparse.Namespace: ) subparsers = parser.add_subparsers(dest="command", required=True) + subparsers.add_parser( + "enqueue-changed", + help="enqueue relation pairs for all Git-changed files without processing them", + ) + + subparsers.add_parser( + "sync-queue", + help="synchronize current relation pairs and mark stale or removed pairs outdated without processing", + ) + run_cycle_parser = subparsers.add_parser( "run-cycle", - help="reconcile and check relation pairs until completion or inconsistency", + help="reconcile and process one dependency-ready frontier", ) - add_jobs_argument(run_cycle_parser) + add_agent_jobs_argument(run_cycle_parser) process_queue_parser = subparsers.add_parser( "process-queue", - help="process queued relation pairs until completion or inconsistency", + help="rediscover, reconcile, and process one dependency-ready frontier", + ) + add_agent_jobs_argument(process_queue_parser) + + subparsers.add_parser( + "frontier", + help="show the current dependency-ready changed files without reconciling the queue", ) - add_jobs_argument(process_queue_parser) enqueue_parser = subparsers.add_parser("enqueue", help="manually enqueue one file's depmesh relation pairs") enqueue_parser.add_argument("files", nargs="*", help="project paths or root-anchored artifact ids") @@ -2584,6 +4001,11 @@ def parse_args() -> argparse.Namespace: action="store_true", help="include all stored relation-pair fields", ) + list_pairs_parser.add_argument( + "--current", + action="store_true", + help="include only records matching current file checksums and depmesh relations", + ) list_pairs_parser.add_argument( "--status", action="append", @@ -2625,12 +4047,21 @@ def main() -> int: args = parse_args() configure_consistency(mode=args.mode) + if args.command == "enqueue-changed": + return int(enqueue_changed()) + + if args.command == "sync-queue": + return int(sync_queue()) + if args.command == "run-cycle": return int(run_cycle(args)) if args.command == "process-queue": return int(process_queue(args)) + if args.command == "frontier": + return int(show_frontier()) + if args.command == "enqueue": return int(enqueue_files(parse_enqueue_files(args))) diff --git a/changes/unreleased.md b/changes/unreleased.md index 373eb997..04878eb4 100644 --- a/changes/unreleased.md +++ b/changes/unreleased.md @@ -1,2 +1,12 @@ -No changes. +### Migration + +Run migrations `ffun migrate`. + +### Changes + +- ff-639 — Implemented `ffun.locks` module as a universal distributed lock manager for backend modules. +- ff-639 — Implemented `ffun.audit` module to record and query important operations in the backend. +- ff-639 — Implemented `ffun.entitlements` module to manage user entitlements. + - Two entitlement types are introduced: `day_tokens` and `month_tokens`. + - Implemented CLI `ffun entitlements` to manage user entitlements. diff --git a/consistency.toml b/consistency.toml index 05ea5b96..8d88eb0b 100644 --- a/consistency.toml +++ b/consistency.toml @@ -5,14 +5,58 @@ mode = "incremental" runtime_dir = ".session/inconsistency-check" comparison_base_refs = ["main", "origin/main"] allowed_file_relations = ["governed_by"] -jobs = 10 +# Maximum concurrent child Codex consistency checks. +agent_jobs = 20 +# Maximum concurrent depmesh subprocesses in one dependency-discovery wave. +discovery_jobs = 20 [journal] -cmd = ["./bin/taskwarior.sh", "log", "+journal", "+consistency", "kind:{kind}", "{message}"] - +# Disabled because recursive dependency traversal can produce hundreds of subprocess-backed journal writes. +# Uncomment to restore detailed consistency-check journaling. +# cmd = ["./bin/taskwarior.sh", "log", "+journal", "+consistency", "kind:{kind}", "{message}"] + +# Child-model options, ordered by estimated consistency-check quality. +# Estimates are specific to this project's diff-to-specification workload. Relative cost uses +# gpt-5.6-terra with high reasoning as the 1x baseline; actual speed and cost depend on input size, +# tool use, reasoning tokens, rate limits, and the configured job concurrency. +# +# | Rank | Configuration | Estimated quality | Estimated speed | Estimated cost | Relative cost | Best use | +# | ---: | --- | --- | --- | --- | ---: | --- | +# | 1 | gpt-5.6-sol + xhigh | Maximum, ~5/5 | Very slow | Extreme | ~2.5-4x | Exceptional quality-first audits | +# | 2 | gpt-5.6-sol + high | Very high, ~4.8/5 | Slow | Very high | ~2x | Critical concurrency, transaction, or migration work | +# | 3 | gpt-5.6-terra + high (RECOMMENDED) | High, ~4.4/5 | Moderate | High | 1x | Regular project-wide consistency checking | +# | 4 | gpt-5.6-terra + medium | Good-high, ~4/5 | Fast | Medium | ~0.6-0.8x | Routine changes after validating checker recall | +# | 5 | gpt-5.6-luna + high | Good, ~3.7/5 | Fast to very fast | Low | ~0.35-0.5x | Large cost-sensitive runs with accepted extra risk | +# +# Current model positioning and API-price proxy: https://developers.openai.com/api/docs/models/compare +# +# Child command argument reference: +# - `codex exec` runs Codex non-interactively for one relation-pair check. +# - `--strict-config` rejects configuration keys unknown to the installed Codex version. +# - `--model gpt-5.6-terra` pins the balanced model instead of inheriting a user default. +# - `-c model_reasoning_effort="high"` gives the checker enough depth for semantic diff attribution, +# transactions, concurrency, temporal behavior, and cross-artifact contracts. +# - `-c model_verbosity="low"` keeps the schema-constrained final JSON concise. +# - `-c model_reasoning_summary="none"` suppresses reasoning summaries that this script does not consume. +# - `-c personality="none"` removes conversational style from the machine-oriented checker response. +# - `-c web_search="disabled"` keeps checks deterministic and local to repository evidence. +# - `--cd {project_root}` makes the repository root the child's working directory. +# - `--sandbox read-only` permits repository inspection but prevents child-agent edits. +# - `-c approval_policy="never"` prevents an unattended child from waiting for interactive approval. +# - `--ephemeral` avoids persisting a reusable Codex session for each relation pair. +# - `--output-schema {schema_path}` constrains the final response to the generated checker JSON schema. +# - `--output-last-message {output_path}` writes the final structured response where the script expects it. +# - `-` reads the generated checker prompt from standard input and therefore must remain the last argument. [agent] cmd = [ "codex", "exec", + "--strict-config", + "--model", "gpt-5.6-terra", + "-c", "model_reasoning_effort=\"high\"", + "-c", "model_verbosity=\"low\"", + "-c", "model_reasoning_summary=\"none\"", + "-c", "personality=\"none\"", + "-c", "web_search=\"disabled\"", "--cd", "{project_root}", "--sandbox", "read-only", "-c", "approval_policy=\"never\"", @@ -21,6 +65,7 @@ cmd = [ "--output-last-message", "{output_path}", "-" ] +# Maximum wall-clock time for one child process; this is enforced by the checker script, not Codex. timeout_seconds = 3600 [prompt] diff --git a/depmesh.toml b/depmesh.toml index 5a04ad3d..3c0be903 100644 --- a/depmesh.toml +++ b/depmesh.toml @@ -118,6 +118,54 @@ output = { type = "union", items = [ { type = "files", pattern = "@/specs/**/*.md" }, ] } +# CLI command-family specifications are governed by shared CLI behavior. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/specs/behavior/cli/*.md" } +output = { type = "list", artifacts = ["@/specs/behavior/cli.md"] } + +# Shared CLI behavior governs every command-family specification. +[[rules]] +relation = "governs" +input = { type = "one_of", artifacts = ["@/specs/behavior/cli.md"] } +output = { type = "files", pattern = "@/specs/behavior/cli/*.md" } + +# CLI command-family specifications are governed by matching backend module specifications when they exist. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/specs/behavior/cli/{*module}.md" } +output = { type = "files", pattern = "@/specs/backend_architecture/modules/{module}.md" } + +# Backend module specifications govern matching CLI command-family specifications when they exist. +[[rules]] +relation = "governs" +input = { type = "glob", pattern = "@/specs/backend_architecture/modules/{*module}.md" } +output = { type = "files", pattern = "@/specs/behavior/cli/{module}.md" } + +# Backend module specifications are additionally governed by their common structure requirements. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/specs/backend_architecture/modules/*.md" } +output = { type = "list", artifacts = ["@/specs/meta/backend_modules.md"] } + +# The backend module specification requirements govern every backend module specification. +[[rules]] +relation = "governs" +input = { type = "one_of", artifacts = ["@/specs/meta/backend_modules.md"] } +output = { type = "files", pattern = "@/specs/backend_architecture/modules/*.md" } + +# Backend module specifications are additionally governed by the backend database architecture. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/specs/backend_architecture/modules/*.md" } +output = { type = "list", artifacts = ["@/specs/backend_architecture/db.md"] } + +# The backend database architecture governs every backend module specification. +[[rules]] +relation = "governs" +input = { type = "one_of", artifacts = ["@/specs/backend_architecture/db.md"] } +output = { type = "files", pattern = "@/specs/backend_architecture/modules/*.md" } + # Backend Python files are governed by all backend architecture specs. [[rules]] relation = "governed_by" @@ -130,6 +178,54 @@ relation = "governs" input = { type = "glob", pattern = "@/specs/backend_architecture/*.md" } output = { type = "files", pattern = "@/ffun/ffun/**/*.py" } +# Backend CLI Python files are additionally governed by shared CLI behavior. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/ffun/ffun/cli/**/*.py" } +output = { type = "list", artifacts = ["@/specs/behavior/cli.md"] } + +# Shared CLI behavior governs all backend CLI Python files. +[[rules]] +relation = "governs" +input = { type = "one_of", artifacts = ["@/specs/behavior/cli.md"] } +output = { type = "files", pattern = "@/ffun/ffun/cli/**/*.py" } + +# CLI command modules are additionally governed by matching command-family specifications when they exist. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/ffun/ffun/cli/commands/{*command}.py" } +output = { type = "files", pattern = "@/specs/behavior/cli/{command}.md" } + +# CLI command-family specifications govern matching command modules when they exist. +[[rules]] +relation = "governs" +input = { type = "glob", pattern = "@/specs/behavior/cli/{*command}.md" } +output = { type = "files", pattern = "@/ffun/ffun/cli/commands/{command}.py" } + +# CLI command tests are additionally governed by matching command-family specifications when they exist. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/ffun/ffun/cli/commands/tests/test_{*command}.py" } +output = { type = "files", pattern = "@/specs/behavior/cli/{command}.md" } + +# CLI command-family specifications govern matching command tests when they exist. +[[rules]] +relation = "governs" +input = { type = "glob", pattern = "@/specs/behavior/cli/{*command}.md" } +output = { type = "files", pattern = "@/ffun/ffun/cli/commands/tests/test_{command}.py" } + +# Backend module Python files are additionally governed by a matching module spec when one exists. +[[rules]] +relation = "governed_by" +input = { type = "glob", pattern = "@/ffun/ffun/{*module}/**/*.py" } +output = { type = "files", pattern = "@/specs/backend_architecture/modules/{module}.md" } + +# A backend module spec governs all Python files in the matching module. +[[rules]] +relation = "governs" +input = { type = "glob", pattern = "@/specs/backend_architecture/modules/{*module}.md" } +output = { type = "files", pattern = "@/ffun/ffun/{module}/**/*.py" } + # Frontend source files are governed by all frontend architecture specs. [[rules]] relation = "governed_by" diff --git a/ffun/ffun/audit/__init__.py b/ffun/ffun/audit/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ffun/ffun/audit/domain.py b/ffun/ffun/audit/domain.py new file mode 100644 index 00000000..9a3512d8 --- /dev/null +++ b/ffun/ffun/audit/domain.py @@ -0,0 +1,5 @@ +from ffun.audit import operations + +new_audit_record_id = operations.new_audit_record_id +record = operations.record +load_records_for_subject = operations.load_records_for_subject diff --git a/ffun/ffun/audit/entities.py b/ffun/ffun/audit/entities.py new file mode 100644 index 00000000..748b9a0f --- /dev/null +++ b/ffun/ffun/audit/entities.py @@ -0,0 +1,28 @@ +import datetime +import enum +import uuid +from typing import NewType + +from ffun.core.entities import BaseEntity +from ffun.domain.entities import SerializedId + +AuditRecordId = NewType("AuditRecordId", uuid.UUID) +AuditEventName = NewType("AuditEventName", str) + + +class AuditEntityKind(enum.IntEnum): + user = 1 + admin = 2 + psp = 3 + system = 4 + + +class AuditRecord(BaseEntity): + id: AuditRecordId + created_at: datetime.datetime + event: AuditEventName + actor_kind: AuditEntityKind + actor_id: SerializedId + subject_kind: AuditEntityKind + subject_id: SerializedId + attributes: dict[str, object] diff --git a/ffun/ffun/audit/migrations/20260716_01_c0DeX-audit-records.py b/ffun/ffun/audit/migrations/20260716_01_c0DeX-audit-records.py new file mode 100644 index 00000000..472e67a7 --- /dev/null +++ b/ffun/ffun/audit/migrations/20260716_01_c0DeX-audit-records.py @@ -0,0 +1,45 @@ +""" +audit-records +""" + +from typing import Any + +from psycopg import Connection +from yoyo import step + +__depends__: set[str] = set() + + +sql_create_audit_records = """ +-- Append-only durable records of audited business changes and events. +CREATE TABLE a_records ( + id UUID PRIMARY KEY, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + event TEXT NOT NULL, + actor_kind SMALLINT NOT NULL, + actor_id TEXT NOT NULL, + subject_kind SMALLINT NOT NULL, + subject_id TEXT NOT NULL, + attributes JSONB NOT NULL +) +""" + + +sql_create_audit_records_subject_index = """ +CREATE INDEX a_records_subject_kind_subject_id_created_at_id_idx +ON a_records (subject_kind, subject_id, created_at, id) +""" + + +def apply_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute(sql_create_audit_records) + cursor.execute(sql_create_audit_records_subject_index) + + +def rollback_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute("DROP TABLE a_records") + + +steps = [step(apply_step, rollback_step)] diff --git a/ffun/ffun/audit/operations.py b/ffun/ffun/audit/operations.py new file mode 100644 index 00000000..bb50584c --- /dev/null +++ b/ffun/ffun/audit/operations.py @@ -0,0 +1,93 @@ +import types +import uuid +from collections.abc import Mapping +from typing import cast + +from psycopg.types.json import Jsonb + +from ffun.audit.entities import AuditEntityKind, AuditEventName, AuditRecord, AuditRecordId +from ffun.core.postgresql import ExecuteType +from ffun.domain.entities import SerializedId + +_EMPTY_ATTRIBUTES: Mapping[str, object] = types.MappingProxyType({}) + + +def new_audit_record_id() -> AuditRecordId: + return AuditRecordId(uuid.uuid4()) + + +async def record( # noqa: CFQ002 + execute: ExecuteType, + *, + event: AuditEventName, + actor_kind: AuditEntityKind, + actor_id: SerializedId, + subject_kind: AuditEntityKind, + subject_id: SerializedId, + attributes: Mapping[str, object] = _EMPTY_ATTRIBUTES, +) -> AuditRecordId: + record_id = new_audit_record_id() + + sql = """ + INSERT INTO a_records ( + id, + event, + actor_kind, + actor_id, + subject_kind, + subject_id, + attributes + ) + VALUES ( + %(id)s, + %(event)s, + %(actor_kind)s, + %(actor_id)s, + %(subject_kind)s, + %(subject_id)s, + %(attributes)s + ) + """ + + await execute( + sql, + { + "id": record_id, + "event": event, + "actor_kind": int(actor_kind), + "actor_id": actor_id, + "subject_kind": int(subject_kind), + "subject_id": subject_id, + "attributes": Jsonb(dict(attributes)), + }, + ) + + return record_id + + +async def load_records_for_subject( + execute: ExecuteType, + *, + subject_kind: AuditEntityKind, + subject_id: SerializedId, +) -> list[AuditRecord]: + sql = """ + SELECT * + FROM a_records + WHERE subject_kind = %(subject_kind)s + AND subject_id = %(subject_id)s + ORDER BY created_at, id + """ + + rows = cast( + list[dict[str, object]], + await execute( + sql, + { + "subject_kind": int(subject_kind), + "subject_id": subject_id, + }, + ), + ) + + return [AuditRecord.model_validate(row) for row in rows] diff --git a/ffun/ffun/audit/tests/__init__.py b/ffun/ffun/audit/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ffun/ffun/audit/tests/helpers.py b/ffun/ffun/audit/tests/helpers.py new file mode 100644 index 00000000..92402c10 --- /dev/null +++ b/ffun/ffun/audit/tests/helpers.py @@ -0,0 +1,21 @@ +from typing import cast + +from ffun.audit.entities import AuditRecord, AuditRecordId +from ffun.core.postgresql import execute + + +async def load_audit_record(record_id: AuditRecordId) -> AuditRecord: + sql = """ + SELECT * + FROM a_records + WHERE id = %(id)s + """ + + rows = cast( + list[dict[str, object]], + await execute(sql, {"id": record_id}), # type: ignore[misc] + ) + + assert len(rows) == 1 + + return AuditRecord.model_validate(rows[0]) diff --git a/ffun/ffun/audit/tests/test_domain.py b/ffun/ffun/audit/tests/test_domain.py new file mode 100644 index 00000000..672e53e9 --- /dev/null +++ b/ffun/ffun/audit/tests/test_domain.py @@ -0,0 +1,16 @@ +from ffun.audit import domain, operations + + +class TestNewAuditRecordId: + def test_reexports_operation(self) -> None: + assert domain.new_audit_record_id is operations.new_audit_record_id + + +class TestRecord: + def test_reexports_operation(self) -> None: + assert domain.record is operations.record + + +class TestLoadRecordsForSubject: + def test_reexports_operation(self) -> None: + assert domain.load_records_for_subject is operations.load_records_for_subject diff --git a/ffun/ffun/audit/tests/test_entities.py b/ffun/ffun/audit/tests/test_entities.py new file mode 100644 index 00000000..7c13b9b9 --- /dev/null +++ b/ffun/ffun/audit/tests/test_entities.py @@ -0,0 +1,9 @@ +from ffun.audit.entities import AuditEntityKind + + +class TestAuditEntityKind: + def test_stable_values(self) -> None: + assert AuditEntityKind.user.value == 1 + assert AuditEntityKind.admin.value == 2 + assert AuditEntityKind.psp.value == 3 + assert AuditEntityKind.system.value == 4 diff --git a/ffun/ffun/audit/tests/test_operations.py b/ffun/ffun/audit/tests/test_operations.py new file mode 100644 index 00000000..ea660083 --- /dev/null +++ b/ffun/ffun/audit/tests/test_operations.py @@ -0,0 +1,202 @@ +import asyncio +import datetime +import uuid +from typing import cast + +import pytest +from psycopg import IntegrityError +from pytest_mock import MockerFixture + +from ffun.audit import operations +from ffun.audit.entities import AuditEntityKind, AuditEventName +from ffun.audit.tests.helpers import load_audit_record +from ffun.core.postgresql import execute, transaction +from ffun.core.tests.helpers import TableSizeDelta, TableSizeNotChanged +from ffun.domain.entities import SerializedId + + +class TestNewAuditRecordId: + def test_returns_unique_uuid(self) -> None: + first_id = operations.new_audit_record_id() + second_id = operations.new_audit_record_id() + + assert isinstance(first_id, uuid.UUID) + assert isinstance(second_id, uuid.UUID) + assert first_id != second_id + + +class TestRecord: + @pytest.mark.asyncio + async def test_default_attributes(self) -> None: + async with TableSizeDelta("a_records", delta=1): + record_id = await operations.record( + execute, + event=AuditEventName("system_started"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.system, + subject_id=SerializedId("system"), + ) + + record = await load_audit_record(record_id) + assert record.attributes == {} + + @pytest.mark.asyncio + async def test_inserts_record(self) -> None: + before_insert = datetime.datetime.now(tz=datetime.timezone.utc) + + async with TableSizeDelta("a_records", delta=1): + record_id = await operations.record( + execute, + event=AuditEventName("user_changed"), + actor_kind=AuditEntityKind.admin, + actor_id=SerializedId("admin-1"), + subject_kind=AuditEntityKind.user, + subject_id=SerializedId("user-1"), + attributes={"enabled": True}, + ) + + assert isinstance(record_id, uuid.UUID) + + record = await load_audit_record(record_id) + assert record.id == record_id + assert before_insert <= record.created_at <= datetime.datetime.now(tz=datetime.timezone.utc) + assert record.event == "user_changed" + assert record.actor_kind == AuditEntityKind.admin + assert record.actor_id == "admin-1" + assert record.subject_kind == AuditEntityKind.user + assert record.subject_id == "user-1" + assert record.attributes == {"enabled": True} + + @pytest.mark.asyncio + async def test_duplicate_id_does_not_replace_record(self, mocker: MockerFixture) -> None: + record_id = operations.new_audit_record_id() + mocker.patch.object(operations, "new_audit_record_id", return_value=record_id) + + async with TableSizeDelta("a_records", delta=1): + await operations.record( + execute, + event=AuditEventName("original_event"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system-1"), + subject_kind=AuditEntityKind.user, + subject_id=SerializedId("user-1"), + attributes={}, + ) + + integrity_error = cast(type[Exception], IntegrityError) + + async with TableSizeNotChanged("a_records"): + with pytest.raises(integrity_error): + await operations.record( + execute, + event=AuditEventName("replacement_event"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system-2"), + subject_kind=AuditEntityKind.user, + subject_id=SerializedId("user-2"), + attributes={"replacement": True}, + ) + + record = await load_audit_record(record_id) + assert record.event == "original_event" + assert record.actor_id == "system-1" + assert record.subject_id == "user-1" + assert record.attributes == {} + + @pytest.mark.asyncio + async def test_caller_transaction_rollback_removes_record(self) -> None: + class RollbackTransaction(Exception): + pass + + record_id = None + subject_id = SerializedId(f"user-{uuid.uuid4()}") + + async with TableSizeNotChanged("a_records"): + with pytest.raises(RollbackTransaction): + async with transaction() as transaction_execute: + record_id = await operations.record( + transaction_execute, + event=AuditEventName("user_changed"), + actor_kind=AuditEntityKind.admin, + actor_id=SerializedId("admin-1"), + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + ) + raise RollbackTransaction() + + assert record_id is not None + records = await operations.load_records_for_subject( + execute, + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + ) + assert records == [] + + +class TestLoadRecordsForSubject: + @pytest.mark.asyncio + async def test_missing(self) -> None: + subject_id = SerializedId(f"missing-{uuid.uuid4()}") + + assert ( + await operations.load_records_for_subject( + execute, + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + ) + == [] + ) + + @pytest.mark.asyncio + async def test_filters_by_subject_and_orders_oldest_first(self) -> None: + subject_id = SerializedId(f"user-{uuid.uuid4()}") + + async with TableSizeDelta("a_records", delta=4): + first_id = await operations.record( + execute, + event=AuditEventName("first_event"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + ) + + await operations.record( + execute, + event=AuditEventName("different_subject_kind"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.system, + subject_id=subject_id, + ) + await operations.record( + execute, + event=AuditEventName("different_subject_id"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.user, + subject_id=SerializedId(f"user-{uuid.uuid4()}"), + ) + + await asyncio.sleep(0.001) + + second_id = await operations.record( + execute, + event=AuditEventName("second_event"), + actor_kind=AuditEntityKind.admin, + actor_id=SerializedId("admin"), + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + attributes={"sequence": 2}, + ) + + records = await operations.load_records_for_subject( + execute, + subject_kind=AuditEntityKind.user, + subject_id=subject_id, + ) + + assert [record.id for record in records] == [first_id, second_id] + assert [record.event for record in records] == ["first_event", "second_event"] + assert records[1].attributes == {"sequence": 2} diff --git a/ffun/ffun/cli/application.py b/ffun/ffun/cli/application.py index d1374bbd..ddc0011b 100644 --- a/ffun/ffun/cli/application.py +++ b/ffun/ffun/cli/application.py @@ -2,6 +2,7 @@ from ffun.cli.commands import cleaner # noqa: F401 from ffun.cli.commands import debug # noqa: F401 +from ffun.cli.commands import entitlements # noqa: F401 from ffun.cli.commands import estimates # noqa: F401 from ffun.cli.commands import experiments # noqa: F401 from ffun.cli.commands import feeds # noqa: F401 @@ -31,6 +32,7 @@ app.add_typer(users.cli_app, name="users") app.add_typer(queues.cli_app, name="queues") app.add_typer(debug.cli_app, name="debug") +app.add_typer(entitlements.cli_app, name="entitlements") if __name__ == "__main__": diff --git a/ffun/ffun/cli/commands/cleaner.py b/ffun/ffun/cli/commands/cleaner.py index e111acbe..8c3b04f8 100644 --- a/ffun/ffun/cli/commands/cleaner.py +++ b/ffun/ffun/cli/commands/cleaner.py @@ -5,6 +5,7 @@ from ffun.application.application import with_app from ffun.core import logging from ffun.domain.entities import TagId +from ffun.entitlements import domain as e_domain from ffun.feeds import domain as f_domain from ffun.library import domain as l_domain from ffun.meta import domain as m_domain @@ -39,6 +40,12 @@ async def run_clean(chunk: int) -> None: logger.info("cleaning_orphaned_tags_finished") + logger.info("cleaning_expired_entitlements_started") + + deleted = await e_domain.cleanup_expired_entitlements() + + logger.info("cleaning_expired_entitlements_finished", deleted=deleted) + logger.info("cleaning_finished") diff --git a/ffun/ffun/cli/commands/entitlements.py b/ffun/ffun/cli/commands/entitlements.py new file mode 100644 index 00000000..eb13ab31 --- /dev/null +++ b/ffun/ffun/cli/commands/entitlements.py @@ -0,0 +1,221 @@ +import asyncio +import datetime +import json +import uuid +from collections.abc import Coroutine + +import typer + +from ffun.application.application import with_app +from ffun.audit.entities import AuditEntityKind +from ffun.core import errors as core_errors +from ffun.core.entities import BaseEntity +from ffun.domain.entities import SerializedId, UserId +from ffun.entitlements import domain as e_domain +from ffun.entitlements.entities import EffectiveEntitlementInterval, EntitlementKindId, EntitlementSourceId + +cli_app = typer.Typer() + + +class SourceChangeCommand(BaseEntity): + source: EntitlementSourceId + user_id: UserId + kind_id: EntitlementKindId + granted: bool + value: int | None + starts_at: datetime.datetime + expires_at: datetime.datetime + actor_kind: AuditEntityKind + actor_id: SerializedId + + +class ListEntitlementsCommand(BaseEntity): + user_ids: list[UserId] + kind_ids: list[EntitlementKindId] + + +def run_async_command(command: Coroutine[object, object, None]) -> None: + try: + asyncio.run(command) + except core_errors.Error as error: + typer.echo(str(error), err=True) + raise typer.Exit(code=1) from error + + +def entitlement_kind_from_name(raw_kind: str) -> EntitlementKindId: + try: + return EntitlementKindId[raw_kind] + except KeyError as error: + valid_names = ", ".join(kind.name for kind in EntitlementKindId) + raise typer.BadParameter(f"unknown entitlement kind {raw_kind!r}; expected one of: {valid_names}") from error + + +def actor_kind_from_name(raw_kind: str) -> AuditEntityKind: + try: + return AuditEntityKind[raw_kind] + except KeyError as error: + valid_names = ", ".join(kind.name for kind in AuditEntityKind) + raise typer.BadParameter(f"unknown actor kind {raw_kind!r}; expected one of: {valid_names}") from error + + +def timestamp_from_string(raw_timestamp: str | None, *, option_name: str) -> datetime.datetime | None: + if raw_timestamp is None: + return None + + try: + timestamp = datetime.datetime.fromisoformat(raw_timestamp) + except ValueError as error: + raise typer.BadParameter("expected an ISO 8601 timestamp", param_hint=option_name) from error + + if timestamp.tzinfo is None or timestamp.utcoffset() is None: + raise typer.BadParameter("timestamp must include an explicit UTC offset", param_hint=option_name) + + return timestamp + + +def resolve_timestamps( + starts_at: datetime.datetime | None, + expires_at: datetime.datetime | None, + captured_at: datetime.datetime, +) -> tuple[datetime.datetime, datetime.datetime]: + return ( + starts_at if starts_at is not None else captured_at, + expires_at if expires_at is not None else captured_at + datetime.timedelta(days=31), + ) + + +async def run_source_change(command: SourceChangeCommand) -> None: + async with with_app(): + await e_domain.change_source_entitlement( + source=command.source, + user_id=command.user_id, + kind_id=command.kind_id, + granted=command.granted, + value=command.value, + starts_at=command.starts_at, + expires_at=command.expires_at, + actor_kind=command.actor_kind, + actor_id=command.actor_id, + ) + + +def change_source_entitlement( # noqa: CFQ002 + *, + user_id: uuid.UUID, + kind: str, + source: str, + granted: bool, + value: int | None, + starts_at: str | None, + expires_at: str | None, + actor_kind: str, + actor_id: str, +) -> None: + captured_at = datetime.datetime.now(tz=datetime.UTC) + resolved_starts_at, resolved_expires_at = resolve_timestamps( + timestamp_from_string(starts_at, option_name="--starts-at"), + timestamp_from_string(expires_at, option_name="--expires-at"), + captured_at, + ) + run_async_command( + run_source_change( + SourceChangeCommand( + source=EntitlementSourceId(source), + user_id=UserId(user_id), + kind_id=entitlement_kind_from_name(kind), + granted=granted, + value=value, + starts_at=resolved_starts_at, + expires_at=resolved_expires_at, + actor_kind=actor_kind_from_name(actor_kind), + actor_id=SerializedId(actor_id), + ) + ) + ) + + +@cli_app.command() # type: ignore +def grant( # noqa: CFQ002 + user_id: uuid.UUID = typer.Option(..., "--user-id"), + kind: str = typer.Option(..., "--kind"), + source: str = typer.Option("system", "--source"), + value: int = typer.Option(..., "--value"), + starts_at: str | None = typer.Option(None, "--starts-at"), + expires_at: str | None = typer.Option(None, "--expires-at"), + actor_kind: str = typer.Option("admin", "--actor-kind"), + actor_id: str = typer.Option("admin", "--actor-id"), +) -> None: + change_source_entitlement( + user_id=user_id, + kind=kind, + source=source, + granted=True, + value=value, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=actor_kind, + actor_id=actor_id, + ) + + +@cli_app.command() # type: ignore +def revoke( # noqa: CFQ002 + user_id: uuid.UUID = typer.Option(..., "--user-id"), + kind: str = typer.Option(..., "--kind"), + source: str = typer.Option("system", "--source"), + starts_at: str | None = typer.Option(None, "--starts-at"), + expires_at: str | None = typer.Option(None, "--expires-at"), + actor_kind: str = typer.Option("admin", "--actor-kind"), + actor_id: str = typer.Option("admin", "--actor-id"), +) -> None: + change_source_entitlement( + user_id=user_id, + kind=kind, + source=source, + granted=False, + value=None, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=actor_kind, + actor_id=actor_id, + ) + + +def entitlement_record( + user_id: UserId, + kind_id: EntitlementKindId, + interval: EffectiveEntitlementInterval | None, +) -> dict[str, object]: + return { + "user_id": str(user_id), + "kind": kind_id.name, + "kind_id": kind_id.value, + "granted": interval is not None, + "value": interval.value if interval is not None else None, + "starts_at": interval.starts_at.isoformat() if interval is not None else None, + "expires_at": interval.expires_at.isoformat() if interval is not None else None, + } + + +async def run_list(command: ListEntitlementsCommand) -> None: + async with with_app(): + result = await e_domain.get_entitlements(command.user_ids, command.kind_ids) + + for user_id, entitlements in result.items(): + for kind_id, interval in entitlements.items(): + typer.echo(json.dumps(entitlement_record(user_id, kind_id, interval))) + + +@cli_app.command("list") # type: ignore +def list_entitlements( + user_ids: list[uuid.UUID] = typer.Option(..., "--user-id"), + kinds: list[str] | None = typer.Option(None, "--kind"), +) -> None: + run_async_command( + run_list( + ListEntitlementsCommand( + user_ids=[UserId(user_id) for user_id in user_ids], + kind_ids=[entitlement_kind_from_name(kind) for kind in kinds or []], + ) + ) + ) diff --git a/ffun/ffun/cli/commands/queues.py b/ffun/ffun/cli/commands/queues.py index 7d1f6d87..ba4ffb15 100644 --- a/ffun/ffun/cli/commands/queues.py +++ b/ffun/ffun/cli/commands/queues.py @@ -4,7 +4,7 @@ from ffun.application.application import with_app from ffun.core import logging -from ffun.queues import operations as q_operations +from ffun.queues import domain as q_domain from ffun.queues.entities import QueueKind logger = logging.get_module_logger() @@ -43,7 +43,7 @@ async def cleanup_queues(clean_all: bool, queue: str | None, subqueue: int | Non logger.info("queues_cleanup_all_started") for queue_kind in QueueKind: - await q_operations.tech_clear_queue(queue_kind) + await q_domain.tech_clear_queue(queue_kind) logger.info("queues_cleanup_all_finished") return @@ -54,7 +54,7 @@ async def cleanup_queues(clean_all: bool, queue: str | None, subqueue: int | Non logger.info("queue_cleanup_started", queue=queue_kind.name, primary_id=queue_kind.value, secondary_id=subqueue) - await q_operations.tech_clear_queue(queue_kind, secondary_id=subqueue) + await q_domain.tech_clear_queue(queue_kind, secondary_id=subqueue) logger.info("queue_cleanup_finished", queue=queue_kind.name, primary_id=queue_kind.value, secondary_id=subqueue) diff --git a/ffun/ffun/cli/commands/tests/test_cleaner.py b/ffun/ffun/cli/commands/tests/test_cleaner.py new file mode 100644 index 00000000..521df763 --- /dev/null +++ b/ffun/ffun/cli/commands/tests/test_cleaner.py @@ -0,0 +1,44 @@ +import contextlib + +import pytest +from pytest_mock import MockerFixture + +from ffun.cli.commands import cleaner + + +class TestRunClean: + @pytest.mark.asyncio + async def test_cleans_expired_entitlements_after_orphaned_data(self, mocker: MockerFixture) -> None: + mocker.patch.object(cleaner, "with_app", return_value=contextlib.nullcontext()) + logger_info = mocker.patch.object(cleaner.logger, "info") + + operations: list[str] = [] + + async def clean_entries(*, chunk: int) -> int: + assert chunk == 17 + operations.append("entries") + return 0 + + async def clean_feeds(*, chunk: int) -> int: + assert chunk == 17 + operations.append("feeds") + return 0 + + async def clean_tags(*, chunk: int) -> int: + assert chunk == 17 + operations.append("tags") + return 0 + + async def clean_entitlements() -> int: + operations.append("entitlements") + return 3 + + mocker.patch.object(cleaner.m_domain, "clean_orphaned_entries", side_effect=clean_entries) + mocker.patch.object(cleaner.m_domain, "clean_orphaned_feeds", side_effect=clean_feeds) + mocker.patch.object(cleaner.m_domain, "clean_orphaned_tags", side_effect=clean_tags) + mocker.patch.object(cleaner.e_domain, "cleanup_expired_entitlements", side_effect=clean_entitlements) + + await cleaner.run_clean(chunk=17) + + assert operations == ["entries", "feeds", "tags", "entitlements"] + logger_info.assert_any_call("cleaning_expired_entitlements_finished", deleted=3) diff --git a/ffun/ffun/cli/commands/tests/test_entitlements.py b/ffun/ffun/cli/commands/tests/test_entitlements.py new file mode 100644 index 00000000..5677b001 --- /dev/null +++ b/ffun/ffun/cli/commands/tests/test_entitlements.py @@ -0,0 +1,316 @@ +import asyncio +import contextlib +import datetime +import json + +import pytest +import typer +from pytest_mock import MockerFixture + +from ffun.audit.entities import AuditEntityKind +from ffun.cli.commands import entitlements +from ffun.core import errors as core_errors +from ffun.domain.domain import new_user_id +from ffun.domain.entities import SerializedId, UserId +from ffun.entitlements.entities import EffectiveEntitlementInterval, EntitlementKindId, EntitlementSourceId +from ffun.entitlements.tests.make import make_effective_entitlement_interval + + +class TestEntitlementKindFromName: + @pytest.mark.parametrize("kind", list(EntitlementKindId)) + def test_registered_name(self, kind: EntitlementKindId) -> None: + assert entitlements.entitlement_kind_from_name(kind.name) == kind + + def test_rejects_numeric_value(self) -> None: + with pytest.raises(typer.BadParameter): + entitlements.entitlement_kind_from_name(str(EntitlementKindId.day_tokens.value)) + + def test_rejects_unknown_name(self) -> None: + with pytest.raises(typer.BadParameter): + entitlements.entitlement_kind_from_name("unknown") + + +class TestActorKindFromName: + @pytest.mark.parametrize("kind", list(AuditEntityKind)) + def test_registered_name(self, kind: AuditEntityKind) -> None: + assert entitlements.actor_kind_from_name(kind.name) == kind + + def test_rejects_unknown_name(self) -> None: + with pytest.raises(typer.BadParameter): + entitlements.actor_kind_from_name("unknown") + + +class TestTimestampFromString: + def test_missing_value(self) -> None: + assert entitlements.timestamp_from_string(None, option_name="--starts-at") is None + + def test_iso_8601_with_utc_offset(self) -> None: + timestamp = datetime.datetime.now(tz=datetime.timezone(datetime.timedelta(hours=2))) + + assert entitlements.timestamp_from_string(timestamp.isoformat(), option_name="--starts-at") == timestamp + + def test_rejects_invalid_value(self) -> None: + with pytest.raises(typer.BadParameter): + entitlements.timestamp_from_string("not-a-timestamp", option_name="--starts-at") + + def test_rejects_value_without_utc_offset(self) -> None: + timestamp = datetime.datetime.now().replace(tzinfo=None) + + with pytest.raises(typer.BadParameter): + entitlements.timestamp_from_string(timestamp.isoformat(), option_name="--starts-at") + + +class TestResolveTimestamps: + def test_defaults_from_one_captured_timestamp(self) -> None: + captured_at = datetime.datetime.now(tz=datetime.UTC) + + starts_at, expires_at = entitlements.resolve_timestamps(None, None, captured_at) + + assert starts_at == captured_at + assert expires_at == captured_at + datetime.timedelta(days=31) + + def test_preserves_explicit_values(self) -> None: + starts_at = datetime.datetime.now(tz=datetime.UTC) + expires_at = starts_at + datetime.timedelta(days=7) + + assert entitlements.resolve_timestamps(starts_at, expires_at, starts_at) == (starts_at, expires_at) + + def test_defaults_only_start(self) -> None: + captured_at = datetime.datetime.now(tz=datetime.UTC) + expires_at = captured_at + datetime.timedelta(days=7) + + assert entitlements.resolve_timestamps(None, expires_at, captured_at) == (captured_at, expires_at) + + def test_defaults_only_expiration(self) -> None: + captured_at = datetime.datetime.now(tz=datetime.UTC) + starts_at = captured_at - datetime.timedelta(days=1) + + assert entitlements.resolve_timestamps(starts_at, None, captured_at) == ( + starts_at, + captured_at + datetime.timedelta(days=31), + ) + + +class TestRunSourceChange: + @pytest.mark.asyncio + async def test_passes_command_to_domain(self, mocker: MockerFixture) -> None: + mocker.patch.object(entitlements, "with_app", return_value=contextlib.nullcontext()) + received_commands: list[entitlements.SourceChangeCommand] = [] + + async def change_source_entitlement( # noqa: CFQ002 + *, + source: EntitlementSourceId, + user_id: UserId, + kind_id: EntitlementKindId, + granted: bool, + value: int | None, + starts_at: datetime.datetime, + expires_at: datetime.datetime, + actor_kind: AuditEntityKind, + actor_id: SerializedId, + ) -> tuple[bool, int | None]: + received_commands.append( + entitlements.SourceChangeCommand( + source=source, + user_id=user_id, + kind_id=kind_id, + granted=granted, + value=value, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=actor_kind, + actor_id=actor_id, + ) + ) + return (True, value) + + mocker.patch.object( + entitlements.e_domain, + "change_source_entitlement", + side_effect=change_source_entitlement, + ) + starts_at = datetime.datetime.now(tz=datetime.UTC) + command = entitlements.SourceChangeCommand( + source=EntitlementSourceId("test"), + user_id=new_user_id(), + kind_id=EntitlementKindId.day_tokens, + granted=True, + value=10, + starts_at=starts_at, + expires_at=starts_at + datetime.timedelta(days=1), + actor_kind=AuditEntityKind.admin, + actor_id=SerializedId("test-admin"), + ) + + await entitlements.run_source_change(command) + + assert received_commands == [command] + + +class TestChangeSourceEntitlement: + @pytest.mark.asyncio + async def test_builds_and_runs_command(self, mocker: MockerFixture) -> None: + received_commands: list[entitlements.SourceChangeCommand] = [] + + async def run_source_change(command: entitlements.SourceChangeCommand) -> None: + received_commands.append(command) + + mocker.patch.object(entitlements, "run_source_change", side_effect=run_source_change) + user_id = new_user_id() + started_at = datetime.datetime.now(tz=datetime.UTC) + + await asyncio.to_thread( + entitlements.change_source_entitlement, + user_id=user_id, + kind="day_tokens", + source="test", + granted=True, + value=10, + starts_at=None, + expires_at=None, + actor_kind="admin", + actor_id="test-admin", + ) + + finished_at = datetime.datetime.now(tz=datetime.UTC) + assert len(received_commands) == 1 + command = received_commands[0] + assert started_at <= command.starts_at <= finished_at + assert command.expires_at == command.starts_at + datetime.timedelta(days=31) + assert command == entitlements.SourceChangeCommand( + source=EntitlementSourceId("test"), + user_id=user_id, + kind_id=EntitlementKindId.day_tokens, + granted=True, + value=10, + starts_at=command.starts_at, + expires_at=command.expires_at, + actor_kind=AuditEntityKind.admin, + actor_id=SerializedId("test-admin"), + ) + + +class TestRunAsyncCommand: + @pytest.mark.asyncio + async def test_success(self) -> None: + called = False + + async def command() -> None: + nonlocal called + called = True + + await asyncio.to_thread(entitlements.run_async_command, command()) + + assert called + + @pytest.mark.asyncio + async def test_project_error_exits_nonzero(self, capsys: pytest.CaptureFixture[str]) -> None: + async def command() -> None: + raise core_errors.CoreError(reason="invalid command") + + with pytest.raises(typer.Exit) as raised: + await asyncio.to_thread(entitlements.run_async_command, command()) + + assert raised.value.exit_code == 1 + assert "CoreError" in capsys.readouterr().err + + @pytest.mark.asyncio + async def test_unexpected_error_propagates(self) -> None: + async def command() -> None: + raise RuntimeError("unexpected failure") + + with pytest.raises(RuntimeError, match="unexpected failure"): + await asyncio.to_thread(entitlements.run_async_command, command()) + + +class TestEntitlementRecord: + def test_granted(self) -> None: + user_id = new_user_id() + interval = make_effective_entitlement_interval(user_id=user_id, kind_id=EntitlementKindId.day_tokens) + + assert entitlements.entitlement_record(user_id, EntitlementKindId.day_tokens, interval) == { + "user_id": str(user_id), + "kind": "day_tokens", + "kind_id": 1, + "granted": True, + "value": interval.value, + "starts_at": interval.starts_at.isoformat(), + "expires_at": interval.expires_at.isoformat(), + } + + def test_not_granted(self) -> None: + user_id = new_user_id() + + assert entitlements.entitlement_record(user_id, EntitlementKindId.month_tokens, None) == { + "user_id": str(user_id), + "kind": "month_tokens", + "kind_id": 2, + "granted": False, + "value": None, + "starts_at": None, + "expires_at": None, + } + + +class TestRunList: + @pytest.mark.asyncio + async def test_outputs_domain_result_as_json( + self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str] + ) -> None: + mocker.patch.object(entitlements, "with_app", return_value=contextlib.nullcontext()) + user_id = new_user_id() + interval = make_effective_entitlement_interval( + user_id=user_id, + kind_id=EntitlementKindId.day_tokens, + ) + received_user_ids: list[list[UserId]] = [] + received_kind_ids: list[list[EntitlementKindId]] = [] + + async def get_entitlements( + user_ids: list[UserId], kind_ids: list[EntitlementKindId] + ) -> dict[UserId, dict[EntitlementKindId, EffectiveEntitlementInterval | None]]: # noqa: TAE002 + received_user_ids.append(user_ids) + received_kind_ids.append(kind_ids) + return { + user_id: { + EntitlementKindId.day_tokens: interval, + EntitlementKindId.month_tokens: None, + } + } + + mocker.patch.object( + entitlements.e_domain, + "get_entitlements", + side_effect=get_entitlements, + ) + command = entitlements.ListEntitlementsCommand( + user_ids=[user_id], + kind_ids=[EntitlementKindId.day_tokens, EntitlementKindId.month_tokens], + ) + + await entitlements.run_list(command) + + assert received_user_ids == [command.user_ids] + assert received_kind_ids == [command.kind_ids] + granted_record: dict[str, object] = { + "user_id": str(user_id), + "kind": "day_tokens", + "kind_id": 1, + "granted": True, + "value": interval.value, + "starts_at": interval.starts_at.isoformat(), + "expires_at": interval.expires_at.isoformat(), + } + not_granted_record: dict[str, object] = { + "user_id": str(user_id), + "kind": "month_tokens", + "kind_id": 2, + "granted": False, + "value": None, + "starts_at": None, + "expires_at": None, + } + assert capsys.readouterr().out.splitlines() == [ + json.dumps(granted_record), + json.dumps(not_granted_record), + ] diff --git a/ffun/ffun/conftest.py b/ffun/ffun/conftest.py index dcb9c653..8c006d3a 100644 --- a/ffun/ffun/conftest.py +++ b/ffun/ffun/conftest.py @@ -1,9 +1,12 @@ -from typing import AsyncGenerator +from contextlib import AbstractAsyncContextManager +from typing import AsyncGenerator, cast from unittest import mock import fastapi import pytest_asyncio from httpx import ASGITransport, AsyncClient +from pyleak import no_event_loop_blocking, no_task_leaks +from pyleak.base import LeakAction from ffun.application import application from ffun.auth.tests.fixtures import * # noqa @@ -19,6 +22,18 @@ from ffun.users.tests.fixtures import * # noqa +@pytest_asyncio.fixture(autouse=True) # type: ignore +async def detect_async_leaks() -> AsyncGenerator[None, None]: + async with cast(AbstractAsyncContextManager[object], no_task_leaks(action=LeakAction.RAISE)): + # TODO: Replace Rich exception rendering with a lightweight alternative; large tracebacks + # can synchronously block the event loop long enough to trigger this detector. + async with cast( + AbstractAsyncContextManager[object], + no_event_loop_blocking(action=LeakAction.RAISE, threshold=5.0), + ): + yield + + @pytest_asyncio.fixture(scope="session", autouse=True) # type: ignore async def prepare_db( # app: AsyncGenerator[fastapi.FastAPI, None], diff --git a/ffun/ffun/core/tests/helpers.py b/ffun/ffun/core/tests/helpers.py index 91850eee..ddaa5100 100644 --- a/ffun/ffun/core/tests/helpers.py +++ b/ffun/ffun/core/tests/helpers.py @@ -11,6 +11,7 @@ from structlog import contextvars as structlog_contextvars from structlog.testing import LogCapture +from ffun.core import postgresql from ffun.core.postgresql import execute from ffun.domain.entities import UserId @@ -18,6 +19,11 @@ LOG_RECORDS = list[MutableMapping[str, object]] +def assert_pool_capacity_at_least(required_size: int) -> None: + assert postgresql.POOL is not None + assert postgresql.POOL.max_size >= required_size + + class Comparator: message = "Error" diff --git a/ffun/ffun/domain/entities.py b/ffun/ffun/domain/entities.py index d79e5fc6..2251eb89 100644 --- a/ffun/ffun/domain/entities.py +++ b/ffun/ffun/domain/entities.py @@ -2,6 +2,8 @@ import uuid from typing import NewType +SerializedId = NewType("SerializedId", str) + UserId = NewType("UserId", uuid.UUID) EntryId = NewType("EntryId", uuid.UUID) FeedId = NewType("FeedId", uuid.UUID) diff --git a/ffun/ffun/entitlements/__init__.py b/ffun/ffun/entitlements/__init__.py new file mode 100644 index 00000000..428c469a --- /dev/null +++ b/ffun/ffun/entitlements/__init__.py @@ -0,0 +1 @@ +"""Source-owned and effective user entitlements.""" diff --git a/ffun/ffun/entitlements/domain.py b/ffun/ffun/entitlements/domain.py new file mode 100644 index 00000000..efeb4e38 --- /dev/null +++ b/ffun/ffun/entitlements/domain.py @@ -0,0 +1,357 @@ +import datetime +import itertools +from collections.abc import Mapping, Sequence +from typing import cast + +from ffun.audit import domain as audit_domain +from ffun.audit.entities import AuditEntityKind, AuditEventName +from ffun.core import logging +from ffun.core.postgresql import ExecuteType, execute +from ffun.domain.entities import SerializedId, UserId +from ffun.entitlements import entities as entitlement_entities +from ffun.entitlements import errors, operations +from ffun.entitlements.entities import ( + EffectiveEntitlementInterval, + EffectiveEntitlementState, + EntitlementKind, + EntitlementKindId, + EntitlementSourceId, + MergePolicy, + SourceEntitlement, +) +from ffun.locks.domain import locked_transaction +from ffun.locks.entities import LockKind + +logger = logging.get_module_logger() + + +class _SourceChangeOutcome: + __slots__ = ("changed", "effective_state", "effective_intervals") + + def __init__( + self, + *, + changed: bool, + effective_state: EffectiveEntitlementState, + effective_intervals: list[EffectiveEntitlementInterval], + ) -> None: + self.changed = changed + self.effective_state = effective_state + self.effective_intervals = effective_intervals + + +def get_entitlement_kind(kind_id: EntitlementKindId) -> EntitlementKind: + for kind in entitlement_entities.ENTITLEMENT_KINDS: + if kind.id == kind_id: + return kind + + raise errors.UnknownEntitlementKind(kind_id=kind_id) + + +def validate_source_change( # noqa: CFQ002, CCR001 + *, + source: EntitlementSourceId, + kind_id: EntitlementKindId, + granted: bool, + value: int | None, + starts_at: datetime.datetime, + expires_at: datetime.datetime, + actor_id: SerializedId, +) -> EntitlementKind: + kind = get_entitlement_kind(kind_id) + + if not source: + raise errors.InvalidSourceEntitlement(reason="Entitlement source must not be empty") + + if granted and value is None: + raise errors.InvalidSourceEntitlement(reason="A granted entitlement must have an integer value") + + if not granted and value is not None: + raise errors.InvalidSourceEntitlement(reason="A revoked entitlement must not have a value") + + if starts_at.tzinfo is None or starts_at.utcoffset() is None: + raise errors.InvalidSourceEntitlement(reason="Entitlement activation timestamp must have a UTC offset") + + if expires_at.tzinfo is None or expires_at.utcoffset() is None: + raise errors.InvalidSourceEntitlement(reason="Entitlement expiration timestamp must have a UTC offset") + + if starts_at >= expires_at: + raise errors.InvalidSourceEntitlement( + reason="Entitlement activation timestamp must be earlier than expiration" + ) + + if not actor_id or not actor_id.strip(): + raise errors.InvalidActorId(reason="Audit actor id must not be empty") + + return kind + + +def merge_values(policy: MergePolicy, values: Sequence[int]) -> int: + if not values: + raise errors.InvalidMergeValues(reason="At least one entitlement value is required for merging") + + if policy == MergePolicy.max: + return max(values) + + if policy == MergePolicy.min: + return min(values) + + if policy == MergePolicy.sum: + return sum(values) + + raise AssertionError(f"Unsupported entitlement merge policy: {policy}") + + +def build_effective_timeline( # noqa: CCR001 + *, + user_id: UserId, + kind_id: EntitlementKindId, + merge_policy: MergePolicy, + source_entitlements: Sequence[SourceEntitlement], + evaluation_time: datetime.datetime, +) -> list[EffectiveEntitlementInterval]: + granted_entitlements = [entitlement for entitlement in source_entitlements if entitlement.granted] + boundaries = sorted( + { + boundary + for entitlement in granted_entitlements + for boundary in (entitlement.starts_at, entitlement.expires_at) + } + ) + intervals: list[EffectiveEntitlementInterval] = [] + + for starts_at, expires_at in itertools.pairwise(boundaries): + if expires_at <= evaluation_time: + continue + + values = [ + entitlement.value + for entitlement in granted_entitlements + if entitlement.starts_at <= starts_at and expires_at <= entitlement.expires_at + ] + merged_value = merge_values(merge_policy, [value for value in values if value is not None]) if values else None + + if merged_value is None: + continue + + if intervals and intervals[-1].value == merged_value and intervals[-1].expires_at == starts_at: + intervals[-1] = intervals[-1].replace(expires_at=expires_at) + continue + + intervals.append( + EffectiveEntitlementInterval( + user_id=user_id, + kind_id=kind_id, + value=merged_value, + starts_at=starts_at, + expires_at=expires_at, + ) + ) + + return intervals + + +def effective_state_at( + intervals: Sequence[EffectiveEntitlementInterval], evaluation_time: datetime.datetime +) -> EffectiveEntitlementState: + for interval in intervals: + if interval.starts_at <= evaluation_time < interval.expires_at: + return (True, interval.value) + + return (False, None) + + +async def _apply_source_change( + execute: ExecuteType, + *, + kind: EntitlementKind, + new_source_state: SourceEntitlement, + evaluation_time: datetime.datetime, + actor_kind: AuditEntityKind, + actor_id: SerializedId, +) -> _SourceChangeOutcome: + previous_source_state = await operations.load_source_entitlement( + execute, + new_source_state.user_id, + new_source_state.kind_id, + new_source_state.source, + ) + previous_effective_intervals = await operations.load_effective_intervals( + execute, + new_source_state.user_id, + new_source_state.kind_id, + ending_after=evaluation_time, + ) + + if previous_source_state == new_source_state: + return _SourceChangeOutcome( + changed=False, + effective_state=effective_state_at(previous_effective_intervals, evaluation_time), + effective_intervals=previous_effective_intervals, + ) + + await operations.upsert_source_entitlement(execute, new_source_state) + source_entitlements = await operations.load_source_entitlements( + execute, + new_source_state.user_id, + new_source_state.kind_id, + ) + new_effective_intervals = build_effective_timeline( + user_id=new_source_state.user_id, + kind_id=new_source_state.kind_id, + merge_policy=kind.merge_policy, + source_entitlements=source_entitlements, + evaluation_time=evaluation_time, + ) + await operations.replace_effective_intervals( + execute, + new_source_state.user_id, + new_source_state.kind_id, + new_effective_intervals, + ) + effective_state = effective_state_at(new_effective_intervals, evaluation_time) + await audit_domain.record( + execute, + event=AuditEventName("source_entitlement_changed"), + actor_kind=actor_kind, + actor_id=actor_id, + subject_kind=AuditEntityKind.user, + subject_id=SerializedId(str(new_source_state.user_id)), + attributes={ + "source": new_source_state.source, + "kind_id": new_source_state.kind_id, + "previous_source_state": ( + cast(dict[str, object], previous_source_state.model_dump(mode="json")) + if previous_source_state is not None + else None + ), + "new_source_state": cast(dict[str, object], new_source_state.model_dump(mode="json")), + "previous_effective_intervals": [ + cast(dict[str, object], interval.model_dump(mode="json")) for interval in previous_effective_intervals + ], + "new_effective_intervals": [ + cast(dict[str, object], interval.model_dump(mode="json")) for interval in new_effective_intervals + ], + }, + ) + + return _SourceChangeOutcome( + changed=True, + effective_state=effective_state, + effective_intervals=new_effective_intervals, + ) + + +def _emit_business_events(source_state: SourceEntitlement, outcome: _SourceChangeOutcome) -> None: + logger.business_event( + "source_entitlement_changed", + user_id=source_state.user_id, + source=source_state.source, + kind_id=source_state.kind_id, + granted=source_state.granted, + value=source_state.value, + starts_at=source_state.starts_at.isoformat(), + expires_at=source_state.expires_at.isoformat(), + ) + logger.business_event( + "entitlement_changed", + user_id=source_state.user_id, + kind_id=source_state.kind_id, + granted=outcome.effective_state[0], + value=outcome.effective_state[1], + new_effective_intervals=[ + { + "value": interval.value, + "starts_at": interval.starts_at.isoformat(), + "expires_at": interval.expires_at.isoformat(), + } + for interval in outcome.effective_intervals + ], + ) + + +async def change_source_entitlement( # noqa: CFQ002 + *, + source: EntitlementSourceId, + user_id: UserId, + kind_id: EntitlementKindId, + granted: bool, + value: int | None, + starts_at: datetime.datetime, + expires_at: datetime.datetime, + actor_kind: AuditEntityKind, + actor_id: SerializedId, +) -> EffectiveEntitlementState: + kind = validate_source_change( + source=source, + kind_id=kind_id, + granted=granted, + value=value, + starts_at=starts_at, + expires_at=expires_at, + actor_id=actor_id, + ) + evaluation_time = datetime.datetime.now(tz=datetime.UTC) + new_source_state = SourceEntitlement( + source=source, + user_id=user_id, + kind_id=kind_id, + granted=granted, + value=value, + starts_at=starts_at, + expires_at=expires_at, + ) + + async with locked_transaction(LockKind("entitlements_user_kind"), user_id, kind_id) as transaction_execute: + outcome = await _apply_source_change( + transaction_execute, + kind=kind, + new_source_state=new_source_state, + evaluation_time=evaluation_time, + actor_kind=actor_kind, + actor_id=actor_id, + ) + + if outcome.changed: + _emit_business_events(new_source_state, outcome) + + return outcome.effective_state + + +async def get_entitlements( + user_ids: list[UserId], kind_ids: list[EntitlementKindId] +) -> Mapping[UserId, Mapping[EntitlementKindId, EffectiveEntitlementInterval | None]]: + selected_user_ids = list({user_id: None for user_id in user_ids}) + selected_kind_ids = ( + list({kind_id: None for kind_id in kind_ids}) + if kind_ids + else [kind.id for kind in entitlement_entities.ENTITLEMENT_KINDS] + ) + + for kind_id in selected_kind_ids: + get_entitlement_kind(kind_id) + + result: dict[UserId, dict[EntitlementKindId, EffectiveEntitlementInterval | None]] = { + user_id: {kind_id: None for kind_id in selected_kind_ids} for user_id in selected_user_ids + } + + if not selected_user_ids or not selected_kind_ids: + return result + + evaluation_time = datetime.datetime.now(tz=datetime.UTC) + active_intervals = await operations.load_active_intervals( + execute, + selected_user_ids, + selected_kind_ids, + evaluation_time=evaluation_time, + ) + + for interval in active_intervals: + result[interval.user_id][interval.kind_id] = interval + + return result + + +async def cleanup_expired_entitlements() -> int: + cleanup_time = datetime.datetime.now(tz=datetime.UTC) + return await operations.delete_expired_effective_intervals(execute, cleanup_time) diff --git a/ffun/ffun/entitlements/entities.py b/ffun/ffun/entitlements/entities.py new file mode 100644 index 00000000..24703065 --- /dev/null +++ b/ffun/ffun/entitlements/entities.py @@ -0,0 +1,116 @@ +import datetime +import enum +from typing import NewType, TypeAlias + +import pydantic + +from ffun.core.entities import BaseEntity +from ffun.domain.entities import UserId + +EntitlementSourceId = NewType("EntitlementSourceId", str) +EffectiveEntitlementState: TypeAlias = tuple[bool, int | None] + + +class EntitlementKindId(enum.IntEnum): + day_tokens = 1 + month_tokens = 2 + + +class MergePolicy(enum.StrEnum): + max = "max" + min = "min" + sum = "sum" + + +class EntitlementKind(BaseEntity): + id: EntitlementKindId + merge_policy: MergePolicy + + +ENTITLEMENT_KINDS: tuple[EntitlementKind, ...] = ( + EntitlementKind(id=EntitlementKindId.day_tokens, merge_policy=MergePolicy.max), + EntitlementKind(id=EntitlementKindId.month_tokens, merge_policy=MergePolicy.max), +) + + +class SourceEntitlement(BaseEntity): + source: EntitlementSourceId + user_id: UserId + kind_id: EntitlementKindId + granted: bool = pydantic.Field(strict=True) + value: int | None = pydantic.Field(strict=True) + starts_at: datetime.datetime + expires_at: datetime.datetime + + @pydantic.model_validator(mode="after") + def validate_state(self) -> "SourceEntitlement": # noqa: CCR001 + if self.granted and self.value is None: + raise ValueError("A granted entitlement must have an integer value") + + if not self.granted and self.value is not None: + raise ValueError("A revoked entitlement must not have a value") + + if self.starts_at.tzinfo is None or self.starts_at.utcoffset() is None: + raise ValueError("Entitlement activation timestamp must have a UTC offset") + + if self.expires_at.tzinfo is None or self.expires_at.utcoffset() is None: + raise ValueError("Entitlement expiration timestamp must have a UTC offset") + + if self.starts_at >= self.expires_at: + raise ValueError("Entitlement activation timestamp must be earlier than expiration") + + return self + + def to_revoked( + self, + *, + starts_at: datetime.datetime, + expires_at: datetime.datetime, + ) -> "SourceEntitlement": + return SourceEntitlement( + source=self.source, + user_id=self.user_id, + kind_id=self.kind_id, + granted=False, + value=None, + starts_at=starts_at, + expires_at=expires_at, + ) + + def to_granted( + self, + *, + value: int, + starts_at: datetime.datetime, + expires_at: datetime.datetime, + ) -> "SourceEntitlement": + return SourceEntitlement( + source=self.source, + user_id=self.user_id, + kind_id=self.kind_id, + granted=True, + value=value, + starts_at=starts_at, + expires_at=expires_at, + ) + + +class EffectiveEntitlementInterval(BaseEntity): + user_id: UserId + kind_id: EntitlementKindId + value: int + starts_at: datetime.datetime + expires_at: datetime.datetime + + @pydantic.model_validator(mode="after") + def validate_interval(self) -> "EffectiveEntitlementInterval": + if self.starts_at.tzinfo is None or self.starts_at.utcoffset() is None: + raise ValueError("Effective entitlement activation timestamp must have a UTC offset") + + if self.expires_at.tzinfo is None or self.expires_at.utcoffset() is None: + raise ValueError("Effective entitlement expiration timestamp must have a UTC offset") + + if self.starts_at >= self.expires_at: + raise ValueError("Effective entitlement activation timestamp must be earlier than expiration") + + return self diff --git a/ffun/ffun/entitlements/errors.py b/ffun/ffun/entitlements/errors.py new file mode 100644 index 00000000..75ee40c2 --- /dev/null +++ b/ffun/ffun/entitlements/errors.py @@ -0,0 +1,25 @@ +from ffun.core import errors + + +class Error(errors.Error): + pass + + +class UnknownEntitlementKind(Error): + pass + + +class InvalidSourceEntitlement(Error): + pass + + +class InvalidActorId(Error): + pass + + +class InvalidMergeValues(Error): + pass + + +class InvalidStoredEntitlement(Error): + pass diff --git a/ffun/ffun/entitlements/migrations/20260717_01_c0DeX-entitlements.py b/ffun/ffun/entitlements/migrations/20260717_01_c0DeX-entitlements.py new file mode 100644 index 00000000..7f7b0d86 --- /dev/null +++ b/ffun/ffun/entitlements/migrations/20260717_01_c0DeX-entitlements.py @@ -0,0 +1,62 @@ +""" +entitlements +""" + +from typing import Any + +from psycopg import Connection +from yoyo import step + +__depends__: set[str] = set() + + +sql_create_source_entitlements = """ +-- Stores the latest entitlement state supplied by every source. +CREATE TABLE en_source_entitlements ( + source_id TEXT NOT NULL, + user_id UUID NOT NULL, + kind_id SMALLINT NOT NULL, + granted BOOLEAN NOT NULL, + value BIGINT, + starts_at TIMESTAMP WITH TIME ZONE NOT NULL, + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (user_id, kind_id, source_id) +) +""" + +sql_create_entitlements = """ +-- Materialized effective entitlement intervals derived from the source entitlement table. +CREATE TABLE en_entitlements ( + user_id UUID NOT NULL, + kind_id SMALLINT NOT NULL, + value BIGINT NOT NULL, + starts_at TIMESTAMP WITH TIME ZONE NOT NULL, + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (user_id, kind_id, starts_at) +) +""" + +sql_create_entitlements_expires_at_idx = """ +-- Supports removal of expired effective intervals. +CREATE INDEX en_entitlements_expires_at_idx ON en_entitlements (expires_at) +""" + + +def apply_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute(sql_create_source_entitlements) + cursor.execute(sql_create_entitlements) + cursor.execute(sql_create_entitlements_expires_at_idx) + + +def rollback_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute("DROP TABLE en_entitlements") + cursor.execute("DROP TABLE en_source_entitlements") + + +steps = [step(apply_step, rollback_step)] diff --git a/ffun/ffun/entitlements/operations.py b/ffun/ffun/entitlements/operations.py new file mode 100644 index 00000000..9dbfb13d --- /dev/null +++ b/ffun/ffun/entitlements/operations.py @@ -0,0 +1,215 @@ +import datetime +from collections.abc import Mapping +from typing import Any + +from pydantic import ValidationError +from pypika import Parameter, PostgreSQLQuery + +from ffun.core.postgresql import ExecuteType +from ffun.domain.entities import UserId +from ffun.entitlements import errors +from ffun.entitlements.entities import ( + EffectiveEntitlementInterval, + EntitlementKindId, + EntitlementSourceId, + SourceEntitlement, +) + + +def row_to_source_entitlement(row: Mapping[str, object]) -> SourceEntitlement: + try: + return SourceEntitlement.model_validate(row) + except ValidationError as exception: + raise errors.InvalidStoredEntitlement(entity_kind="source_entitlement") from exception + + +def row_to_effective_interval(row: Mapping[str, object]) -> EffectiveEntitlementInterval: + try: + return EffectiveEntitlementInterval.model_validate(row) + except ValidationError as exception: + raise errors.InvalidStoredEntitlement(entity_kind="effective_entitlement_interval") from exception + + +async def load_source_entitlement( + execute: ExecuteType, + user_id: UserId, + kind_id: EntitlementKindId, + source: EntitlementSourceId, +) -> SourceEntitlement | None: + sql = """ + SELECT source_id AS source, user_id, kind_id, granted, value, starts_at, expires_at + FROM en_source_entitlements + WHERE user_id = %(user_id)s + AND kind_id = %(kind_id)s + AND source_id = %(source)s + """ + + rows = await execute(sql, {"user_id": user_id, "kind_id": kind_id, "source": source}) + + if not rows: + return None + + return row_to_source_entitlement(rows[0]) + + +async def upsert_source_entitlement(execute: ExecuteType, entitlement: SourceEntitlement) -> None: + sql = """ + INSERT INTO en_source_entitlements ( + source_id, + user_id, + kind_id, + granted, + value, + starts_at, + expires_at + ) + VALUES ( + %(source)s, + %(user_id)s, + %(kind_id)s, + %(granted)s, + %(value)s, + %(starts_at)s, + %(expires_at)s + ) + ON CONFLICT (user_id, kind_id, source_id) DO UPDATE + SET granted = EXCLUDED.granted, + value = EXCLUDED.value, + starts_at = EXCLUDED.starts_at, + expires_at = EXCLUDED.expires_at, + updated_at = CURRENT_TIMESTAMP + """ + + await execute( + sql, + { + "source": entitlement.source, + "user_id": entitlement.user_id, + "kind_id": entitlement.kind_id, + "granted": entitlement.granted, + "value": entitlement.value, + "starts_at": entitlement.starts_at, + "expires_at": entitlement.expires_at, + }, + ) + + +async def load_source_entitlements( + execute: ExecuteType, user_id: UserId, kind_id: EntitlementKindId +) -> list[SourceEntitlement]: + sql = """ + SELECT source_id AS source, user_id, kind_id, granted, value, starts_at, expires_at + FROM en_source_entitlements + WHERE user_id = %(user_id)s + AND kind_id = %(kind_id)s + ORDER BY starts_at, expires_at, source_id + """ + + rows = await execute(sql, {"user_id": user_id, "kind_id": kind_id}) + return [row_to_source_entitlement(row) for row in rows] + + +async def load_effective_intervals( + execute: ExecuteType, + user_id: UserId, + kind_id: EntitlementKindId, + *, + ending_after: datetime.datetime, +) -> list[EffectiveEntitlementInterval]: + sql = """ + SELECT user_id, kind_id, value, starts_at, expires_at + FROM en_entitlements + WHERE user_id = %(user_id)s + AND kind_id = %(kind_id)s + AND expires_at > %(ending_after)s + ORDER BY starts_at + """ + + rows = await execute( + sql, + {"user_id": user_id, "kind_id": kind_id, "ending_after": ending_after}, + ) + return [row_to_effective_interval(row) for row in rows] + + +async def replace_effective_intervals( + execute: ExecuteType, + user_id: UserId, + kind_id: EntitlementKindId, + intervals: list[EffectiveEntitlementInterval], +) -> None: + sql_delete = """ + DELETE FROM en_entitlements + WHERE user_id = %(user_id)s + AND kind_id = %(kind_id)s + """ + + await execute(sql_delete, {"user_id": user_id, "kind_id": kind_id}) + + if not intervals: + return + + query = PostgreSQLQuery.into("en_entitlements").columns("user_id", "kind_id", "value", "starts_at", "expires_at") + arguments: dict[str, Any] = {} + + for index, interval in enumerate(intervals): + arguments.update( + { + f"user_id_{index}": interval.user_id, + f"kind_id_{index}": interval.kind_id, + f"value_{index}": interval.value, + f"starts_at_{index}": interval.starts_at, + f"expires_at_{index}": interval.expires_at, + } + ) + query = query.insert( + Parameter(f"%(user_id_{index})s"), + Parameter(f"%(kind_id_{index})s"), + Parameter(f"%(value_{index})s"), + Parameter(f"%(starts_at_{index})s"), + Parameter(f"%(expires_at_{index})s"), + ) + + await execute(str(query), arguments) + + +async def load_active_intervals( + execute: ExecuteType, + user_ids: list[UserId], + kind_ids: list[EntitlementKindId], + *, + evaluation_time: datetime.datetime, +) -> list[EffectiveEntitlementInterval]: + if not user_ids or not kind_ids: + return [] + + sql = """ + SELECT user_id, kind_id, value, starts_at, expires_at + FROM en_entitlements + WHERE user_id = ANY(%(user_ids)s) + AND kind_id = ANY(%(kind_ids)s) + AND starts_at <= %(evaluation_time)s + AND %(evaluation_time)s < expires_at + ORDER BY user_id, kind_id + """ + + rows = await execute( + sql, + { + "user_ids": user_ids, + "kind_ids": kind_ids, + "evaluation_time": evaluation_time, + }, + ) + return [row_to_effective_interval(row) for row in rows] + + +async def delete_expired_effective_intervals(execute: ExecuteType, cleanup_time: datetime.datetime) -> int: + sql = """ + DELETE FROM en_entitlements + WHERE expires_at <= %(cleanup_time)s + RETURNING user_id + """ + + rows = await execute(sql, {"cleanup_time": cleanup_time}) + return len(rows) diff --git a/ffun/ffun/entitlements/tests/__init__.py b/ffun/ffun/entitlements/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ffun/ffun/entitlements/tests/helpers.py b/ffun/ffun/entitlements/tests/helpers.py new file mode 100644 index 00000000..e36217c1 --- /dev/null +++ b/ffun/ffun/entitlements/tests/helpers.py @@ -0,0 +1,59 @@ +import datetime +from typing import cast + +from ffun.core.postgresql import execute +from ffun.domain.entities import UserId +from ffun.entitlements import operations +from ffun.entitlements.entities import EntitlementKindId, SourceEntitlement + + +async def load_source_entitlement_timestamps( + entitlement: SourceEntitlement, +) -> tuple[datetime.datetime, datetime.datetime]: + arguments: dict[str, object] = { + "user_id": entitlement.user_id, + "kind_id": entitlement.kind_id, + "source_id": entitlement.source, + } + rows = cast( + list[dict[str, datetime.datetime]], + await execute( + """ + SELECT created_at, updated_at + FROM en_source_entitlements + WHERE user_id = %(user_id)s + AND kind_id = %(kind_id)s + AND source_id = %(source_id)s + """, + arguments, + ), + ) + assert len(rows) == 1 + return rows[0]["created_at"], rows[0]["updated_at"] + + +async def load_effective_interval_timestamps( + user_id: UserId, + kind_id: EntitlementKindId, +) -> list[tuple[datetime.datetime, datetime.datetime]]: + arguments: dict[str, object] = {"user_id": user_id, "kind_id": kind_id} + rows = cast( + list[dict[str, datetime.datetime]], + await execute( + """ + SELECT created_at, updated_at + FROM en_entitlements + WHERE user_id = %(user_id)s AND kind_id = %(kind_id)s + ORDER BY starts_at + """, + arguments, + ), + ) + return [(row["created_at"], row["updated_at"]) for row in rows] + + +async def clear_effective_intervals() -> None: + await operations.delete_expired_effective_intervals( + execute, + datetime.datetime.max.replace(tzinfo=datetime.UTC), + ) diff --git a/ffun/ffun/entitlements/tests/make.py b/ffun/ffun/entitlements/tests/make.py new file mode 100644 index 00000000..40b0cbf4 --- /dev/null +++ b/ffun/ffun/entitlements/tests/make.py @@ -0,0 +1,50 @@ +import datetime + +from ffun.domain.domain import new_user_id +from ffun.domain.entities import UserId +from ffun.entitlements.entities import ( + EffectiveEntitlementInterval, + EntitlementKindId, + EntitlementSourceId, + SourceEntitlement, +) + + +def make_effective_entitlement_interval( + *, + user_id: UserId | None = None, + kind_id: EntitlementKindId = EntitlementKindId.day_tokens, + value: int = 10, + starts_at: datetime.datetime | None = None, + expires_at: datetime.datetime | None = None, +) -> EffectiveEntitlementInterval: + now = datetime.datetime.now(tz=datetime.UTC) + return EffectiveEntitlementInterval( + user_id=user_id or new_user_id(), + kind_id=kind_id, + value=value, + starts_at=starts_at or now - datetime.timedelta(days=1), + expires_at=expires_at or now + datetime.timedelta(days=1), + ) + + +def make_source_entitlement( # noqa: CFQ002 + *, + user_id: UserId | None = None, + source: EntitlementSourceId = EntitlementSourceId("test"), + kind_id: EntitlementKindId = EntitlementKindId.day_tokens, + granted: bool = True, + value: int | None = 10, + starts_at: datetime.datetime | None = None, + expires_at: datetime.datetime | None = None, +) -> SourceEntitlement: + now = datetime.datetime.now(tz=datetime.UTC) + return SourceEntitlement( + source=source, + user_id=user_id or new_user_id(), + kind_id=kind_id, + granted=granted, + value=value, + starts_at=starts_at or now - datetime.timedelta(days=1), + expires_at=expires_at or now + datetime.timedelta(days=1), + ) diff --git a/ffun/ffun/entitlements/tests/test_domain.py b/ffun/ffun/entitlements/tests/test_domain.py new file mode 100644 index 00000000..40b2eb08 --- /dev/null +++ b/ffun/ffun/entitlements/tests/test_domain.py @@ -0,0 +1,863 @@ +import datetime +from typing import cast + +import pytest +from pytest_mock import MockerFixture + +from ffun.audit import domain as audit_domain +from ffun.audit.entities import AuditEntityKind +from ffun.core.postgresql import execute, transaction +from ffun.core.tests.helpers import ( + TableSizeDelta, + TableSizeNotChanged, + assert_logs_has_business_event, + assert_logs_has_no_business_event, + capture_logs, +) +from ffun.domain.domain import new_user_id +from ffun.domain.entities import SerializedId +from ffun.entitlements import domain +from ffun.entitlements import entities as entitlement_entities +from ffun.entitlements import errors, operations +from ffun.entitlements.entities import EntitlementKindId, EntitlementSourceId, MergePolicy +from ffun.entitlements.tests.helpers import clear_effective_intervals +from ffun.entitlements.tests.make import make_effective_entitlement_interval, make_source_entitlement + +_DAY_TOKENS = EntitlementKindId.day_tokens +_MONTH_TOKENS = EntitlementKindId.month_tokens +_SOURCE = EntitlementSourceId("test") +_ACTOR_KIND = AuditEntityKind.admin +_ACTOR_ID = SerializedId("test-admin") + + +class TestGetEntitlementKind: + def test_known_kind(self) -> None: + kind = domain.get_entitlement_kind(_DAY_TOKENS) + + assert kind.id == _DAY_TOKENS + assert kind.merge_policy == MergePolicy.max + + def test_unconfigured_kind(self, mocker: MockerFixture) -> None: + mocker.patch.object( + entitlement_entities, + "ENTITLEMENT_KINDS", + (entitlement_entities.ENTITLEMENT_KINDS[0],), + ) + + with pytest.raises(errors.UnknownEntitlementKind): + domain.get_entitlement_kind(_MONTH_TOKENS) + + +class TestValidateSourceChange: + def test_valid(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + kind = domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + assert kind.id == _DAY_TOKENS + + def test_unconfigured_kind(self, mocker: MockerFixture) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + mocker.patch.object( + entitlement_entities, + "ENTITLEMENT_KINDS", + (entitlement_entities.ENTITLEMENT_KINDS[0],), + ) + + with pytest.raises(errors.UnknownEntitlementKind): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_MONTH_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + def test_empty_source(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId(""), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + def test_granted_requires_value(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=None, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + def test_revoked_requires_no_value(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=False, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + def test_timestamps_require_utc_offsets(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now.replace(tzinfo=None), + expires_at=now + datetime.timedelta(days=1), + actor_id=_ACTOR_ID, + ) + + def test_expiration_requires_utc_offset(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=(now + datetime.timedelta(days=1)).replace(tzinfo=None), + actor_id=_ACTOR_ID, + ) + + def test_activation_must_be_before_expiration(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidSourceEntitlement): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now, + actor_id=_ACTOR_ID, + ) + + def test_actor_id_must_not_be_empty(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidActorId): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=SerializedId(" "), + ) + + def test_empty_actor_id(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(errors.InvalidActorId): + domain.validate_source_change( + source=EntitlementSourceId("test"), + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_id=SerializedId(""), + ) + + +class TestMergeValues: + @pytest.mark.parametrize( + ("policy", "expected"), + [(MergePolicy.max, 7), (MergePolicy.min, 2), (MergePolicy.sum, 14)], + ) + def test_policies(self, policy: MergePolicy, expected: int) -> None: + assert domain.merge_values(policy, [5, 2, 7]) == expected + + @pytest.mark.parametrize( + ("policy", "expected"), + [(MergePolicy.max, 5), (MergePolicy.min, 5), (MergePolicy.sum, 10)], + ) + def test_duplicate_values(self, policy: MergePolicy, expected: int) -> None: + assert domain.merge_values(policy, [5, 5]) == expected + + def test_empty_values(self) -> None: + with pytest.raises(errors.InvalidMergeValues, match="At least one"): + domain.merge_values(MergePolicy.max, []) + + def test_unsupported_policy(self) -> None: + with pytest.raises(AssertionError, match="Unsupported"): + domain.merge_values(cast(MergePolicy, "unsupported"), [1]) + + +class TestBuildEffectiveTimeline: + def test_empty_source_entitlements(self) -> None: + assert ( + domain.build_effective_timeline( + user_id=new_user_id(), + kind_id=_DAY_TOKENS, + merge_policy=MergePolicy.max, + source_entitlements=[], + evaluation_time=datetime.datetime.now(tz=datetime.UTC), + ) + == [] + ) + + def test_merges_boundaries_and_coalesces_equal_values(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + first = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("first"), + value=10, + starts_at=now - datetime.timedelta(days=2), + expires_at=now + datetime.timedelta(days=2), + ) + second = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("second"), + value=20, + starts_at=now + datetime.timedelta(days=1), + expires_at=now + datetime.timedelta(days=3), + ) + + intervals = domain.build_effective_timeline( + user_id=user_id, + kind_id=_DAY_TOKENS, + merge_policy=MergePolicy.max, + source_entitlements=[first, second], + evaluation_time=now, + ) + + assert [(interval.value, interval.starts_at, interval.expires_at) for interval in intervals] == [ + (10, first.starts_at, second.starts_at), + (20, second.starts_at, second.expires_at), + ] + + def test_skips_expired_state(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + expired = make_source_entitlement( + user_id=user_id, + starts_at=now - datetime.timedelta(days=2), + expires_at=now - datetime.timedelta(days=1), + ) + + assert ( + domain.build_effective_timeline( + user_id=user_id, + kind_id=_DAY_TOKENS, + merge_policy=MergePolicy.max, + source_entitlements=[expired], + evaluation_time=now, + ) + == [] + ) + + def test_skips_revoked_state(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + revoked = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("revoked"), + granted=False, + value=None, + ) + + assert ( + domain.build_effective_timeline( + user_id=user_id, + kind_id=_DAY_TOKENS, + merge_policy=MergePolicy.max, + source_entitlements=[revoked], + evaluation_time=now, + ) + == [] + ) + + def test_preserves_gap_between_disjoint_grants(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + first = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("first"), + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + second = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("second"), + starts_at=now + datetime.timedelta(days=2), + expires_at=now + datetime.timedelta(days=3), + ) + + intervals = domain.build_effective_timeline( + user_id=user_id, + kind_id=_DAY_TOKENS, + merge_policy=MergePolicy.max, + source_entitlements=[first, second], + evaluation_time=now, + ) + + assert [(interval.starts_at, interval.expires_at) for interval in intervals] == [ + (first.starts_at, first.expires_at), + (second.starts_at, second.expires_at), + ] + + def test_skips_interval_expiring_at_evaluation_time(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + expired = make_source_entitlement( + starts_at=now - datetime.timedelta(days=1), + expires_at=now, + ) + + assert ( + domain.build_effective_timeline( + user_id=expired.user_id, + kind_id=expired.kind_id, + merge_policy=MergePolicy.max, + source_entitlements=[expired], + evaluation_time=now, + ) + == [] + ) + + +class TestEffectiveStateAt: + def test_empty_intervals(self) -> None: + assert domain.effective_state_at([], datetime.datetime.now(tz=datetime.UTC)) == (False, None) + + def test_half_open_interval(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + interval = make_effective_entitlement_interval( + user_id=new_user_id(), + kind_id=_DAY_TOKENS, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + assert domain.effective_state_at([interval], interval.starts_at) == (True, 10) + assert domain.effective_state_at([interval], interval.expires_at) == (False, None) + + +class TestApplySourceChange: + @pytest.mark.asyncio + async def test_stores_new_source_state_and_effective_interval(self) -> None: + source_state = make_source_entitlement() + evaluation_time = datetime.datetime.now(tz=datetime.UTC) + + async with ( + TableSizeDelta("en_source_entitlements", delta=1), + TableSizeDelta("en_entitlements", delta=1), + TableSizeDelta("a_records", delta=1), + ): + async with transaction() as transaction_execute: + outcome = await domain._apply_source_change( + transaction_execute, + kind=domain.get_entitlement_kind(source_state.kind_id), + new_source_state=source_state, + evaluation_time=evaluation_time, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert outcome.changed + assert outcome.effective_state == (True, source_state.value) + assert len(outcome.effective_intervals) == 1 + + +class TestEmitBusinessEvents: + def test_emits_source_and_effective_state(self) -> None: + source_state = make_source_entitlement() + effective_interval = make_effective_entitlement_interval( + user_id=source_state.user_id, + kind_id=source_state.kind_id, + value=cast(int, source_state.value), + starts_at=source_state.starts_at, + expires_at=source_state.expires_at, + ) + outcome = domain._SourceChangeOutcome( + changed=True, + effective_state=(True, source_state.value), + effective_intervals=[effective_interval], + ) + + with capture_logs() as logs: + domain._emit_business_events(source_state, outcome) + + assert_logs_has_business_event( + logs, + "source_entitlement_changed", + user_id=source_state.user_id, + source=source_state.source, + kind_id=source_state.kind_id.value, + granted=True, + value=source_state.value, + starts_at=source_state.starts_at.isoformat(), + expires_at=source_state.expires_at.isoformat(), + ) + assert_logs_has_business_event( + logs, + "entitlement_changed", + user_id=source_state.user_id, + kind_id=source_state.kind_id.value, + granted=True, + value=source_state.value, + new_effective_intervals=[ + { + "value": effective_interval.value, + "starts_at": effective_interval.starts_at.isoformat(), + "expires_at": effective_interval.expires_at.isoformat(), + } + ], + ) + + +class TestChangeSourceEntitlement: + @pytest.mark.asyncio + async def test_invalid_input_does_not_change_persistence(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + + async with TableSizeNotChanged("en_source_entitlements"): + async with TableSizeNotChanged("en_entitlements"): + async with TableSizeNotChanged("a_records"): + with pytest.raises(errors.InvalidSourceEntitlement): + await domain.change_source_entitlement( + source=EntitlementSourceId(""), + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + @pytest.mark.asyncio + async def test_stores_state_timeline_audit_and_events(self) -> None: + user_id = new_user_id() + starts_at = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(days=1) + expires_at = starts_at + datetime.timedelta(days=2) + + with capture_logs() as logs: + async with TableSizeDelta("en_source_entitlements", delta=1): + async with TableSizeDelta("en_entitlements", delta=1): + async with TableSizeDelta("a_records", delta=1): + state = await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert state == (True, 10) + expected_source = make_source_entitlement( + user_id=user_id, + starts_at=starts_at, + expires_at=expires_at, + ) + assert ( + await operations.load_source_entitlement( + execute, + user_id, + _DAY_TOKENS, + EntitlementSourceId("test"), + ) + == expected_source + ) + expected_interval = make_effective_entitlement_interval( + user_id=user_id, + kind_id=_DAY_TOKENS, + value=10, + starts_at=starts_at, + expires_at=expires_at, + ) + intervals = await operations.load_effective_intervals( + execute, + user_id, + _DAY_TOKENS, + ending_after=starts_at, + ) + assert intervals == [expected_interval] + records = await audit_domain.load_records_for_subject( + execute, + subject_kind=AuditEntityKind.user, + subject_id=SerializedId(str(user_id)), + ) + assert len(records) == 1 + assert records[0].attributes == { + "source": "test", + "kind_id": _DAY_TOKENS.value, + "previous_source_state": None, + "new_source_state": cast(dict[str, object], expected_source.model_dump(mode="json")), + "previous_effective_intervals": [], + "new_effective_intervals": [cast(dict[str, object], expected_interval.model_dump(mode="json"))], + } + assert_logs_has_business_event( + logs, + "source_entitlement_changed", + user_id=user_id, + source="test", + kind_id=_DAY_TOKENS.value, + granted=True, + value=10, + starts_at=starts_at.isoformat(), + expires_at=expires_at.isoformat(), + ) + assert_logs_has_business_event( + logs, + "entitlement_changed", + user_id=user_id, + kind_id=_DAY_TOKENS.value, + granted=True, + value=10, + new_effective_intervals=[ + {"value": 10, "starts_at": starts_at.isoformat(), "expires_at": expires_at.isoformat()} + ], + ) + + @pytest.mark.asyncio + async def test_identical_state_is_no_op(self) -> None: + user_id = new_user_id() + starts_at = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(days=1) + expires_at = starts_at + datetime.timedelta(days=2) + await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + with capture_logs() as logs: + async with TableSizeNotChanged("en_source_entitlements"): + async with TableSizeNotChanged("en_entitlements"): + async with TableSizeNotChanged("a_records"): + state = await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert state == (True, 10) + assert_logs_has_no_business_event(logs, "source_entitlement_changed") + assert_logs_has_no_business_event(logs, "entitlement_changed") + + @pytest.mark.asyncio + async def test_multiple_sources_merge_and_revoke_independently(self) -> None: + user_id = new_user_id() + starts_at = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(days=1) + expires_at = starts_at + datetime.timedelta(days=3) + async with ( + TableSizeDelta("en_source_entitlements", delta=1), + TableSizeDelta("en_entitlements", delta=1), + TableSizeDelta("a_records", delta=1), + ): + await domain.change_source_entitlement( + source=EntitlementSourceId("first"), + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + async with ( + TableSizeDelta("en_source_entitlements", delta=1), + TableSizeNotChanged("en_entitlements"), + TableSizeDelta("a_records", delta=1), + ): + state = await domain.change_source_entitlement( + source=EntitlementSourceId("second"), + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=20, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + assert state == (True, 20) + + async with ( + TableSizeNotChanged("en_source_entitlements"), + TableSizeNotChanged("en_entitlements"), + TableSizeDelta("a_records", delta=1), + ): + state = await domain.change_source_entitlement( + source=EntitlementSourceId("second"), + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=False, + value=None, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert state == (True, 10) + sources = await operations.load_source_entitlements(execute, user_id, _DAY_TOKENS) + assert len(sources) == 2 + assert {source.source: source.granted for source in sources} == {"first": True, "second": False} + + @pytest.mark.asyncio + async def test_future_state_replaces_current_source_contribution(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + async with ( + TableSizeDelta("en_source_entitlements", delta=1), + TableSizeDelta("en_entitlements", delta=1), + TableSizeDelta("a_records", delta=1), + ): + await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=now - datetime.timedelta(days=1), + expires_at=now + datetime.timedelta(days=1), + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + future_start = now + datetime.timedelta(days=2) + async with ( + TableSizeNotChanged("en_source_entitlements"), + TableSizeNotChanged("en_entitlements"), + TableSizeDelta("a_records", delta=1), + ): + state = await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=20, + starts_at=future_start, + expires_at=future_start + datetime.timedelta(days=1), + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert state == (False, None) + intervals = await operations.load_effective_intervals( + execute, + user_id, + _DAY_TOKENS, + ending_after=now, + ) + assert [(interval.value, interval.starts_at) for interval in intervals] == [(20, future_start)] + + @pytest.mark.asyncio + async def test_audit_failure_rolls_back_without_events(self, mocker: MockerFixture) -> None: + user_id = new_user_id() + starts_at = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(days=1) + expires_at = starts_at + datetime.timedelta(days=2) + mocker.patch.object(audit_domain, "record", side_effect=RuntimeError("audit failed")) + + with capture_logs() as logs: + async with TableSizeNotChanged("en_source_entitlements"): + async with TableSizeNotChanged("en_entitlements"): + async with TableSizeNotChanged("a_records"): + with pytest.raises(RuntimeError, match="audit failed"): + await domain.change_source_entitlement( + source=_SOURCE, + user_id=user_id, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + assert_logs_has_no_business_event(logs, "source_entitlement_changed") + assert_logs_has_no_business_event(logs, "entitlement_changed") + + +class TestCleanupExpiredEntitlements: + @pytest.mark.asyncio + async def test_no_expired_rows(self) -> None: + await clear_effective_intervals() + + async with TableSizeNotChanged("en_entitlements"): + deleted = await domain.cleanup_expired_entitlements() + + assert deleted == 0 + + @pytest.mark.asyncio + async def test_deletes_expired_effective_rows_only(self) -> None: + await clear_effective_intervals() + + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + expired = make_effective_entitlement_interval( + user_id=user_id, + kind_id=_DAY_TOKENS, + value=10, + starts_at=now - datetime.timedelta(days=2), + expires_at=now - datetime.timedelta(days=1), + ) + source = make_source_entitlement( + user_id=user_id, + kind_id=_DAY_TOKENS, + starts_at=expired.starts_at, + expires_at=expired.expires_at, + ) + await operations.upsert_source_entitlement(execute, source) + + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, _DAY_TOKENS, [expired]) + + async with TableSizeNotChanged("en_source_entitlements"): + async with TableSizeDelta("en_entitlements", delta=-1): + deleted = await domain.cleanup_expired_entitlements() + + assert deleted == 1 + assert await operations.load_source_entitlement(execute, user_id, _DAY_TOKENS, source.source) == source + + +class TestGetEntitlements: + @pytest.mark.asyncio + async def test_returns_every_user_and_selected_kind(self) -> None: + entitled_user = new_user_id() + other_user = new_user_id() + starts_at = datetime.datetime.now(tz=datetime.UTC) - datetime.timedelta(days=1) + expires_at = starts_at + datetime.timedelta(days=2) + await domain.change_source_entitlement( + source=_SOURCE, + user_id=entitled_user, + kind_id=_DAY_TOKENS, + granted=True, + value=10, + starts_at=starts_at, + expires_at=expires_at, + actor_kind=_ACTOR_KIND, + actor_id=_ACTOR_ID, + ) + + listed = await domain.get_entitlements( + [entitled_user, other_user], + [_DAY_TOKENS, _MONTH_TOKENS], + ) + + assert listed == { + entitled_user: { + _DAY_TOKENS: make_effective_entitlement_interval( + user_id=entitled_user, + kind_id=_DAY_TOKENS, + value=10, + starts_at=starts_at, + expires_at=expires_at, + ), + _MONTH_TOKENS: None, + }, + other_user: {_DAY_TOKENS: None, _MONTH_TOKENS: None}, + } + + @pytest.mark.asyncio + async def test_empty_kind_list_selects_all_configured_kinds(self) -> None: + user_id = new_user_id() + + assert await domain.get_entitlements([user_id], []) == {user_id: {_DAY_TOKENS: None, _MONTH_TOKENS: None}} + + @pytest.mark.asyncio + async def test_empty_user_list(self) -> None: + assert await domain.get_entitlements([], [_DAY_TOKENS]) == {} + + @pytest.mark.asyncio + async def test_duplicate_user_ids(self) -> None: + user_id = new_user_id() + + assert await domain.get_entitlements([user_id, user_id], [_DAY_TOKENS]) == {user_id: {_DAY_TOKENS: None}} + + @pytest.mark.asyncio + async def test_duplicate_kind_ids(self) -> None: + user_id = new_user_id() + + assert await domain.get_entitlements([user_id], [_DAY_TOKENS, _DAY_TOKENS]) == {user_id: {_DAY_TOKENS: None}} + + @pytest.mark.asyncio + async def test_unconfigured_kind(self, mocker: MockerFixture) -> None: + mocker.patch.object( + entitlement_entities, + "ENTITLEMENT_KINDS", + (entitlement_entities.ENTITLEMENT_KINDS[0],), + ) + + with pytest.raises(errors.UnknownEntitlementKind): + await domain.get_entitlements([new_user_id()], [_MONTH_TOKENS]) + + @pytest.mark.asyncio + async def test_query_does_not_remove_expired_rows(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + expired = make_effective_entitlement_interval( + user_id=user_id, + kind_id=_DAY_TOKENS, + value=10, + starts_at=now - datetime.timedelta(days=2), + expires_at=now - datetime.timedelta(days=1), + ) + + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, _DAY_TOKENS, [expired]) + + async with TableSizeNotChanged("en_entitlements"): + listed = await domain.get_entitlements([user_id], [_DAY_TOKENS]) + + assert listed == {user_id: {_DAY_TOKENS: None}} diff --git a/ffun/ffun/entitlements/tests/test_entities.py b/ffun/ffun/entitlements/tests/test_entities.py new file mode 100644 index 00000000..861afbaf --- /dev/null +++ b/ffun/ffun/entitlements/tests/test_entities.py @@ -0,0 +1,127 @@ +import datetime +from typing import cast + +import pydantic +import pytest + +from ffun.entitlements.entities import ENTITLEMENT_KINDS, EntitlementKindId, MergePolicy +from ffun.entitlements.tests.make import make_effective_entitlement_interval, make_source_entitlement + + +class TestEntitlementKindId: + def test_members_are_closed_and_stable(self) -> None: + assert list(EntitlementKindId) == [ + EntitlementKindId.day_tokens, + EntitlementKindId.month_tokens, + ] + assert [kind_id.value for kind_id in EntitlementKindId] == [1, 2] + + with pytest.raises(ValueError): + EntitlementKindId(999) + + +class TestEntitlementKinds: + def test_registry_defines_every_kind_once(self) -> None: + assert [kind.id for kind in ENTITLEMENT_KINDS] == list(EntitlementKindId) + assert [kind.merge_policy for kind in ENTITLEMENT_KINDS] == [MergePolicy.max, MergePolicy.max] + + +class TestSourceEntitlement: + def test_init__granted_requires_boolean(self) -> None: + with pytest.raises(pydantic.ValidationError, match="valid boolean"): + make_source_entitlement(granted=cast(bool, 1)) + + @pytest.mark.parametrize("value", [None, True]) + def test_init__grant_requires_integer_value(self, value: object) -> None: + with pytest.raises(pydantic.ValidationError, match="integer"): + make_source_entitlement(value=cast(int | None, value)) + + def test_init__revocation_requires_no_value(self) -> None: + with pytest.raises(pydantic.ValidationError, match="must not have a value"): + make_source_entitlement(granted=False, value=10) + + def test_init__activation_requires_utc_offset(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must have a UTC offset"): + make_source_entitlement(starts_at=now.replace(tzinfo=None)) + + def test_init__expiration_requires_utc_offset(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="expiration timestamp must have a UTC offset"): + make_source_entitlement(expires_at=now.replace(tzinfo=None)) + + def test_init__activation_must_be_before_expiration(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must be earlier than expiration"): + make_source_entitlement(starts_at=now, expires_at=now) + + def test_to_revoked__returns_revocation_for_new_interval(self) -> None: + entitlement = make_source_entitlement() + starts_at = entitlement.expires_at + expires_at = starts_at + datetime.timedelta(days=1) + + revoked = entitlement.to_revoked(starts_at=starts_at, expires_at=expires_at) + + assert revoked == make_source_entitlement( + source=entitlement.source, + user_id=entitlement.user_id, + kind_id=entitlement.kind_id, + granted=False, + value=None, + starts_at=starts_at, + expires_at=expires_at, + ) + + def test_to_revoked__validates_new_interval(self) -> None: + entitlement = make_source_entitlement() + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must be earlier than expiration"): + entitlement.to_revoked(starts_at=now, expires_at=now) + + def test_to_granted__returns_grant_for_new_interval(self) -> None: + entitlement = make_source_entitlement(granted=False, value=None) + starts_at = entitlement.expires_at + expires_at = starts_at + datetime.timedelta(days=1) + + granted = entitlement.to_granted(value=20, starts_at=starts_at, expires_at=expires_at) + + assert granted == make_source_entitlement( + source=entitlement.source, + user_id=entitlement.user_id, + kind_id=entitlement.kind_id, + granted=True, + value=20, + starts_at=starts_at, + expires_at=expires_at, + ) + + def test_to_granted__validates_new_interval(self) -> None: + entitlement = make_source_entitlement(granted=False, value=None) + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must be earlier than expiration"): + entitlement.to_granted(value=20, starts_at=now, expires_at=now) + + +class TestEffectiveEntitlementInterval: + def test_init__activation_requires_utc_offset(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must have a UTC offset"): + make_effective_entitlement_interval(starts_at=now.replace(tzinfo=None)) + + def test_init__expiration_requires_utc_offset(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="expiration timestamp must have a UTC offset"): + make_effective_entitlement_interval(expires_at=now.replace(tzinfo=None)) + + def test_init__activation_must_be_before_expiration(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + + with pytest.raises(pydantic.ValidationError, match="activation timestamp must be earlier than expiration"): + make_effective_entitlement_interval(starts_at=now, expires_at=now) diff --git a/ffun/ffun/entitlements/tests/test_operations.py b/ffun/ffun/entitlements/tests/test_operations.py new file mode 100644 index 00000000..cf62f9fc --- /dev/null +++ b/ffun/ffun/entitlements/tests/test_operations.py @@ -0,0 +1,532 @@ +import asyncio +import datetime +from typing import cast + +import pytest +from psycopg.errors import UniqueViolation +from pydantic import ValidationError + +from ffun.core.postgresql import execute, transaction +from ffun.core.tests.helpers import TableSizeDelta, TableSizeNotChanged +from ffun.domain.domain import new_user_id +from ffun.entitlements import errors, operations +from ffun.entitlements.entities import EntitlementKindId, EntitlementSourceId +from ffun.entitlements.tests.helpers import load_effective_interval_timestamps, load_source_entitlement_timestamps +from ffun.entitlements.tests.make import make_effective_entitlement_interval, make_source_entitlement + + +class TestRowToSourceEntitlement: + def test_converts_row(self) -> None: + entitlement = make_source_entitlement() + + assert operations.row_to_source_entitlement(entitlement.model_dump()) == entitlement # type: ignore[misc] + + def test_invalid_row_raises_module_error(self) -> None: + with pytest.raises(errors.InvalidStoredEntitlement) as exception_info: + operations.row_to_source_entitlement({}) + + assert "entity_kind=source_entitlement" in str(exception_info.value) + assert isinstance(exception_info.value.__cause__, ValidationError) + + +class TestRowToEffectiveInterval: + def test_converts_row(self) -> None: + now = datetime.datetime.now(tz=datetime.UTC) + interval = make_effective_entitlement_interval( + user_id=new_user_id(), + kind_id=EntitlementKindId.day_tokens, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + assert operations.row_to_effective_interval(interval.model_dump()) == interval # type: ignore[misc] + + def test_invalid_row_raises_module_error(self) -> None: + with pytest.raises(errors.InvalidStoredEntitlement) as exception_info: + operations.row_to_effective_interval({}) + + assert "entity_kind=effective_entitlement_interval" in str(exception_info.value) + assert isinstance(exception_info.value.__cause__, ValidationError) + + +class TestLoadSourceEntitlement: + @pytest.mark.asyncio + async def test_missing(self) -> None: + loaded = await operations.load_source_entitlement( + execute, + new_user_id(), + EntitlementKindId.day_tokens, + EntitlementSourceId("missing"), + ) + + assert loaded is None + + @pytest.mark.asyncio + async def test_loads_source_entitlement(self) -> None: + entitlement = make_source_entitlement() + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + assert ( + await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + == entitlement + ) + + +class TestUpsertSourceEntitlement: + @pytest.mark.asyncio + async def test_inserts_grant(self) -> None: + entitlement = make_source_entitlement() + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + assert ( + await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + == entitlement + ) + + created_at, updated_at = await load_source_entitlement_timestamps(entitlement) + assert created_at == updated_at + + @pytest.mark.asyncio + async def test_inserts_revocation(self) -> None: + entitlement = make_source_entitlement(granted=False, value=None) + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + assert ( + await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + == entitlement + ) + + created_at, updated_at = await load_source_entitlement_timestamps(entitlement) + assert created_at == updated_at + + @pytest.mark.asyncio + async def test_replaces_grant(self) -> None: + entitlement = make_source_entitlement() + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + created_at, _ = await load_source_entitlement_timestamps(entitlement) + + await asyncio.sleep(0.001) + + replacement = entitlement.to_granted( + value=20, + starts_at=entitlement.starts_at, + expires_at=entitlement.expires_at, + ) + + async with TableSizeNotChanged("en_source_entitlements"): + await operations.upsert_source_entitlement(execute, replacement) + + loaded = await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + assert loaded == replacement + + replaced_created_at, replaced_updated_at = await load_source_entitlement_timestamps(replacement) + assert replaced_created_at == created_at + assert replaced_updated_at > created_at + + @pytest.mark.asyncio + async def test_replaces_with_revocation(self) -> None: + entitlement = make_source_entitlement() + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + replacement = entitlement.to_revoked( + starts_at=entitlement.starts_at, + expires_at=entitlement.expires_at, + ) + + async with TableSizeNotChanged("en_source_entitlements"): + await operations.upsert_source_entitlement(execute, replacement) + + assert ( + await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + == replacement + ) + + @pytest.mark.asyncio + async def test_replaces_revocation_with_grant(self) -> None: + entitlement = make_source_entitlement(granted=False, value=None) + + async with TableSizeDelta("en_source_entitlements", delta=1): + await operations.upsert_source_entitlement(execute, entitlement) + + replacement = entitlement.to_granted( + value=20, + starts_at=entitlement.starts_at, + expires_at=entitlement.expires_at, + ) + + async with TableSizeNotChanged("en_source_entitlements"): + await operations.upsert_source_entitlement(execute, replacement) + + assert ( + await operations.load_source_entitlement( + execute, + entitlement.user_id, + entitlement.kind_id, + entitlement.source, + ) + == replacement + ) + + +class TestLoadSourceEntitlements: + @pytest.mark.asyncio + async def test_no_source_entitlements(self) -> None: + assert ( + await operations.load_source_entitlements( + execute, + new_user_id(), + EntitlementKindId.day_tokens, + ) + == [] + ) + + @pytest.mark.asyncio + async def test_loads_all_sources_in_time_order(self) -> None: + user_id = new_user_id() + now = datetime.datetime.now(tz=datetime.UTC) + later = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("later"), + starts_at=now, + expires_at=now + datetime.timedelta(days=2), + ) + earlier = make_source_entitlement( + user_id=user_id, + source=EntitlementSourceId("earlier"), + starts_at=now - datetime.timedelta(days=1), + expires_at=now + datetime.timedelta(days=1), + ) + await operations.upsert_source_entitlement(execute, later) + await operations.upsert_source_entitlement(execute, earlier) + + loaded = await operations.load_source_entitlements(execute, user_id, EntitlementKindId.day_tokens) + + assert loaded == [earlier, later] + + +class TestLoadEffectiveIntervals: + @pytest.mark.asyncio + async def test_no_effective_intervals(self) -> None: + assert ( + await operations.load_effective_intervals( + execute, + new_user_id(), + EntitlementKindId.day_tokens, + ending_after=datetime.datetime.min.replace(tzinfo=datetime.UTC), + ) + == [] + ) + + @pytest.mark.asyncio + async def test_loads_intervals_ending_after_boundary(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + now = datetime.datetime.now(tz=datetime.UTC) + expired = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=now - datetime.timedelta(days=2), + expires_at=now, + ) + active = expired.replace( + value=20, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + async with TableSizeDelta("en_entitlements", delta=2): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals( + transaction_execute, + user_id, + kind_id, + [expired, active], + ) + + assert await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=now, + ) == [active] + + +class TestReplaceEffectiveIntervals: + @pytest.mark.asyncio + async def test_replaces_complete_user_kind_timeline(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + now = datetime.datetime.now(tz=datetime.UTC) + first = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=now - datetime.timedelta(days=1), + expires_at=now + datetime.timedelta(days=1), + ) + second = first.replace(value=20, starts_at=first.expires_at, expires_at=now + datetime.timedelta(days=2)) + + async with TableSizeDelta("en_entitlements", delta=2): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [first, second]) + + loaded = await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=now, + ) + assert loaded == [first, second] + + timestamp_rows = await load_effective_interval_timestamps(user_id, kind_id) + assert len(timestamp_rows) == 2 + assert all(created_at == updated_at for created_at, updated_at in timestamp_rows) + + async with TableSizeDelta("en_entitlements", delta=-1): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [second]) + + assert await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=now, + ) == [second] + + @pytest.mark.asyncio + async def test_empty_intervals_delete_complete_timeline(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + now = datetime.datetime.now(tz=datetime.UTC) + interval = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + async with TableSizeDelta("en_entitlements", delta=1): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [interval]) + + async with TableSizeDelta("en_entitlements", delta=-1): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, []) + + assert ( + await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=now, + ) + == [] + ) + + @pytest.mark.asyncio + async def test_insert_failure_rolls_back_timeline_deletion(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + now = datetime.datetime.now(tz=datetime.UTC) + original = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [original]) + + duplicate = original.replace(value=20) + unique_violation = cast(type[Exception], UniqueViolation) + + async with TableSizeNotChanged("en_entitlements"): + with pytest.raises(unique_violation): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals( + transaction_execute, + user_id, + kind_id, + [duplicate, duplicate], + ) + + assert await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=now, + ) == [original] + + +class TestLoadActiveIntervals: + @pytest.mark.asyncio + async def test_half_open_interval(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + starts_at = datetime.datetime.now(tz=datetime.UTC) + interval = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=starts_at, + expires_at=starts_at + datetime.timedelta(days=1), + ) + + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [interval]) + + assert await operations.load_active_intervals(execute, [user_id], [kind_id], evaluation_time=starts_at) == [ + interval + ] + assert ( + await operations.load_active_intervals(execute, [user_id], [kind_id], evaluation_time=interval.expires_at) + == [] + ) + + @pytest.mark.asyncio + async def test_empty_user_filter(self) -> None: + assert ( + await operations.load_active_intervals( + execute, + [], + [EntitlementKindId.day_tokens], + evaluation_time=datetime.datetime.now(tz=datetime.UTC), + ) + == [] + ) + + @pytest.mark.asyncio + async def test_empty_kind_filter(self) -> None: + assert ( + await operations.load_active_intervals( + execute, + [new_user_id()], + [], + evaluation_time=datetime.datetime.now(tz=datetime.UTC), + ) + == [] + ) + + @pytest.mark.asyncio + async def test_duplicate_user_filter(self) -> None: + interval = make_effective_entitlement_interval() + + async with TableSizeDelta("en_entitlements", delta=1): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals( + transaction_execute, + interval.user_id, + interval.kind_id, + [interval], + ) + + assert await operations.load_active_intervals( + execute, + [interval.user_id, interval.user_id], + [interval.kind_id], + evaluation_time=interval.starts_at, + ) == [interval] + + @pytest.mark.asyncio + async def test_duplicate_kind_filter(self) -> None: + interval = make_effective_entitlement_interval() + + async with TableSizeDelta("en_entitlements", delta=1): + async with transaction() as transaction_execute: + await operations.replace_effective_intervals( + transaction_execute, + interval.user_id, + interval.kind_id, + [interval], + ) + + assert await operations.load_active_intervals( + execute, + [interval.user_id], + [interval.kind_id, interval.kind_id], + evaluation_time=interval.starts_at, + ) == [interval] + + +class TestDeleteExpiredEffectiveIntervals: + @pytest.mark.asyncio + async def test_no_expired_intervals(self) -> None: + cleanup_time = datetime.datetime.min.replace(tzinfo=datetime.UTC) + + async with TableSizeNotChanged("en_entitlements"): + deleted = await operations.delete_expired_effective_intervals(execute, cleanup_time) + + assert deleted == 0 + + @pytest.mark.asyncio + async def test_deletes_only_expired_intervals(self) -> None: + user_id = new_user_id() + kind_id = EntitlementKindId.day_tokens + now = datetime.datetime.now(tz=datetime.UTC) + cleanup_time = now - datetime.timedelta(days=2) + expired = make_effective_entitlement_interval( + user_id=user_id, + kind_id=kind_id, + value=10, + starts_at=now - datetime.timedelta(days=4), + expires_at=cleanup_time, + ) + active = expired.replace( + starts_at=now, + expires_at=now + datetime.timedelta(days=1), + ) + + async with transaction() as transaction_execute: + await operations.replace_effective_intervals(transaction_execute, user_id, kind_id, [expired, active]) + + async with TableSizeDelta("en_entitlements", delta=-1): + deleted = await operations.delete_expired_effective_intervals(execute, cleanup_time) + + assert deleted == 1 + assert await operations.load_effective_intervals( + execute, + user_id, + kind_id, + ending_after=datetime.datetime.min.replace(tzinfo=datetime.UTC), + ) == [active] diff --git a/ffun/ffun/locks/__init__.py b/ffun/ffun/locks/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ffun/ffun/locks/domain.py b/ffun/ffun/locks/domain.py new file mode 100644 index 00000000..39707ef5 --- /dev/null +++ b/ffun/ffun/locks/domain.py @@ -0,0 +1,100 @@ +import contextlib +import re +import uuid +from collections.abc import AsyncIterator +from types import TracebackType + +from ffun.core.postgresql import ExecuteType, transaction +from ffun.locks import errors, operations +from ffun.locks.entities import LockKind + +_LOCK_KIND_PATTERN = re.compile(r"[a-z][a-z0-9]*(?:_[a-z0-9]+)*") +_LOCK_ARGUMENT_PATTERN = re.compile(r"[A-Za-z0-9._:@/-]+") +_LOCK_KIND_MAX_BYTES = 128 +_LOCK_KEY_MAX_BYTES = 1024 + + +class Lock: + __slots__ = ("_execute", "_raw_lock_kind", "_lock_arguments", "_acquired_identity") + + def __init__(self, execute: ExecuteType, lock_kind: LockKind, *lock_arguments: object) -> None: + self._execute = execute + self._raw_lock_kind = lock_kind + self._lock_arguments = lock_arguments + self._acquired_identity: tuple[str, str] | None = None + + @staticmethod + def _canonicalize_argument(argument: object) -> str: # noqa: CCR001 + if isinstance(argument, bool): + value = "true" if argument else "false" + elif isinstance(argument, int): + try: + value = str(int(argument)) + except ValueError as exception: + raise errors.InvalidLockKey(reason="integer lock argument exceeds the supported size") from exception + elif isinstance(argument, uuid.UUID): + value = str(argument) + elif isinstance(argument, str): + value = str(argument) + else: + raise errors.InvalidLockKey(reason=f"unsupported lock argument type: {type(argument).__name__}") + + if _LOCK_ARGUMENT_PATTERN.fullmatch(value) is None: + raise errors.InvalidLockKey( + reason="lock arguments must be non-empty and contain only supported ASCII characters" + ) + + return value + + @classmethod + def _build_identity(cls, lock_kind: str, lock_arguments: tuple[object, ...]) -> tuple[str, str]: + if not isinstance(lock_kind, str) or _LOCK_KIND_PATTERN.fullmatch(lock_kind) is None: + raise errors.InvalidLockKey(reason="lock kind must be a non-empty lowercase snake_case string") + + if len(lock_kind.encode("utf-8")) > _LOCK_KIND_MAX_BYTES: + raise errors.InvalidLockKey(reason=f"lock kind must not exceed {_LOCK_KIND_MAX_BYTES} bytes") + + lock_key = "|".join(cls._canonicalize_argument(argument) for argument in lock_arguments) + + if len(lock_key.encode("utf-8")) > _LOCK_KEY_MAX_BYTES: + raise errors.InvalidLockKey(reason=f"lock key must not exceed {_LOCK_KEY_MAX_BYTES} bytes") + + return lock_kind, lock_key + + async def __aenter__(self) -> "Lock": + lock_kind, lock_key = self._build_identity(self._raw_lock_kind, self._lock_arguments) + await operations.acquire(self._execute, lock_kind, lock_key) + self._acquired_identity = (lock_kind, lock_key) + return self + + async def __aexit__( + self, + exception_type: type[BaseException] | None, + exception: BaseException | None, + traceback: TracebackType | None, + ) -> bool: + if self._acquired_identity is None: + raise RuntimeError("Lock context exited before acquisition") + + lock_kind, lock_key = self._acquired_identity + + try: + await operations.release(self._execute, lock_kind, lock_key) + except BaseException as cleanup_exception: + if exception is None: + raise + + exception.add_note(f"Lock cleanup failed: {cleanup_exception!r}") + finally: + self._acquired_identity = None + + return False + + +@contextlib.asynccontextmanager +async def locked_transaction(lock_kind: LockKind, *lock_arguments: object) -> AsyncIterator[ExecuteType]: + Lock._build_identity(lock_kind, lock_arguments) + + async with transaction() as execute: + async with Lock(execute, lock_kind, *lock_arguments): + yield execute diff --git a/ffun/ffun/locks/entities.py b/ffun/ffun/locks/entities.py new file mode 100644 index 00000000..ce7871d6 --- /dev/null +++ b/ffun/ffun/locks/entities.py @@ -0,0 +1,3 @@ +from typing import NewType + +LockKind = NewType("LockKind", str) diff --git a/ffun/ffun/locks/errors.py b/ffun/ffun/locks/errors.py new file mode 100644 index 00000000..0a251f51 --- /dev/null +++ b/ffun/ffun/locks/errors.py @@ -0,0 +1,13 @@ +from ffun.core import errors + + +class Error(errors.Error): + pass + + +class InvalidLockKey(Error): + pass + + +class LockInvariantViolation(Error): + pass diff --git a/ffun/ffun/locks/migrations/20260719_01_c0DeX-locks.py b/ffun/ffun/locks/migrations/20260719_01_c0DeX-locks.py new file mode 100644 index 00000000..0b9a79b6 --- /dev/null +++ b/ffun/ffun/locks/migrations/20260719_01_c0DeX-locks.py @@ -0,0 +1,33 @@ +""" +locks +""" + +from typing import Any + +from psycopg import Connection +from yoyo import step + +__depends__: set[str] = set() + + +sql_create_locks = """ +-- Ephemeral exact-key rows used to coordinate holder transactions. +CREATE TABLE lk_locks ( + lock_kind TEXT COLLATE "C" NOT NULL, + lock_key TEXT COLLATE "C" NOT NULL, + PRIMARY KEY (lock_kind, lock_key) +) +""" + + +def apply_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute(sql_create_locks) + + +def rollback_step(conn: Connection[dict[str, Any]]) -> None: + cursor = conn.cursor() + cursor.execute("DROP TABLE lk_locks") + + +steps = [step(apply_step, rollback_step)] diff --git a/ffun/ffun/locks/operations.py b/ffun/ffun/locks/operations.py new file mode 100644 index 00000000..e8a30d21 --- /dev/null +++ b/ffun/ffun/locks/operations.py @@ -0,0 +1,38 @@ +import psycopg + +from ffun.core.postgresql import ExecuteType +from ffun.locks import errors + + +async def acquire(execute: ExecuteType, lock_kind: str, lock_key: str) -> None: + sql = """ + INSERT INTO lk_locks (lock_kind, lock_key) + VALUES (%(lock_kind)s, %(lock_key)s) + """ + + try: + await execute(sql, {"lock_kind": lock_kind, "lock_key": lock_key}) + except psycopg.errors.UniqueViolation as exception: + raise errors.LockInvariantViolation( + lock_kind=lock_kind, + lock_key=lock_key, + reason="an acquisition row already exists", + ) from exception + + +async def release(execute: ExecuteType, lock_kind: str, lock_key: str) -> None: + sql = """ + DELETE FROM lk_locks + WHERE lock_kind = %(lock_kind)s + AND lock_key = %(lock_key)s + RETURNING lock_kind + """ + + rows = await execute(sql, {"lock_kind": lock_kind, "lock_key": lock_key}) + + if len(rows) != 1: + raise errors.LockInvariantViolation( + lock_kind=lock_kind, + lock_key=lock_key, + reason=f"expected to delete one acquisition row, deleted {len(rows)}", + ) diff --git a/ffun/ffun/locks/tests/__init__.py b/ffun/ffun/locks/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ffun/ffun/locks/tests/helpers.py b/ffun/ffun/locks/tests/helpers.py new file mode 100644 index 00000000..8bbecf56 --- /dev/null +++ b/ffun/ffun/locks/tests/helpers.py @@ -0,0 +1,41 @@ +from typing import cast + +from ffun.core.postgresql import ExecuteType + + +async def count_acquisition_rows(execute: ExecuteType) -> int: + rows = cast( + list[dict[str, int]], + await execute( + """ + SELECT count(*) AS number + FROM lk_locks + """ + ), + ) + return rows[0]["number"] + + +async def load_acquisition_rows( + execute: ExecuteType, + lock_kind: str, + lock_key: str | None = None, +) -> list[dict[str, object]]: + if lock_key is None: + sql = """ + SELECT lock_kind, lock_key + FROM lk_locks + WHERE lock_kind = %(lock_kind)s + ORDER BY lock_key + """ + arguments = {"lock_kind": lock_kind} + else: + sql = """ + SELECT lock_kind, lock_key + FROM lk_locks + WHERE lock_kind = %(lock_kind)s + AND lock_key = %(lock_key)s + """ + arguments = {"lock_kind": lock_kind, "lock_key": lock_key} + + return cast(list[dict[str, object]], await execute(sql, arguments)) diff --git a/ffun/ffun/locks/tests/make.py b/ffun/ffun/locks/tests/make.py new file mode 100644 index 00000000..0bd4763e --- /dev/null +++ b/ffun/ffun/locks/tests/make.py @@ -0,0 +1,7 @@ +import uuid + +from ffun.locks.entities import LockKind + + +def new_lock_kind(prefix: str = "test_lock") -> LockKind: + return LockKind(f"{prefix}_{uuid.uuid4().hex}") diff --git a/ffun/ffun/locks/tests/test_domain.py b/ffun/ffun/locks/tests/test_domain.py new file mode 100644 index 00000000..981f0179 --- /dev/null +++ b/ffun/ffun/locks/tests/test_domain.py @@ -0,0 +1,481 @@ +import asyncio +import enum +import uuid +from typing import cast + +import pytest +import pytest_asyncio +from psycopg.errors import UniqueViolation +from pytest_mock import MockerFixture + +from ffun.audit import domain as audit_domain +from ffun.audit.entities import AuditEntityKind, AuditEventName +from ffun.core.postgresql import ExecuteType, execute, transaction +from ffun.core.tests.helpers import TableSizeDelta, TableSizeNotChanged, assert_pool_capacity_at_least +from ffun.domain.entities import SerializedId +from ffun.locks import domain as locks_domain +from ffun.locks import errors +from ffun.locks.domain import Lock, locked_transaction +from ffun.locks.entities import LockKind +from ffun.locks.tests.helpers import load_acquisition_rows +from ffun.locks.tests.make import new_lock_kind + + +class ExampleInt(enum.IntEnum): + one = 1 + + +class ExampleString(enum.StrEnum): + source = "source" + + +@pytest_asyncio.fixture(scope="module", autouse=True) # type: ignore[misc] +async def concurrency_pool(app: object) -> None: + assert_pool_capacity_at_least(2) + + +class TestLock: + @pytest.mark.parametrize( + ("argument", "expected"), + [ + ("source", "source"), + (ExampleString.source, "source"), + (1, "1"), + (-10, "-10"), + (ExampleInt.one, "1"), + (uuid.UUID("74d7d6d5-24bc-4d90-bc84-45b5f0146b21"), "74d7d6d5-24bc-4d90-bc84-45b5f0146b21"), + (True, "true"), + (False, "false"), + ], + ) + def test_canonicalize_argument__supported_value(self, argument: object, expected: str) -> None: + assert Lock._canonicalize_argument(argument) == expected + + @pytest.mark.parametrize( + "argument", + ["", "has space", "has|pipe", "кириллица", 1.5, b"bytes", ["list"]], + ids=["empty", "space", "pipe", "unicode", "float", "bytes", "list"], + ) + def test_canonicalize_argument__invalid_value(self, argument: object) -> None: + with pytest.raises(errors.InvalidLockKey): + Lock._canonicalize_argument(argument) + + def test_build_identity__joins_argument_boundaries(self) -> None: + assert Lock._build_identity("test_lock", ("first", 2, True)) == ("test_lock", "first|2|true") + + def test_build_identity__integer_and_string_share_identity(self) -> None: + assert Lock._build_identity("test_lock", (1,)) == Lock._build_identity("test_lock", ("1",)) + + def test_build_identity__allows_no_arguments(self) -> None: + assert Lock._build_identity("test_lock", ()) == ("test_lock", "") + + @pytest.mark.parametrize( + "lock_kind", + ["", "UPPER_CASE", "has-hyphen", "double__underscore", "trailing_", cast(str, 1)], + ) + def test_build_identity__invalid_kind(self, lock_kind: str) -> None: + with pytest.raises(errors.InvalidLockKey): + Lock._build_identity(lock_kind, ()) + + def test_build_identity__kind_size_boundaries(self) -> None: + assert Lock._build_identity("a" * 128, ()) == ("a" * 128, "") + + with pytest.raises(errors.InvalidLockKey): + Lock._build_identity("a" * 129, ()) + + def test_build_identity__key_size_boundaries(self) -> None: + assert Lock._build_identity("test_lock", ("a" * 1024,)) == ("test_lock", "a" * 1024) + + with pytest.raises(errors.InvalidLockKey): + Lock._build_identity("test_lock", ("a" * 1025,)) + + def test_build_identity__oversized_integer(self) -> None: + with pytest.raises(errors.InvalidLockKey): + Lock._build_identity("test_lock", (10**5000,)) + + @pytest.mark.asyncio + async def test_aenter__returns_lock_and_inserts_canonical_identity(self) -> None: + lock_kind = new_lock_kind() + lock_key = "source|1|true" + + async with TableSizeNotChanged("lk_locks"): + async with transaction() as transaction_execute: + lock = Lock(transaction_execute, lock_kind, "source", 1, True) + + async with lock as acquired_lock: + assert acquired_lock is lock + assert await load_acquisition_rows(transaction_execute, lock_kind) == [ + {"lock_kind": lock_kind, "lock_key": lock_key} + ] + + assert await load_acquisition_rows(transaction_execute, lock_kind) == [] + + @pytest.mark.asyncio + async def test_aenter__reentrant_identity_raises_invariant_violation(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.LockInvariantViolation) as exception_info: + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "one"): + async with Lock(transaction_execute, lock_kind, "one"): + pass + + unique_violation = cast(type[Exception], UniqueViolation) + assert isinstance(exception_info.value.__cause__, unique_violation) + + @pytest.mark.asyncio + async def test_aenter__invalid_identity_does_not_access_database(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.InvalidLockKey): + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "invalid value"): + pass + + @pytest.mark.asyncio + async def test_aexit__propagates_body_exception_and_removes_row(self) -> None: + class ProtectedOperationError(Exception): + pass + + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(ProtectedOperationError): + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "one"): + raise ProtectedOperationError() + + assert await load_acquisition_rows(execute, lock_kind) == [] + + @pytest.mark.asyncio + async def test_aexit__cleanup_failure_preserves_body_exception(self) -> None: + class ProtectedOperationError(Exception): + pass + + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(ProtectedOperationError) as exception_info: + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "one"): + await transaction_execute( + """ + DELETE FROM lk_locks + WHERE lock_kind = %(lock_kind)s + """, + {"lock_kind": lock_kind}, # type: ignore[misc] + ) + raise ProtectedOperationError() + + assert any(note.startswith("Lock cleanup failed:") for note in exception_info.value.__notes__) + + @pytest.mark.asyncio + async def test_aexit__cleanup_failure_propagates_without_body_exception(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.LockInvariantViolation): + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "one"): + await transaction_execute( + """ + DELETE FROM lk_locks + WHERE lock_kind = %(lock_kind)s + """, + {"lock_kind": lock_kind}, # type: ignore[misc] + ) + + @pytest.mark.asyncio + async def test_aexit__before_acquisition_raises(self, mocker: MockerFixture) -> None: + lock = Lock(cast(ExecuteType, mocker.AsyncMock()), new_lock_kind()) + + with pytest.raises(RuntimeError, match="exited before acquisition"): + await lock.__aexit__(None, None, None) + + @pytest.mark.asyncio + async def test_aexit__mutex_remains_held_until_transaction_finishes(self) -> None: + lock_kind = new_lock_kind() + holder_context_exited = asyncio.Event() + finish_holder = asyncio.Event() + waiter_attempting = asyncio.Event() + waiter_entered = asyncio.Event() + + async def hold_mutex() -> None: + async with transaction() as transaction_execute: + async with Lock(transaction_execute, lock_kind, "one"): + pass + + holder_context_exited.set() + await finish_holder.wait() + + async def wait_for_mutex() -> None: + await holder_context_exited.wait() + waiter_attempting.set() + + async with locked_transaction(lock_kind, "one"): + waiter_entered.set() + + async with TableSizeNotChanged("lk_locks"): + holder_task = asyncio.create_task(hold_mutex()) + await holder_context_exited.wait() + waiter_task = asyncio.create_task(wait_for_mutex()) + await waiter_attempting.wait() + + try: + with pytest.raises(TimeoutError): + async with asyncio.timeout(0.05): + await waiter_entered.wait() + finally: + finish_holder.set() + await asyncio.gather(holder_task, waiter_task) + + assert waiter_entered.is_set() + + +class TestLockedTransaction: + @pytest.mark.asyncio + async def test_invalid_identity_does_not_open_transaction(self, mocker: MockerFixture) -> None: + transaction_mock = mocker.patch.object(locks_domain, "transaction") + + with pytest.raises(errors.InvalidLockKey): + async with locked_transaction(LockKind("invalid-kind")): + pass + + transaction_mock.assert_not_called() + + @pytest.mark.asyncio + async def test_returns_owned_transaction_execute(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + async with locked_transaction(lock_kind, "one") as transaction_execute: + assert ( + await transaction_execute( + """ + SELECT 1 AS value + """ + ) + == [{"value": 1}] + ) + + @pytest.mark.asyncio + async def test_transaction_failure_propagates(self, mocker: MockerFixture) -> None: + class TransactionError(Exception): + pass + + class FailingTransaction: + async def __aenter__(self) -> ExecuteType: + raise TransactionError() + + async def __aexit__( + self, + exception_type: type[BaseException] | None, + exception: BaseException | None, + traceback: object | None, + ) -> None: + return None + + mocker.patch.object(locks_domain, "transaction", return_value=FailingTransaction()) + + with pytest.raises(TransactionError): + async with locked_transaction(new_lock_kind(), "one"): + pass + + @pytest.mark.asyncio + async def test_lock_failure_closes_transaction(self, mocker: MockerFixture) -> None: + class AcquisitionError(Exception): + pass + + mocker.patch.object(locks_domain.operations, "acquire", side_effect=AcquisitionError) + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(AcquisitionError): + async with locked_transaction(new_lock_kind(), "one"): + pass + + @pytest.mark.asyncio + async def test_different_identity_does_not_wait(self) -> None: + first_lock_kind = new_lock_kind() + second_lock_kind = new_lock_kind() + holder_entered = asyncio.Event() + finish_holder = asyncio.Event() + + async def hold_first_mutex() -> None: + async with locked_transaction(first_lock_kind, "one"): + holder_entered.set() + await finish_holder.wait() + + async with TableSizeNotChanged("lk_locks"): + holder_task = asyncio.create_task(hold_first_mutex()) + await holder_entered.wait() + + try: + async with asyncio.timeout(1): + async with locked_transaction(second_lock_kind, "one"): + pass + finally: + finish_holder.set() + await holder_task + + @pytest.mark.asyncio + async def test_successful_transaction_allows_waiter_to_enter(self) -> None: + lock_kind = new_lock_kind() + holder_entered = asyncio.Event() + finish_holder = asyncio.Event() + waiter_attempting = asyncio.Event() + waiter_entered = asyncio.Event() + + async def hold_mutex() -> None: + async with locked_transaction(lock_kind, "one"): + holder_entered.set() + await finish_holder.wait() + + async def wait_for_mutex() -> None: + await holder_entered.wait() + waiter_attempting.set() + + async with locked_transaction(lock_kind, "one"): + waiter_entered.set() + + async with TableSizeNotChanged("lk_locks"): + holder_task = asyncio.create_task(hold_mutex()) + await holder_entered.wait() + waiter_task = asyncio.create_task(wait_for_mutex()) + await waiter_attempting.wait() + + try: + with pytest.raises(TimeoutError): + async with asyncio.timeout(0.05): + await waiter_entered.wait() + finally: + finish_holder.set() + await asyncio.gather(holder_task, waiter_task) + + assert waiter_entered.is_set() + + @pytest.mark.asyncio + async def test_failed_transaction_allows_waiter_to_enter(self) -> None: + class ProtectedOperationError(Exception): + pass + + lock_kind = new_lock_kind() + holder_entered = asyncio.Event() + fail_holder = asyncio.Event() + waiter_attempting = asyncio.Event() + waiter_entered = asyncio.Event() + + async def hold_mutex() -> None: + with pytest.raises(ProtectedOperationError): + async with locked_transaction(lock_kind, "one"): + holder_entered.set() + await fail_holder.wait() + raise ProtectedOperationError() + + async def wait_for_mutex() -> None: + await holder_entered.wait() + waiter_attempting.set() + + async with locked_transaction(lock_kind, "one"): + waiter_entered.set() + + async with TableSizeNotChanged("lk_locks"): + holder_task = asyncio.create_task(hold_mutex()) + await holder_entered.wait() + waiter_task = asyncio.create_task(wait_for_mutex()) + await waiter_attempting.wait() + + try: + with pytest.raises(TimeoutError): + async with asyncio.timeout(0.05): + await waiter_entered.wait() + finally: + fail_holder.set() + await asyncio.gather(holder_task, waiter_task) + + assert waiter_entered.is_set() + + @pytest.mark.asyncio + async def test_commits_protected_work(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeDelta("a_records", delta=1), TableSizeNotChanged("lk_locks"): + async with locked_transaction(lock_kind, "commit") as transaction_execute: + record_id = await audit_domain.record( + transaction_execute, + event=AuditEventName("lock_test_committed"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.system, + subject_id=SerializedId("locks"), + ) + + assert ( + await execute( + """ + SELECT id + FROM a_records + WHERE id = %(id)s + """, + {"id": record_id}, # type: ignore[misc] + ) + == [{"id": record_id}] + ) + + @pytest.mark.asyncio + async def test_rolls_back_protected_work(self) -> None: + class ProtectedOperationError(Exception): + pass + + lock_kind = new_lock_kind() + record_id = None + + async with TableSizeNotChanged("a_records"), TableSizeNotChanged("lk_locks"): + with pytest.raises(ProtectedOperationError): + async with locked_transaction(lock_kind, "rollback") as transaction_execute: + record_id = await audit_domain.record( + transaction_execute, + event=AuditEventName("lock_test_rolled_back"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.system, + subject_id=SerializedId("locks"), + ) + raise ProtectedOperationError() + + assert record_id is not None + assert ( + await execute( + """ + SELECT id + FROM a_records + WHERE id = %(id)s + """, + {"id": record_id}, # type: ignore[misc] + ) + == [] + ) + + @pytest.mark.asyncio + async def test_missing_acquisition_row_rolls_back(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("a_records"), TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.LockInvariantViolation): + async with locked_transaction(lock_kind, "missing") as transaction_execute: + await audit_domain.record( + transaction_execute, + event=AuditEventName("lock_test_missing_row"), + actor_kind=AuditEntityKind.system, + actor_id=SerializedId("system"), + subject_kind=AuditEntityKind.system, + subject_id=SerializedId("locks"), + ) + await transaction_execute( + """ + DELETE FROM lk_locks + WHERE lock_kind = %(lock_kind)s + """, + {"lock_kind": lock_kind}, # type: ignore[misc] + ) diff --git a/ffun/ffun/locks/tests/test_operations.py b/ffun/ffun/locks/tests/test_operations.py new file mode 100644 index 00000000..8fb5e1c3 --- /dev/null +++ b/ffun/ffun/locks/tests/test_operations.py @@ -0,0 +1,88 @@ +from typing import cast + +import pytest +from psycopg.errors import UniqueViolation + +from ffun.core.postgresql import execute, transaction +from ffun.core.tests.helpers import TableSizeDelta, TableSizeNotChanged +from ffun.locks import errors, operations +from ffun.locks.tests.helpers import count_acquisition_rows, load_acquisition_rows +from ffun.locks.tests.make import new_lock_kind + + +class TestAcquire: + @pytest.mark.asyncio + async def test_inserts_acquisition_row_until_transaction_rolls_back(self) -> None: + class RollbackTestTransaction(Exception): + pass + + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(RollbackTestTransaction): + async with transaction() as transaction_execute: + async with TableSizeDelta( + "lk_locks", + delta=1, + producer=lambda: count_acquisition_rows(transaction_execute), + ): + await operations.acquire(transaction_execute, lock_kind, "one") + + assert await load_acquisition_rows(transaction_execute, lock_kind) == [ + {"lock_kind": lock_kind, "lock_key": "one"} + ] + raise RollbackTestTransaction() + + assert await load_acquisition_rows(execute, lock_kind) == [] + + @pytest.mark.asyncio + async def test_committed_row_raises_invariant_violation(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeDelta("lk_locks", delta=1): + await execute( + """ + INSERT INTO lk_locks (lock_kind, lock_key) + VALUES (%(lock_kind)s, %(lock_key)s) + """, + {"lock_kind": lock_kind, "lock_key": "committed"}, # type: ignore[misc] + ) + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.LockInvariantViolation) as exception_info: + async with transaction() as transaction_execute: + await operations.acquire(transaction_execute, lock_kind, "committed") + + unique_violation = cast(type[Exception], UniqueViolation) + assert isinstance(exception_info.value.__cause__, unique_violation) + + +class TestRelease: + @pytest.mark.asyncio + async def test_deletes_acquisition_row(self) -> None: + lock_kind = new_lock_kind() + + async with transaction() as transaction_execute: + await operations.acquire(transaction_execute, lock_kind, "one") + + assert await load_acquisition_rows(transaction_execute, lock_kind) == [ + {"lock_kind": lock_kind, "lock_key": "one"} + ] + + async with TableSizeDelta( + "lk_locks", + delta=-1, + producer=lambda: count_acquisition_rows(transaction_execute), + ): + await operations.release(transaction_execute, lock_kind, "one") + + assert await load_acquisition_rows(transaction_execute, lock_kind) == [] + + @pytest.mark.asyncio + async def test_missing_row_raises_invariant_violation(self) -> None: + lock_kind = new_lock_kind() + + async with TableSizeNotChanged("lk_locks"): + with pytest.raises(errors.LockInvariantViolation): + async with transaction() as transaction_execute: + await operations.release(transaction_execute, lock_kind, "missing") diff --git a/ffun/ffun/queues/domain.py b/ffun/ffun/queues/domain.py index 54838e28..6d28d193 100644 --- a/ffun/ffun/queues/domain.py +++ b/ffun/ffun/queues/domain.py @@ -4,3 +4,5 @@ pull = operations.pull acknowledge = operations.acknowledge queues_stats = operations.queues_stats +tech_get_queue_records = operations.tech_get_queue_records +tech_clear_queue = operations.tech_clear_queue diff --git a/ffun/ffun/queues/tests/test_domain.py b/ffun/ffun/queues/tests/test_domain.py index 914f83ef..d4d7fe94 100644 --- a/ffun/ffun/queues/tests/test_domain.py +++ b/ffun/ffun/queues/tests/test_domain.py @@ -64,3 +64,13 @@ async def test_queues_stats(self) -> None: await operations.push(QueueKind.test_queue_1, [make.fake_queue_item(), make.fake_queue_item()]) assert await domain.queues_stats() == {(QueueKind.test_queue_1.value, 1): 2} + + +class TestTechGetQueueRecords: + def test_reexports_operation(self) -> None: + assert domain.tech_get_queue_records is operations.tech_get_queue_records + + +class TestTechClearQueue: + def test_reexports_operation(self) -> None: + assert domain.tech_clear_queue is operations.tech_clear_queue diff --git a/ffun/poetry.lock b/ffun/poetry.lock index b9d77166..6eb29744 100644 --- a/ffun/poetry.lock +++ b/ffun/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.3.2 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.4.1 and should not be changed by hand. [[package]] name = "annotated-types" @@ -860,6 +860,21 @@ files = [ {file = "eradicate-2.3.0.tar.gz", hash = "sha256:06df115be3b87d0fc1c483db22a2ebb12bcf40585722810d809cc770f5031c37"}, ] +[[package]] +name = "exceptiongroup" +version = "1.3.1" +description = "Backport of PEP 654 (exception groups)" +optional = false +python-versions = ">=3.7" +groups = ["dev"] +files = [ + {file = "exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598"}, + {file = "exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219"}, +] + +[package.extras] +test = ["pytest (>=6)"] + [[package]] name = "fastapi" version = "0.115.13" @@ -2672,6 +2687,22 @@ files = [ [package.extras] windows-terminal = ["colorama (>=0.4.6)"] +[[package]] +name = "pyleak" +version = "0.2.0" +description = "Detect leaked asyncio tasks, threads, and event loop blocking in Python. Inspired by Go's goleak" +optional = false +python-versions = ">=3.9" +groups = ["dev"] +files = [ + {file = "pyleak-0.2.0-py3-none-any.whl", hash = "sha256:87534b46abf95eefcf539a31a5a31cb378484ebdfd32600965a9be2aeac3eb95"}, + {file = "pyleak-0.2.0.tar.gz", hash = "sha256:a0e8b94d66963358fdb370bade4caac0ce922f173dd89c853e26870b1f7ef520"}, +] + +[package.dependencies] +exceptiongroup = ">=1.3.0" +sniffio = ">=1.3.0" + [[package]] name = "pyparsing" version = "3.3.2" @@ -4559,4 +4590,4 @@ cffi = ["cffi (>=1.11)"] [metadata] lock-version = "2.1" python-versions = ">=3.13,<3.14" -content-hash = "ff2fe5e5ae0b5858f57918d1fbe5d243aa69b130ba3ae0def43d88305c3b7c67" +content-hash = "a6d2d2a16d6d20dab027d76ad8a2c998166652578980e898ffc0aec32a3b0fd8" diff --git a/ffun/pyproject.toml b/ffun/pyproject.toml index 513328af..cf136f06 100644 --- a/ffun/pyproject.toml +++ b/ffun/pyproject.toml @@ -103,6 +103,7 @@ changy = "0.*" pytest = "8.4.*" pytest-asyncio = "1.0.*" pytest-mock = "3.14.*" +pyleak = "0.2.*" py-spy = "0.4.*" @@ -210,9 +211,10 @@ module = [ disallow_any_expr = false [tool.pytest.ini_options] -# possibly it is not good option value, because we'll lose detecting of lost coroutines -# but for now it is required for some session-scoped fixtures +# Keep session-scoped fixtures and async tests in the same event loop so the PostgreSQL pool +# and other session-lived async entities remain fully accessible from tests. asyncio_default_fixture_loop_scope = "session" +asyncio_default_test_loop_scope = "session" addopts = """-x -vv --strict-markers -p no:cacheprovider""" diff --git a/ffun/tach.toml b/ffun/tach.toml index 0978569f..b81c6e4b 100644 --- a/ffun/tach.toml +++ b/ffun/tach.toml @@ -27,6 +27,7 @@ layers = [ "tags", "feeds", "users", + "shared_services", "foundation", ] @@ -300,7 +301,19 @@ layer = "users" path = "ffun.queues" depends_on = [ ] -layer = "foundation" +layer = "shared_services" + +[[modules]] +path = "ffun.audit" +depends_on = [ +] +layer = "shared_services" + +[[modules]] +path = "ffun.locks" +depends_on = [ +] +layer = "shared_services" [[modules]] path = "ffun.core" diff --git a/specs/backend_architecture/db.md b/specs/backend_architecture/db.md index a8fb7cee..94feae27 100644 --- a/specs/backend_architecture/db.md +++ b/specs/backend_architecture/db.md @@ -6,7 +6,7 @@ This document describes how Feeds Fun backend code works with PostgreSQL, includ ## Scope -This specification covers backend database access from Python code under `ffun/ffun`, database schema migrations owned by backend modules, and tests that verify persistence-backed behavior. +This specification covers backend database schema requirements, backend database access from Python code under `ffun/ffun`, database schema migrations owned by backend modules, and tests that verify persistence-backed behavior. This specification does not cover deployment-specific PostgreSQL administration, production backup strategy, frontend data access, Docker configuration, or database schema details for individual product features. @@ -116,7 +116,9 @@ Operations that need one consistent application-level timestamp across multiple ## Idempotency And Constraints -Schema constraints SHOULD enforce durable invariants such as primary keys, uniqueness, and ownership keys. +Schema constraints MUST be limited to structural storage integrity, such as nullability, primary keys, foreign keys, uniqueness, and ownership keys. + +Business invariants, including allowed values, cross-column value combinations, and state-transition rules, MUST be validated by domain or service logic before an operation persists the state. Database schemas MUST NOT use `CHECK` constraints or other schema-level validation to enforce business invariants. Database operations SHOULD use `ON CONFLICT` when repeated calls are expected to be harmless. @@ -148,6 +150,8 @@ When a concurrency failure means the caller's requested state cannot be guarante Each backend module SHOULD own the tables that correspond to its domain responsibility. +A table owned by one top-level backend module MUST NOT define foreign keys, cascading actions, triggers, or other schema-level references to tables owned by another top-level backend module. Cross-module identifiers MUST be stored as semantic values without database-level references; validation and coordinated state changes MUST go through module domain boundaries. + Table names SHOULD use a short module-related prefix when that keeps ownership clear in SQL and migrations. Schema changes MUST be implemented as yoyo migrations in the owning module's `migrations` package. @@ -168,12 +172,37 @@ When yoyo fails in the development environment because the PostgreSQL database i Migrations SHOULD keep schema DDL, indexes, and data backfills in the same migration only when they are part of one atomic compatibility step. +Migration apply and rollback functions SHOULD execute each SQL statement separately. Related statements MAY remain in the same migration step and transaction when they form one atomic compatibility change. Separate execution improves failure attribution and avoids relying on multi-statement driver behavior. + Large data migrations SHOULD be written so their locking, ordering, and rollback properties are clear from the SQL and local comments. Indexes SHOULD be created or changed in the migration that introduces the query shape or invariant that needs them. Migration SQL SHOULD use the same safety rules as runtime SQL: bind data values as parameters and keep interpolated SQL fragments limited to trusted static identifiers or fragments. +## Schema Data Types + +Schema specifications and new migrations SHOULD use SQL-standard or widely supported data type names when PostgreSQL accepts them with the required semantics. This spelling convention is intended to make schemas clearer and easier to adapt; PostgreSQL remains the backend storage technology. + +In particular, schemas SHOULD use: + +- `SMALLINT`, `INTEGER`, and `BIGINT` instead of PostgreSQL aliases such as `INT2`, `INT4`, and `INT8`. +- `REAL` and `DOUBLE PRECISION` instead of PostgreSQL aliases such as `FLOAT4` and `FLOAT8`. +- `BOOLEAN` instead of the PostgreSQL alias `BOOL`. +- `TIMESTAMP WITH TIME ZONE` instead of the PostgreSQL alias `TIMESTAMPTZ`. +- `TIMESTAMP WITHOUT TIME ZONE` when a timestamp intentionally has no time zone, rather than relying on the shorter `TIMESTAMP` spelling. +- SQL-standard identity syntax, such as `BIGINT GENERATED BY DEFAULT AS IDENTITY`, instead of PostgreSQL pseudo-types such as `SERIAL` and `BIGSERIAL` when the database must generate integer identifiers. + +Closed categorical values SHOULD be represented by stable integer identifiers and stored with `SMALLINT`, `INTEGER`, or `BIGINT`, according to the required range, rather than with string labels. Numeric identifiers decouple persisted identity from human-readable names, avoid data migrations when names change, and generally require less storage and index space. The owning specification MUST define the identifier mapping, and assigned identifiers MUST NOT be changed or reused. + +Python code SHOULD expose these categorical identifiers as enums with the stable integer identifiers as their member values. Enum member names SHOULD provide readable code and interface vocabulary, while enum integer values SHOULD be used for persistence and internal identifiers. + +A schema MAY store categorical strings only when its owning specification explicitly requires string identity. Appropriate reasons include an open-ended value set, compatibility with an external protocol, or a requirement to preserve externally supplied values verbatim. + +PostgreSQL-specific types, operators, and clauses MAY be used when their behavior is intentional and no portable spelling provides the required semantics. Examples include `JSONB`, array and range types, `ON CONFLICT`, and `FOR UPDATE SKIP LOCKED`. + +Existing schemas and historical migrations SHOULD NOT be changed only to replace an equivalent PostgreSQL-specific spelling. A module specification and the migration that introduces its schema SHOULD use the same data type spellings. + ## Schema Naming Database object names SHOULD use lowercase `snake_case`. @@ -194,6 +223,14 @@ Reference columns SHOULD use `_id`, such as `feed_id`, `entry_id`, `use Timestamp columns SHOULD use the `_at` suffix for points in time, such as `created_at`, `updated_at`, or `loaded_at`. +Every newly introduced table SHOULD include `created_at` and `updated_at` columns unless its owning specification documents a concrete semantic reason to omit one or both. The absence of a current consumer is not sufficient justification for an omission. + +Both columns SHOULD use `TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP`. `created_at` MUST identify when the row was first persisted and MUST remain unchanged for the row's lifetime. Every operation that changes persisted row state MUST set `updated_at` to the database's current timestamp in the same statement. + +Immutable append-only tables MAY omit `updated_at` because their rows never change. Derived tables MAY omit either timestamp only when row creation or mutation has no stable meaning; the owning module specification MUST explain that lifecycle explicitly. + +Existing tables and historical migrations SHOULD NOT be changed only to add these columns. The convention applies when introducing a table or making a substantive lifecycle change to an existing table. + Deadline or availability timestamp columns SHOULD use a name that describes the state transition, such as `freezed_till`. Index names for column indexes SHOULD use `__idx`. diff --git a/specs/backend_architecture/entities.md b/specs/backend_architecture/entities.md index 4bf3d454..72c97989 100644 --- a/specs/backend_architecture/entities.md +++ b/specs/backend_architecture/entities.md @@ -24,6 +24,7 @@ The following topics are out of scope: - `value entity` - an entity whose equality is based on its data rather than object identity. - `boundary entity` - an entity that is passed between modules with different responsibilities. - `serialized representation` - a plain data representation prepared for an external protocol such as JSON, HTTP response data, logs, or persistent storage. +- `serialized id` - the canonical string representation of an entity identifier used by a shared boundary that accepts identifiers from multiple domain modules. ## General principles @@ -113,6 +114,8 @@ Shared semantic primitive types SHOULD belong to the domain module. Module-specific semantic primitive types SHOULD belong to the owning module. +`ffun.domain.entities.SerializedId` MUST represent the canonical string form of an entity identifier at shared boundaries that accept identifiers owned by different domain modules. It is a universal domain primitive rather than an identifier owned by any one subsystem. Code within an owning module SHOULD retain the identifier's more specific semantic type until it reaches such a boundary. + ## Entity ownership Shared entity infrastructure MUST belong to the core module. diff --git a/specs/backend_architecture/modules/audit.md b/specs/backend_architecture/modules/audit.md new file mode 100644 index 00000000..51884b93 --- /dev/null +++ b/specs/backend_architecture/modules/audit.md @@ -0,0 +1,169 @@ +# Audit module + +## Goal of the document + +This document describes how the `ffun.audit` backend module stores durable audit records and lets other modules append those records inside their own database transactions. + +## Scope + +This specification covers audit entity references, the append-only audit table, record creation, subject-based record loading, and the transactional interface exposed by `ffun.audit.domain`. + +General audit search, pagination, retention and archival policy, administrative presentation, business-event definitions owned by calling modules, and event sourcing are out of scope. + +## Dictionary + +- `actor entity` - the entity that caused or initiated the audited event. +- `subject entity` - the primary entity affected by or described by the audited event. +- `related entity` - an additional entity associated with the event but not acting as its actor or primary subject. +- `audit entity kind` - an integer enum value that identifies how an audit entity id must be interpreted. + +## Module responsibility + +`ffun.audit` MUST be a shared-service module that owns the common audit record entity, audit event-name type, audit entity kinds, append-only audit persistence, and subject-based record loading. + +The module MUST provide append-only persistence, subject-based record loading, and common audit types only. The calling module owns input validation, the decision to create an audit record, the event name, and the event-specific attributes. + +Audit records MUST be durable database state. Business events and ordinary logs MUST NOT be treated as substitutes for required audit records. + +Audit records describe committed business facts but MUST NOT be used as the source of truth for rebuilding application state. + +## Domain behavior + +### Entity references + +Actor and subject entities can belong to different domains and can use different native id types. Every audit record MUST contain exactly one actor and one subject. Each reference MUST contain both: + +- `kind` - the integer audit entity kind id. +- `id` - the entity's id in a stable string representation. + +`ffun.audit.entities` MUST define a dedicated `AuditEntityKind` based on `enum.IntEnum`. Its initial values MUST be: + +- `user = 1` - a regular application user. +- `admin = 2` - an administrator acting in an administrative capacity. +- `psp = 3` - a payment service provider. +- `system = 4` - an internal automated component. + +New kinds MUST be added explicitly to `AuditEntityKind` before they are used. Numeric kind ids are stable database values and MUST NOT be changed or reused after records use them. + +Calling modules MUST normalize UUID, integer, and string ids to their canonical `ffun.domain.entities.SerializedId` string form before calling `ffun.audit.domain.record`. Calling modules MUST reject empty ids. + +The actor and subject MAY have the same kind or id. Their roles remain distinct. + +Actor and subject entity references MUST be stored in their dedicated columns. They MUST NOT be stored only inside record attributes. + +Additional related entities MAY be stored in `attributes`. Each related entity MUST be represented as an object with `kind` and `id` fields and MAY include a `role` field. The table MUST NOT add fixed columns for third or subsequent entity references. + +### Audit record + +An audit record MUST contain: + +- a unique audit record id. +- the database-generated creation time. +- a stable event name. +- the actor entity kind and id. +- the subject entity kind and id. +- event-specific structured attributes. + +Event names MUST be non-empty `snake_case` values. They SHOULD name the business change rather than the function or HTTP endpoint that produced it. The module that owns an event MUST keep its meaning and attributes stable. + +`ffun.audit.entities` MUST define `AuditEventName` as the semantic string type used for audit event names. Calling modules retain ownership of each event's meaning and MUST pass event names typed as `AuditEventName` to `ffun.audit.domain.record`. + +Attributes MUST be a JSON object. They SHOULD contain only data needed to understand the audited change, such as previous and new values, provider references, or related entities. The actor and subject entity pairs remain the primary entity references. + +### Append-only behavior + +Normal runtime behavior MUST only insert audit records. `ffun.audit` MUST NOT expose operations that update or delete individual records. + +An existing audit record MUST NOT be changed to correct or reinterpret it. A correction MUST be represented by a new audit record whose event and attributes explain the correction according to the calling module's contract. + +Audit record creation MUST NOT use upsert or conflict handling that can replace an existing record. + +Retention or legally required deletion, if introduced, MUST be an explicit administrative policy outside normal record creation and is not defined by this specification. + +### Subject-based record loading + +The module MUST allow callers to load audit records for one exact subject entity kind and id pair. The query MUST return every matching record ordered by `created_at` and then `id`, both ascending, and MUST return an empty list when no records match. + +Loading records MUST be read-only and MUST NOT generate audit records, business events, or ordinary business logs. + +## Database schema + +### `a_records` + +The module MUST own exactly one table, `a_records`. + +```sql +-- Append-only durable records of audited business changes and events. +CREATE TABLE a_records ( + id UUID PRIMARY KEY, -- Unique audit record id generated by Python before insertion. + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Time at which PostgreSQL inserts the record. + event TEXT NOT NULL, -- Stable event name owned by the calling module. + actor_kind SMALLINT NOT NULL, -- AuditEntityKind id of the entity that initiated the event. + actor_id TEXT NOT NULL, -- Canonical string id of the actor entity. + subject_kind SMALLINT NOT NULL, -- AuditEntityKind id of the primary entity affected by the event. + subject_id TEXT NOT NULL, -- Canonical string id of the subject entity. + attributes JSONB NOT NULL -- Event-specific structured data supplied by Python. +); + +CREATE INDEX a_records_subject_kind_subject_id_created_at_id_idx ON a_records (subject_kind, subject_id, created_at, id); -- Supports deterministic loading of one subject's audit history. +``` + +The `actor_kind` and `subject_kind` columns MUST use `SMALLINT`. Calling modules MUST pass values typed as `AuditEntityKind`; the database MUST NOT constrain the set of allowed kind ids. + +Calling modules MUST validate the structure of `attributes`; the database MUST NOT constrain its JSON shape. + +The audit record id MUST be generated in Python and supplied explicitly to the insert. The database MUST NOT define a default for `id`. + +`a_records` intentionally omits `updated_at`. Audit records are immutable and append-only, so `created_at` completely describes their persistence lifecycle and no row update time exists. + +Entity ids MUST NOT use foreign keys because their referenced tables and native types depend on the corresponding kind. + +No additional audit tables, materialized current-state tables, or generic entity registry are permitted by this specification. + +The subject index supports filtering by the complete subject identity and returning records in the required deterministic order. + +## Domain interface + +`ffun.audit.domain` MUST expose one record-creation operation named `record` and one subject query named `load_records_for_subject`. + +`ffun.audit.domain` MUST expose `new_audit_record_id`, which generates and returns a new `AuditRecordId` in the same style as the id factories in `ffun.domain.domain`. + +The operation MUST accept: + +- the caller's `ffun.core.postgresql.ExecuteType` as its first argument. +- keyword-only `event`, `actor_kind`, `actor_id`, `subject_kind`, and `subject_id` arguments, with the event typed as `AuditEventName`, kinds typed as `AuditEntityKind`, and ids typed as `SerializedId`. +- an optional keyword-only `attributes` mapping that defaults to an empty mapping. + +The interface SHOULD be used in this form: + +```python +audit_record_id = await audit.domain.record( + execute, + event=AuditEventName("source_entitlement_changed"), + actor_kind=AuditEntityKind.psp, + actor_id=SerializedId(str(psp_id)), + subject_kind=AuditEntityKind.user, + subject_id=SerializedId(str(user_id)), + attributes={"kind_id": kind_id, "granted": True, "value": 10}, +) +``` + +The calling module MUST pass already validated arguments. The operation MUST NOT normalize native ids or accept raw integer kind ids. It MUST generate the audit record id, insert exactly one row through the provided execute callable, and return the created audit record id. + +The operation MUST NOT open or commit a transaction. A calling module that requires an audit record for a state change MUST call `record` with the same execute callable used for that state change before committing its transaction. + +Failure to insert a required audit record MUST propagate to the caller and cause the caller-owned transaction to roll back. A rolled-back transaction MUST NOT leave an audit record. + +Calling modules SHOULD create audit records only for state changes that actually occurred. Idempotent no-op requests SHOULD NOT create duplicate audit records unless the calling module explicitly defines the request itself as an auditable event. + +`load_records_for_subject` MUST accept the caller's `ffun.core.postgresql.ExecuteType` as its first argument and keyword-only `subject_kind` and `subject_id` arguments typed as `AuditEntityKind` and `SerializedId`. It MUST return a list of validated `AuditRecord` entities in the order defined by the subject-based record loading behavior. + +`load_records_for_subject` MUST use the provided execute callable and MUST NOT open or commit a transaction. Callers MAY pass a transaction-bound execute callable when records written within that transaction must be visible to the query. + +## Audit records + +Module does not define concrete audit events because it provides generic audit storage. + +## Business events + +Audit record creation MUST NOT generate a business event. The calling module owns any business event corresponding to the audited change. `ffun.audit` MAY generate technical logs or metrics, but MUST NOT classify them as business events. diff --git a/specs/backend_architecture/modules/entitlements.md b/specs/backend_architecture/modules/entitlements.md new file mode 100644 index 00000000..23012b82 --- /dev/null +++ b/specs/backend_architecture/modules/entitlements.md @@ -0,0 +1,191 @@ +# Entitlements module + +## Goal of the document + +This document defines source-owned entitlement state, materialized effective entitlement intervals, and entitlement change records owned by `ffun.entitlements`. + +## Scope + +This specification covers the entitlement kind registry, time-bounded source state, effective interval materialization and cleanup, merge behavior, transactions, audit records, and business events. + +Purchased subscription lifecycle, payment service provider protocols, product pricing, frontend behavior, and the concrete set of entitlement sources are out of scope. + +## Dictionary + +- `entitlement kind` - a predefined capability or limit whose registry assigns the policy used to merge values for that kind. +- `source` - a semantic identifier for one system allowed to maintain its own current entitlement state, such as a payment service provider or support tooling. +- `source entitlement` - the latest entitlement state written by one source for one user and entitlement kind. +- `active source entitlement` - a granted source entitlement whose activation time is less than or equal to, and whose expiration time is later than, the time at which effective entitlements are evaluated. +- `effective entitlement interval` - a materialized time interval during which the merged source state grants one entitlement kind to one user with one value. +- `merge policy` - the operation used to combine integer values from multiple granted source entitlements of the same kind. + +## Module responsibility + +`ffun.entitlements` MUST be a domain-level module that owns entitlement entities, persistence, merging, timeline materialization, and queries. Sources MUST change entitlements through its domain boundary; they MUST NOT write entitlement tables directly or change another source's state. + +Callers that check a user's current entitlements MUST read effective entitlements through the module boundary. They SHOULD NOT reproduce merge behavior or derive access directly from the source entitlement table. + +## Domain behavior + +### Entitlement kinds + +`ffun.entitlements.entities` MUST define `EntitlementKindId` as an integer enum with exactly these members and stable values: + +- `day_tokens = 1`. +- `month_tokens = 2`. + +The module MUST maintain an immutable code-owned collection containing exactly one `EntitlementKind` for every `EntitlementKindId` member. Each entry MUST pair the id with its merge policy. This collection is the entitlement kind registry and source of truth for kind metadata; entitlement kinds MUST NOT be stored in a database registry table or runtime settings. + +The registry MUST define these entitlement kinds: + +- `day_tokens` with merge policy `max`. +- `month_tokens` with merge policy `max`. + +Merge policies MUST be a closed set of named values. The supported policies are: + +- `max` - the effective value is the largest candidate value. +- `min` - the effective value is the smallest candidate value. +- `sum` - the effective value is the sum of all candidate values. + +The Python representations of entitlement kind ids and merge policies MUST use their corresponding enums. + +### Source entitlement state + +All sources MUST store their latest state in `en_source_entitlements`, with at most one row per `(user_id, kind_id, source)`. Updating that row replaces the source's previous state. Source ids MUST use a semantic Python type. + +Every source entitlement state MUST have finite activation and expiration timestamps, and its activation timestamp MUST be earlier than its expiration timestamp. A state is inactive before its activation timestamp and at or after its expiration timestamp. An already expired state MAY be stored when it is the latest successfully processed change from its source, but it is immediately inactive. + +A future-dated grant MUST contribute from `starts_at`. Because each source has one row, storing it immediately replaces and deactivates that source's previous contribution. + +A revoked row MUST remain representable as a source's latest state. It removes only that source's contribution; durable history is stored through `ffun.audit`. + +Repeated requests whose `granted`, `value`, `starts_at`, and `expires_at` values already match the stored source state MUST be treated as no-ops. + +### Effective entitlement timeline materialization + +`en_entitlements` MUST store non-overlapping merged grant intervals derived from `en_source_entitlements`. Multiple rows MAY exist for one `(user_id, kind_id)` pair. Absence of a covering interval means the entitlement is not granted. + +Intervals are half-open: `starts_at` is inclusive and `expires_at` is exclusive. Each timeline rebuild MUST capture one evaluation time and use it consistently. For one `(user_id, kind_id)` pair, the module MUST build the timeline as follows: + +1. Load the granted source rows and order their distinct `starts_at` and `expires_at` boundaries. +2. For each interval between consecutive boundaries whose end is later than the evaluation time, select the source rows active throughout it and skip the interval when none are active. +3. Apply the kind's merge policy to the active values and materialize the result. +4. Coalesce adjacent intervals with the same value. + +For source-change domain results and business event payloads, the effective state at an evaluation time is `(true, value)` when a materialized interval covers that time and `(false, null)` otherwise. + +### Effective entitlement queries and cleanup + +Every effective entitlement query MUST capture one evaluation time and select intervals satisfying `starts_at <= evaluation_time < expires_at`. At most one row can match each user-kind pair. Queries MUST NOT mutate entitlement state when time passes. + +The module MUST provide a cleanup method that captures one cleanup time and deletes effective rows with `expires_at <= cleanup_time`. It MUST NOT delete source rows and MAY run opportunistically because expired effective rows do not affect query correctness. + +### Change workflow + +Every source change MUST update its source state and rebuild the affected timeline in one transaction. Changes for the same `(user_id, kind_id)` pair MUST be serialized. + +The workflow MUST: + +1. validate that the entitlement kind is present in the entitlement kind registry, a granted state has an integer value, a revoked state has no value, `starts_at` and `expires_at` are finite timestamps, and `starts_at` is earlier than `expires_at`. +2. capture one evaluation time and load the previous effective intervals ending after that time. +3. store the new source state and derive a timeline from all current source rows for the user and kind. +4. replace all effective rows for that user and kind with the derived intervals. +5. determine the new effective state at the captured time and append the required audit record, including the previous and new effective interval lists, through `ffun.audit.domain` with the same transaction execute callable. +6. commit the source state, effective intervals, and audit record atomically. + +A failed workflow MUST leave the source state, effective timeline, and audit history unchanged. + +## Database schema + +The module MUST own exactly two tables, `en_source_entitlements` for source-owned current state and `en_entitlements` for the merged effective timeline. Both tables use the `en_` prefix. + +### `en_source_entitlements` + +```sql +-- Stores the latest entitlement state supplied by every source. +CREATE TABLE en_source_entitlements ( + source_id TEXT NOT NULL, -- Semantic id of the source that owns this state. + user_id UUID NOT NULL, -- Semantic id of the user whose source state is stored. + kind_id SMALLINT NOT NULL, -- Stable integer EntitlementKindId value configured for this source state. + granted BOOLEAN NOT NULL, -- Whether the source grants the entitlement during this state's activation interval. + value BIGINT, -- Integer grant value; null only for a revoked state. + starts_at TIMESTAMP WITH TIME ZONE NOT NULL, -- Time at which this source state becomes active. + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, -- Time at or after which this source state is inactive. + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Time at which this source first received a state row for the user and kind. + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Time at which the source state row was last replaced. + + PRIMARY KEY (user_id, kind_id, source_id) -- Keeps only the current state from each source for a user and kind. +); +``` + +### `en_entitlements` + +```sql +-- Materialized effective entitlement intervals derived from the source entitlement table. +CREATE TABLE en_entitlements ( + user_id UUID NOT NULL, -- Semantic id of the user who has the effective entitlement. + kind_id SMALLINT NOT NULL, -- Stable integer EntitlementKindId value configured for this interval. + value BIGINT NOT NULL, -- Value produced by the kind's merge policy for this interval. + starts_at TIMESTAMP WITH TIME ZONE NOT NULL, -- Inclusive start of the effective granted interval. + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, -- Exclusive end of the effective granted interval. + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Time at which this materialized interval row was created. + updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Time at which this materialized interval row was last changed. + + PRIMARY KEY (user_id, kind_id, starts_at) -- Identifies each interval in the effective timeline. +); + +CREATE INDEX en_entitlements_expires_at_idx ON en_entitlements (expires_at); -- Supports removal of expired intervals. +``` + +`en_source_entitlements` remains the source of truth. Replacing source state MUST preserve `created_at` and set `updated_at` to the database's current timestamp. Domain logic MUST ensure effective intervals are finite, satisfy `starts_at < expires_at`, and do not overlap for the same user and kind. `kind_id` MUST NOT have a foreign key because kind ids and their merge policies are defined by the code-owned registry. The effective primary key supports user-kind queries, and the expiration index supports cleanup. + +## Domain interface + +`ffun.entitlements.domain` MUST provide source changes, interval cleanup, and a batch effective-entitlement listing function. Operation names are not specified. + +The batch function MUST accept lists of user ids and entitlement kind ids, use one evaluation time for the whole request, and return `Mapping[UserId, Mapping[EntitlementKindId, EffectiveEntitlementInterval | None]]`. An empty entitlement kind id list MUST select every registered entitlement kind. Every requested user and selected kind MUST be present in the result. A value MUST be the complete effective interval that covers the evaluation time when the entitlement is granted, and `None` otherwise. + +## Audit records + +### `source_entitlement_changed` + +Every non-no-op source change MUST append one `source_entitlement_changed` audit record in the same transaction. Its actor MUST identify the initiating user, administrator, payment service provider, or system component; its subject MUST use kind `user` and the affected user id. + +The record attributes MUST include: + +- `source` - semantic id of the source that owns the changed row. +- `kind_id` - entitlement kind id. +- `previous_source_state` and `new_source_state` - the complete JSON-mode Pydantic serialization of the corresponding source entitlement entity; the previous state is `null` when no row existed. +- `previous_effective_intervals` and `new_effective_intervals` - lists containing the complete JSON-mode Pydantic serialization of each corresponding effective entitlement interval whose expiration is later than the workflow's evaluation time, ordered by `starts_at`; an empty list means there are no current or future effective intervals. + +The audit record MUST be appended even when the effective entitlement is unchanged, because the source-owned state changed. A no-op request MUST NOT append an audit record. + +## Business events + +The module MUST generate business events only after a successful source change transaction. No-op and rolled-back transactions MUST NOT generate them. Event payloads MUST describe only the resulting state and MUST NOT contain previous or historical values. + +### `source_entitlement_changed` + +This event MUST be generated whenever a source row changes, even when the effective entitlement does not. + +The event MUST use the affected user as the business event user and include: + +- `source` - semantic source id. +- `kind_id` - entitlement kind id. +- `granted` - whether the source grants the entitlement during the new state's activation interval. +- `value` - new integer value, or `null` for a revoked state. +- `starts_at` - activation time of the new source state. +- `expires_at` - expiration time of the new source state. + +### `entitlement_changed` + +This event MUST be generated after every successful non-no-op source change and timeline rebuild, whether or not the effective timeline changed. + +The event MUST use the affected user as the business event user and include: + +- `kind_id` - entitlement kind id. +- `granted` - whether the entitlement is granted after the timeline rebuild. +- `value` - new effective integer value, or `null` when the effective entitlement was revoked. +- `new_effective_intervals` - the resulting effective intervals represented by their `value`, `starts_at`, and `expires_at` fields. + +Every successful non-no-op source change MUST generate both events. Time passage, queries, and cleanup MUST NOT append entitlement audit records or generate business events. diff --git a/specs/backend_architecture/modules/locks.md b/specs/backend_architecture/modules/locks.md new file mode 100644 index 00000000..9cae4ba5 --- /dev/null +++ b/specs/backend_architecture/modules/locks.md @@ -0,0 +1,236 @@ +# Locks module + +## Goal of the document + +This document describes how the `ffun.locks` backend module provides collision-free, transaction-scoped logical mutexes through PostgreSQL. + +## Scope + +This specification covers lock identity, lock acquisition and release, transaction ownership, the ephemeral lock table, failure behavior, and the public asynchronous context managers exposed by `ffun.locks.domain`. + +Distributed coordination outside PostgreSQL, reader-writer locks, semaphores, long-lived leases, leader election, work queues, and administrative PostgreSQL maintenance are out of scope. + +## Dictionary + +- `logical mutex` - exclusive coordination identified by an application-defined kind and ordered arguments rather than by an existing business row. +- `lock kind` - a stable, module-namespaced string that identifies the purpose and argument semantics of a logical mutex. +- `lock arguments` - ordered scalar values that identify one mutex within a lock kind. +- `canonical lock key` - the deterministic, delimiter-separated ASCII serialization of the lock arguments. +- `acquisition row` - the ephemeral `lk_locks` row whose unique primary key coordinates transactions requesting the same logical mutex. +- `holder transaction` - the explicit database transaction that inserted an acquisition row and has not yet committed or rolled back. + +## Module responsibility + +`ffun.locks` MUST be a shared-service module that owns collision-free logical mutex identity, key validation and encoding, acquisition-row persistence, and the public lock context managers. + +The module MUST provide technical transaction coordination only. Calling modules own the business meaning of each lock kind, the selection and ordering of lock arguments, the choice between caller-owned and lock-owned transaction boundaries, and all protected state changes. + +The module MUST use exact unique database values rather than hashes or PostgreSQL advisory locks. Different logical mutexes MUST NOT block one another because of a hash collision. + +The module MUST NOT create independent connection pools, reserve a second connection for a held lock, or coordinate through process-local state. + +## Domain behavior + +### Lock identity + +A logical mutex MUST be identified by the exact pair `(lock_kind, canonical_lock_key)`. + +Lock kinds MUST be non-empty lowercase `snake_case` strings. A calling module MUST namespace its kinds with enough context to prevent accidental reuse, for example `entitlements_user_kind`. A lock kind's meaning, argument count, argument order, and canonical conversions MUST remain stable after production code uses it. + +Lock kinds intentionally use string identity. They form an open extension namespace owned by calling modules rather than a closed categorical set, so they are an explicit exception to the integer categorical-value preference in `specs/backend_architecture/db.md`. + +`ffun.locks.entities` MUST define `LockKind` as the semantic string type used for lock kinds across module boundaries. Calling modules MUST construct lock kinds explicitly as `LockKind` values rather than passing unqualified strings. + +Lock arguments MUST be ordered. Argument order and canonical string value are part of lock identity; the original Python type is not. The following argument types MUST be supported: + +- `str`, including `enum.StrEnum` values and semantic string types created with `typing.NewType`, encoded unchanged. +- `int`, including semantic integer types created with `typing.NewType`, encoded in canonical base-10 form with a leading `-` for negative values and no redundant leading zeroes. +- `enum.IntEnum`, encoded as its integer value. +- `uuid.UUID`, encoded in its canonical lowercase hyphenated form. +- `bool`, encoded as lowercase `true` or `false`. + +Floating-point values, byte sequences, containers, arbitrary objects, and other unsupported values MUST be rejected before database access. Implementations MUST use the conversions defined above rather than applying generic `str(...)` conversion to arbitrary objects. + +Each converted argument MUST be non-empty and contain only ASCII letters, ASCII digits, `.`, `_`, `:`, `@`, `/`, or `-`. The pipe character `|` MUST separate adjacent converted arguments and MUST NOT be allowed inside an argument. The canonical lock key MUST be the converted arguments joined with `|`, without escaping, leading separators, or trailing separators. For example: + +```text +74d7d6d5-24bc-4d90-bc84-45b5f0146b21|1|source|true +``` + +A lock MAY have no arguments; its canonical key is then the empty string. Empty converted arguments MUST be rejected so a one-argument lock cannot collide with the no-argument lock. + +The delimiter and restricted alphabet make argument boundaries unambiguous without escaping or explicit type tags. Values from different Python types that have the same canonical string intentionally identify the same mutex. For example, integer `1` and string `"1"` both encode as `1`. + +Canonical serialization MUST be deterministic for the same supported values. It MUST NOT use Python object hashes, randomized representations, lossy normalization, or direct concatenation without separators. Encoding changes MUST preserve coordination between concurrently deployed application versions. + +The UTF-8 representation of a lock kind MUST NOT exceed 128 bytes. The UTF-8 representation of a canonical lock key MUST NOT exceed 1024 bytes. Oversized or otherwise invalid identity values MUST raise `ffun.locks.errors.InvalidLockKey` before database access. These conservative limits keep the composite primary key safely indexable and prevent unbounded caller-controlled lock records. + +### Acquisition and release + +Lock acquisition MUST insert one acquisition row with a plain `INSERT` through the holder transaction's execute callable. The insert MUST NOT use `ON CONFLICT`, an upsert, or a preliminary existence query. + +The immediate primary-key uniqueness check is the synchronization primitive. When another transaction has inserted the exact same primary key but has not completed, PostgreSQL waits for that transaction and then rechecks the conflict: + +- if the holder transaction commits after deleting its acquisition row, the waiting insert succeeds. +- if the holder transaction rolls back, the waiting insert succeeds. +- if a transaction commits a live acquisition row, the waiting insert fails with a uniqueness violation. + +The context manager MUST enter the protected body only after its insert succeeds. + +On normal context exit, the context manager MUST delete its acquisition row through the same execute callable. The delete MUST verify that exactly one row was deleted. + +On exceptional exit from a caller-owned lock context, the context manager SHOULD delete its acquisition row when the transaction remains usable. If the protected operation has already put the transaction into a failed state, rollback is the cleanup mechanism. Cleanup failure MUST NOT suppress the original protected-operation exception. + +On exceptional exit from a transaction-owning lock context, the context manager MUST roll back its holder transaction. It does not need to delete the acquisition row separately because rollback removes the insertion atomically with the protected work. + +Insert and delete MUST belong to the same holder transaction. Their successful commit leaves no live acquisition row. Rollback at any point also leaves no live acquisition row. + +Deleting the acquisition row does not release the mutex immediately: competing uniqueness checks continue to wait until the holder transaction commits or rolls back. Callers using `Lock` SHOULD complete the surrounding transaction promptly after leaving the lock context; `locked_transaction` completes its transaction as part of context exit. + +### Transaction contract + +Every logical mutex MUST run inside one explicit database transaction that contains both its acquisition row and all protected database work. + +`Lock` MUST participate in a caller-owned transaction. The execute callable passed to `Lock` MUST be the execute callable of that transaction. + +`Lock` MUST NOT open, commit, or roll back a transaction. It MUST NOT call the top-level autocommitted execute helper for acquisition or cleanup. + +Passing an autocommitted execute callable to `Lock` violates this contract because acquisition could commit a live row before cleanup. Such a row would survive process failure and prevent future acquisition. + +`locked_transaction` MUST own its transaction through the shared transaction infrastructure in `ffun.core.postgresql`. It MUST NOT open an additional connection after starting that transaction, and it MUST use the owned transaction's execute callable for acquisition, cleanup, and protected work. + +Calling code MUST NOT use `locked_transaction` inside an existing database transaction. It MUST use `Lock` when the logical mutex needs to participate in a transaction whose boundary is already owned by the caller. + +The protected database reads and writes MUST use the same transaction execute callable. Using a separate transaction for protected state would allow the lock lifecycle and protected state change to succeed or fail independently. + +The mutex is held until the holder transaction completes. Consequently, leaving a `Lock` context does not make the key available while its caller-owned transaction remains open. Leaving a `locked_transaction` context completes its owned transaction, so its lexical context and mutex lifetime coincide. + +The lock is not reentrant. Code MUST NOT nest acquisition of the same canonical lock identity within one transaction. Such an insertion conflicts with the transaction's own acquisition row and invalidates the transaction. + +When one transaction needs multiple logical mutexes, it MUST acquire them in deterministic order by their canonical `(lock_kind, lock_key)` pairs and release the contexts in reverse order. Calling modules SHOULD acquire logical mutexes before locking or mutating other database rows to reduce deadlock risk. + +PostgreSQL statement cancellation, lock timeouts, deadlock detection, connection failures, and other unexpected infrastructure failures MUST propagate and cause the holder transaction to roll back. + +### Required verification + +Module tests MUST use the real test PostgreSQL service for concurrency behavior. They MUST verify that: + +- a second transaction requesting the same identity waits until the holder transaction completes and then acquires the mutex. +- leaving the lock context does not unblock a waiter before the holder transaction completes. +- successful exit from `locked_transaction` commits protected work, removes the acquisition row, and then allows a waiter to enter. +- exceptional exit from `locked_transaction` rolls back both protected work and the acquisition row before allowing a waiter to enter. +- different lock identities can be acquired concurrently. +- normal commit, protected-body failure, and transaction rollback leave no live acquisition rows. +- a deliberately committed acquisition row causes `LockInvariantViolation` rather than silent takeover. +- canonical encoding distinguishes argument boundaries and argument order, allows boolean values, and intentionally gives integer and string values the same identity when their canonical strings match. +- invalid, unsupported, and oversized identity values are rejected before any acquisition row is inserted. + +### Invariant violations + +A committed acquisition row violates the module's lifecycle invariant. Normal acquisition MUST NOT silently delete, replace, or take ownership of such a row. + +The module MUST convert a uniqueness violation caused by an existing committed row or by same-transaction reentrant acquisition into `ffun.locks.errors.LockInvariantViolation`. The affected transaction remains failed and MUST roll back. + +If normal cleanup does not delete exactly one acquisition row, the module MUST raise `LockInvariantViolation` so the holder transaction rolls back. + +`ffun.locks.errors` MUST define the module root `Error`, `InvalidLockKey`, and `LockInvariantViolation` according to `specs/backend_architecture/errors.md`. + +## Database schema + +### `lk_locks` + +The module MUST own exactly one table, `lk_locks`. + +```sql +-- Ephemeral exact-key rows used to coordinate holder transactions. +CREATE TABLE lk_locks ( + lock_kind TEXT COLLATE "C" NOT NULL, -- Stable open namespace; C collation preserves exact textual identity. + lock_key TEXT COLLATE "C" NOT NULL, -- Canonical ASCII key; C collation preserves exact textual identity. + PRIMARY KEY (lock_kind, lock_key) -- Immediate exact-key uniqueness coordinates competing transactions. +); +``` + +The primary key MUST be immediate and non-deferrable. Its exact uniqueness behavior is the mutex implementation and MUST NOT be replaced with hash uniqueness. + +The `"C"` collation MUST be used so textual primary-key identity follows deterministic exact values rather than locale-sensitive equivalence. + +The table intentionally omits `created_at` and `updated_at`. Acquisition rows have no durable lifecycle: they MUST be inserted and deleted in one transaction or removed by rollback, and no live row may commit. Timestamps would therefore describe neither durable creation nor mutation and would add width and write churn to a hot technical table. + +No secondary indexes are required because runtime acquisition and release address the complete primary key and normal committed state contains no rows. The table MUST NOT define foreign keys, business-value constraints, ownership columns, lease deadlines, or payload columns. + +Normal committed state MUST contain zero live rows. Insert-and-delete transactions still create dead tuples and write-ahead-log traffic, so ordinary PostgreSQL vacuum behavior and operational monitoring remain relevant even though the logical table does not grow without bound. + +The module MUST NOT expose normal runtime cleanup that deletes committed rows. A committed row indicates transaction-contract misuse or an implementation defect and requires explicit diagnosis before repair. + +## Domain interface + +### `Lock` + +`ffun.locks.domain` MUST expose an asynchronous context manager named `Lock`. + +`Lock` MUST accept: + +- the caller's transaction-scoped `ffun.core.postgresql.ExecuteType` as its first argument. +- a `LockKind` as its second argument. +- zero or more positional lock arguments after the kind. + +The interface SHOULD be used in this form: + +```python +from ffun.core.postgresql import transaction +from ffun.locks.domain import Lock +from ffun.locks.entities import LockKind + +async with transaction() as execute: + async with Lock(execute, LockKind("entitlements_user_kind"), user_id, kind_id): + # Read and change the state protected by this logical mutex. + ... +``` + +On context entry, `Lock` MUST validate and encode the identity, insert the acquisition row, wait when another holder transaction owns the same identity, and enter the protected body only after acquisition. + +On context exit, `Lock` MUST perform the release behavior defined above and MUST NOT suppress exceptions from the protected body. + +`Lock` SHOULD be used when the caller already owns a transaction or when one transaction must acquire multiple logical mutexes. + +### `locked_transaction` + +`ffun.locks.domain` MUST expose an asynchronous context manager named `locked_transaction`. + +`locked_transaction` MUST accept: + +- a `LockKind` as its first argument. +- zero or more positional lock arguments after the kind. + +The interface SHOULD be used in this form: + +```python +from ffun.locks.domain import locked_transaction +from ffun.locks.entities import LockKind + +async with locked_transaction(LockKind("entitlements_user_kind"), user_id, kind_id) as execute: + # All protected database work uses the yielded transaction execute callable. + ... +``` + +On context entry, `locked_transaction` MUST validate and encode the lock identity, open one transaction through `ffun.core.postgresql`, insert the acquisition row through that transaction, wait when another holder owns the same identity, and yield the transaction's `ExecuteType` callable after acquisition. + +If transaction creation or acquisition fails, `locked_transaction` MUST roll back and close the transaction before propagating the failure. + +On successful body completion, `locked_transaction` MUST delete exactly one acquisition row and commit the transaction before returning. The mutex is released by that commit. + +On exceptional body completion, `locked_transaction` MUST roll back the transaction and MUST NOT suppress the protected-body exception. The mutex is released by that rollback. + +`locked_transaction` SHOULD be preferred when the protected workflow can give it ownership of the complete transaction because its lexical context then accurately represents the mutex lifetime. + +`locked_transaction` and `Lock` MUST use the same lock identity, acquisition, cleanup, and invariant-violation behavior. + +Calling modules MUST import `Lock` and `locked_transaction` from `ffun.locks.domain`. They MUST NOT depend on the acquisition-row schema directly. + +## Audit records + +Module does not produce audit records because logical mutex acquisition is transient technical coordination rather than a durable business fact. + +## Business events + +Module does not produce business events. Calling modules own any business events generated by state changes performed while a logical mutex is held. diff --git a/specs/backend_architecture/modules_layout.md b/specs/backend_architecture/modules_layout.md index 50f54b3c..a370e013 100644 --- a/specs/backend_architecture/modules_layout.md +++ b/specs/backend_architecture/modules_layout.md @@ -29,12 +29,14 @@ Product-specific choices that configure those reusable modules for Feeds Fun SHO - `ffun.api` owns HTTP API endpoint wiring. - `ffun.api.spa` owns API endpoints used by the frontend. - `ffun.application` owns application construction and application-wide settings. +- `ffun.audit` owns append-only audit records created transactionally by backend modules. - `ffun.auth` owns authentication and authorization logic. - `ffun.cli` owns command-line commands for managing the application. - `ffun.core` owns framework-level base classes, utilities, logging, metrics, PostgreSQL helpers, plugins, and shared infrastructure. - `ffun.data_protection` owns data protection and privacy-related behavior. - `ffun.dispatcher` owns dispatching entries to tag processor queues and tracking per-processor entry processing status. - `ffun.domain` owns cross-domain entities and domain utilities. +- `ffun.entitlements` owns source-specific user entitlement state and merged effective entitlements. - `ffun.feeds` owns feed storage and feed management. - `ffun.feeds_collections` owns curated feed collection configuration and behavior. - `ffun.feeds_discoverer` owns discovery of feeds for external sites. @@ -43,6 +45,7 @@ Product-specific choices that configure those reusable modules for Feeds Fun SHO - `ffun.integrations` owns source-specific integration plugins. - `ffun.librarian` owns tag processor orchestration and processor implementations. - `ffun.library` owns storage and management of news entries. +- `ffun.locks` owns collision-free, transaction-scoped logical mutexes backed by PostgreSQL. - `ffun.llms_framework` owns provider-neutral LLM framework logic. - `ffun.loader` owns loading news entries from feeds. - `ffun.markers` owns read/unread and similar entry markers. @@ -134,10 +137,14 @@ Domain functions SHOULD hide low-level communication details from callers. Calle Top/input layers such as `ffun.api`, `ffun.api.spa`, and `ffun.cli` SHOULD call domain boundaries instead of operations when invoking business behavior. -When a domain-level function only exposes an operation function without adding behavior, `domain.py` SHOULD prefer a direct assignment alias, such as `save_feed = operations.save_feed`, instead of a trivial wrapper. +When a domain-level function only exposes an operation function without adding real domain behavior, such as a business decision, orchestration, validation, or result transformation, `domain.py` MUST re-export the operation instead of introducing a trivial wrapper. A direct re-export preserves one callable contract and keeps its signature and error behavior identical at both boundaries by construction; a wrapper creates a second contract that can drift without adding domain semantics. A direct assignment alias, such as `save_feed = operations.save_feed`, is the conventional form. + +Generating persistence ids, applying storage defaults, and returning identifiers for inserted records are operation-level mechanics. They MUST NOT be treated as domain behavior that justifies a domain wrapper. Domain wrappers SHOULD be used when they add real behavior, such as orchestration, validation, transaction ownership, fallback logic, caching, error conversion, or result shaping. +A module MAY expose a coherent module-owned capability through its public domain boundary before production callers exist when a plausible future use is anticipated. The absence of a current runtime caller is not, by itself, a reason to keep that capability private or test-only. Such an interface MUST still have a defined contract and direct tests. + ### `settings.py` The `settings` submodule owns module-specific configuration parsing and defaults. @@ -198,21 +205,33 @@ Production modules MUST NOT import `tests.helpers`. Backend modules have different dependency roles. +Production database schemas and runtime database access in one top-level module MUST NOT reference tables owned by another top-level module. Cross-module persistence behavior MUST go through the owning modules' public domain boundaries instead of bypassing those boundaries with SQL. + +Tests MAY reference tables owned by other top-level modules when direct database access is useful for test setup, assertions, or cleanup. This exception MUST NOT introduce cross-module references in production schemas, migrations, or runtime database operations. + ### Foundational modules -`ffun.core`, `ffun.domain`, and `ffun.product` are foundational modules. They own shared technical primitives, cross-domain value types and utilities, and product-wide definitions that bind reusable domain mechanisms to Feeds Fun product choices. Other backend modules MAY import any submodule of a foundational module when they need functionality owned by that submodule. Foundational modules SHOULD avoid depending on domain-level or edge-layer modules, so shared primitives do not acquire feature or interface dependencies. +`ffun.core`, `ffun.domain`, and `ffun.product` are foundational modules. They own shared technical primitives, cross-domain value types and utilities, and product-wide definitions that bind reusable domain mechanisms to Feeds Fun product choices. Other backend modules MAY import any submodule of a foundational module when they need functionality owned by that submodule. Foundational modules MUST NOT depend on shared-service, domain-level, or edge-layer modules, so shared primitives do not acquire higher-level dependencies. + +### Shared-service modules + +`ffun.audit`, `ffun.locks`, and `ffun.queues` are shared-service modules. They provide common capabilities that are always available to more specialized domain-level and edge-layer modules while remaining outside the foundational core. + +Shared-service modules MAY depend on foundational modules and on the public `domain`, `entities`, or `errors` boundaries of other shared-service modules. They MUST NOT depend on domain-level or edge-layer modules. + +Production callers MUST use a shared-service module through its `domain`, `entities`, or `errors` boundary. They MUST NOT import a shared-service module's implementation submodules, including `operations`. ### Domain-level modules Most backend modules are domain-level modules. They own reusable business capabilities for one domain area and SHOULD keep product-specific choices and application/interface wiring outside their implementation. Product-specific choices SHOULD live in `ffun.product`; external interface concerns SHOULD live in edge-layer modules. -Production code in one domain-level module that needs types, values, behavior, or errors from another domain-level module MUST use only that module's `domain`, `entities`, or `errors` submodules. Domain-level modules SHOULD expose cross-module production API through those submodules when such API is needed. +Production code in one domain-level module that needs types, values, behavior, or errors from another domain-level or shared-service module MUST use only that module's `domain`, `entities`, or `errors` submodules. Domain-level and shared-service modules SHOULD expose cross-module production API through those submodules when such API is needed. A supported domain API MAY return or accept objects whose concrete classes are defined in the module's implementation submodules. The cross-module dependency rule constrains the import path used by callers; it does not require every object reachable through a public API to be defined in `domain`, `entities`, or `errors`. Callers MUST still import and call the supported API through the allowed submodules, and MUST NOT import implementation submodules only to access the same objects directly. Tests MAY additionally import another domain-level module's `tests.make` and `tests.helpers` submodules for reusable test data construction and setup helpers. Production code MUST NOT import another module's `tests`, `tests.make`, or `tests.helpers` submodules. -Domain-level modules MUST NOT import implementation submodules from another domain-level module. Domain-level modules MUST NOT import another domain-level module's `operations` submodule. +Domain-level modules MUST NOT import implementation submodules from another domain-level or shared-service module. Domain-level modules MUST NOT import another module's `operations` submodule. ### Edge-layer modules diff --git a/specs/backend_architecture/python.md b/specs/backend_architecture/python.md new file mode 100644 index 00000000..6a4c00c7 --- /dev/null +++ b/specs/backend_architecture/python.md @@ -0,0 +1,52 @@ +# Backend Python architecture + +## Goal of the document + +This document describes language-level implementation conventions for Python code in the Feeds Fun backend. + +## Scope + +This specification applies to Python code under `ffun/ffun` and covers project-wide conventions for using Python language features and runtime constructs. + +Module ownership, domain behavior, entity modeling, error behavior, test organization, formatting, and third-party implementation details are out of scope. + +## Dictionary + +- `project-controlled class` — a class whose instance layout is defined by Feeds Fun code rather than generated or prescribed by a framework, library, standard-library protocol, metaclass, or inherited implementation contract. +- `test class` — a class defined only to organize or support automated tests. +- `instance dictionary` — the per-instance `__dict__` used to store dynamically named attributes. + +## Runtime type validation + +Functions called through project-controlled typed interfaces SHOULD treat annotated parameter types as caller contracts. They SHOULD NOT perform runtime checks solely to verify that arguments match their declared types. + +Runtime validation remains appropriate at untyped or external boundaries, including HTTP input, CLI input, configuration, deserialized database rows, third-party responses, and plugin data. + +Code MUST still validate semantic constraints that type annotations cannot express, such as non-empty identifiers, timezone awareness, numeric ranges, configured values, and valid cross-field combinations. + +## Class instance layout + +Explicit instance layouts prevent unintended instance dictionaries and dynamically named state, and they keep attribute ownership predictable across inheritance. `__slots__` is required for covered classes because it enforces this layout constraint at the Python class boundary. + +New or substantially changed project-controlled classes other than test classes MUST define `__slots__` explicitly. Test classes are excluded from this convention and MAY omit `__slots__` without an explanatory comment. + +A class that introduces instance attributes MUST list every attribute it introduces in `__slots__`. A class that introduces no instance attributes MUST use `__slots__ = ()`. + +Every project-controlled subclass covered by this convention MUST define its own `__slots__`, including an empty declaration when it introduces no attributes. A subclass MUST NOT repeat slot names owned by a base class. + +Classes MUST NOT include `__dict__` in `__slots__` unless dynamically named instance attributes are an intentional part of the class contract. Classes MUST include `__weakref__` only when instances need weak-reference support and no base class already provides it. + +Existing classes SHOULD NOT be changed solely to adopt `__slots__`. They MUST adopt this convention when their instance layout or implementation is substantially changed, unless an exception below applies. + +### Exceptions + +A class MAY omit an explicit `__slots__` declaration when one or more of the following conditions apply: + +- a base class already provides an instance dictionary and preserving that inherited layout is required, so a subclass declaration would not provide the intended restriction. +- a framework, library, standard-library protocol, metaclass, or generated implementation owns the instance layout or requires dynamic attributes. +- dynamically named instance attributes are intentional behavior of the class. +- a concrete serialization, proxying, instrumentation, pickling, or interoperability requirement is incompatible with a slotted instance layout. + +Common examples include Pydantic models, enum classes, exception classes, protocols, and framework-defined subclasses whose parent implementation controls instance storage. + +When the reason for omitting `__slots__` is not evident from the base class or implemented protocol, the class MUST have an adjacent comment that states the concrete reason. diff --git a/specs/backend_architecture/tests.md b/specs/backend_architecture/tests.md index 5e1db5e6..ca45d933 100644 --- a/specs/backend_architecture/tests.md +++ b/specs/backend_architecture/tests.md @@ -398,6 +398,14 @@ Tests that change the current working directory MUST restore it before the test Tests SHOULD assert structured values before rendered text when structured values are available. +Persistence operation tests MUST verify the expected persisted effects when the behavior under test inserts or deletes rows, including the number of affected records when cardinality is part of the contract. + +Tests for rejected writes, idempotent no-ops, or other paths whose contract includes unchanged persisted state MUST verify that invariant. + +Assertions about persisted effects MUST isolate the operations responsible for those effects. They MUST supplement, not replace, structured assertions about persisted content when record contents are part of the behavior under test. + +Tests whose subject is a public count operation MAY verify affected-record cardinality through that operation. + Rendered output tests SHOULD assert exact output only for stable API, CLI, or persisted-state contracts. Rendered output tests MAY assert selected lines, fields, or records when exact text is intentionally outside the relevant specification. diff --git a/specs/behavior/cli.md b/specs/behavior/cli.md new file mode 100644 index 00000000..372dc92e --- /dev/null +++ b/specs/behavior/cli.md @@ -0,0 +1,46 @@ +# CLI behavior + +## Goal of the document + +This document describes the stable shared behavior of the Feeds Fun backend command-line interface. + +## Scope + +This specification covers the root backend command and behavior shared by its specified command families. + +Domain behavior, command-family-specific contracts, and frontend interfaces are out of scope. + +## Root command + +The backend CLI MUST be invoked as `ffun` and MUST use a root command group. + +Public command families and standalone commands MUST be exposed as root subcommands. A command family MAY define nested commands for its operations. + +Command names, arguments, options, output, and errors defined by command-family specifications are stable public contracts. + +## Root subcommands + +The root command MUST expose the following command groups: + +- `cleaner` — Cleans orphaned data and expired effective entitlements, and runs tag and feed normalization operations. +- `debug` — Loads and inspects feeds through the available parsing paths. +- [`entitlements`](cli/entitlements.md) — Manages source entitlements and lists effective entitlements. +- `estimates` — Estimates entry publication rates for feeds and collections. +- `experiments` — Runs ad hoc backend data experiments. +- `feeds` — Performs administrative operations on feeds and their user links. +- `fixtures` — Populates the database with development fixture data. +- `metrics` — Reports system-level and per-user operational metrics. +- `processors-quality` — Evaluates processor output and maintains processor quality reference data. +- `profile` — Runs ad hoc backend profiling scenarios. +- `queues` — Cleans all or selected processing queues. +- `user-settings` — Performs maintenance of persisted user settings. +- `users` — Performs user administration and identity-provider imports. + +The root command MUST expose the following standalone commands: + +- `dispatcher-failed-entries-count` — Reports failed entry counts for each processor. +- `dispatcher-failed-entries-move-to-queue` — Moves failed entries back to a selected processor queue. +- `migrate` — Applies pending database migrations. +- `normalize-entries` — Detects and optionally applies entry normalization changes. +- `print-configs` — Prints the settings discovered for backend components. +- `workers` — Runs selected background workers until stopped. diff --git a/specs/behavior/cli/entitlements.md b/specs/behavior/cli/entitlements.md new file mode 100644 index 00000000..944905fc --- /dev/null +++ b/specs/behavior/cli/entitlements.md @@ -0,0 +1,77 @@ +# Entitlements CLI + +## Goal of the document + +This document describes the Feeds Fun CLI command family for managing and inspecting user entitlements. + +## Scope + +This specification covers the public `ffun entitlements` command family and the entitlement capabilities it exposes. + +Entitlement domain rules, persistence, audit records, business events, and other CLI command families are out of scope. Output formats for `grant` and `revoke` are not yet specified. + +## Command group + +The root CLI MUST expose `entitlements` as a command group. + +The command group MUST provide CLI access to source entitlement changes and batch effective-entitlement listings. + +## Commands + +The `grant` and `revoke` commands MUST capture one current timestamp and use it consistently to resolve omitted timestamp parameters. + +Kind parameters MUST accept an `EntitlementKindId` enum member name and resolve it to the corresponding enum member before invoking the entitlement domain. Valid names MUST be derived from the enum rather than duplicated in the CLI specification or implementation. + +### `ffun entitlements grant` + +Stores a granted entitlement state for one source, user, and entitlement kind. + +Parameters: + +- `--user-id UUID` — required id of the affected user. +- `--kind NAME` — required registered entitlement kind name. +- `--source ID` — semantic id of the source that owns the state; defaults to `system`. +- `--value INTEGER` — required entitlement value. +- `--starts-at TIMESTAMP` — inclusive activation time in ISO 8601 format with an explicit UTC offset; defaults to the captured current timestamp. +- `--expires-at TIMESTAMP` — exclusive expiration time in ISO 8601 format with an explicit UTC offset; defaults to the captured current timestamp plus 31 days. +- `--actor-kind {user|admin|psp|system}` — kind of the actor initiating the change; defaults to `admin`. +- `--actor-id ID` — stable id of the actor initiating the change; defaults to `admin`. + +### `ffun entitlements revoke` + +Stores a revoked entitlement state for one source, user, and entitlement kind. + +Parameters: + +- `--user-id UUID` — required id of the affected user. +- `--kind NAME` — required registered entitlement kind name. +- `--source ID` — semantic id of the source that owns the state; defaults to `system`. +- `--starts-at TIMESTAMP` — inclusive activation time in ISO 8601 format with an explicit UTC offset; defaults to the captured current timestamp. +- `--expires-at TIMESTAMP` — exclusive expiration time in ISO 8601 format with an explicit UTC offset; defaults to the captured current timestamp plus 31 days. +- `--actor-kind {user|admin|psp|system}` — kind of the actor initiating the change; defaults to `admin`. +- `--actor-id ID` — stable id of the actor initiating the change; defaults to `admin`. + +### `ffun entitlements list` + +Lists effective entitlements at one evaluation time and prints one JSON object on its own line for every requested user and selected entitlement kind, including pairs whose entitlement is not granted. + +Parameters: + +- `--user-id UUID` — required affected-user filter; MAY be supplied multiple times. +- `--kind NAME` — optional entitlement-kind filter; MAY be supplied multiple times. When omitted, the command returns all registered entitlement kinds for every requested user. + +Every output object MUST contain these fields: + +- `user_id` — requested user id serialized as a UUID string. +- `kind` — entitlement kind enum member name. +- `kind_id` — stable integer value of the entitlement kind enum member. +- `granted` — whether an effective interval covers the command's evaluation time. +- `value` — effective integer value when granted, otherwise `null`. +- `starts_at` — inclusive start of the active effective interval as an ISO 8601 timestamp with an explicit UTC offset when granted, otherwise `null`. +- `expires_at` — exclusive end of the active effective interval as an ISO 8601 timestamp with an explicit UTC offset when granted, otherwise `null`. + +All fields MUST be present in every output object. Records MUST preserve the first occurrence order of requested users. Kind records MUST preserve the first occurrence order of explicit kind filters, or entitlement registry order when the kind filter is omitted. + +## Integration boundary + +Entitlement commands MUST invoke the public `ffun.entitlements.domain` interface. They MUST NOT reproduce entitlement validation, merging, timeline materialization, audit, or business-event behavior. diff --git a/specs/dictionary.md b/specs/dictionary.md index ead4fcce..9bbecd69 100644 --- a/specs/dictionary.md +++ b/specs/dictionary.md @@ -18,6 +18,7 @@ Detailed behavior, implementation requirements, and configuration schemas are ou - `rule` - user-defined score expression based on tags. - `collection` - curated feed collection configuration. - `integration` - external source-specific behavior, such as YouTube or Reddit support. +- `audit record` - append-only durable record of a business change or event, including its actor and subject entities. - `backend` - Python application in `ffun/ffun`. - `frontend` - Vue application in `site/src`. - `development helper` - Docker-backed command in `bin`. diff --git a/specs/intro.md b/specs/intro.md index b5c11d03..276422eb 100644 --- a/specs/intro.md +++ b/specs/intro.md @@ -13,7 +13,10 @@ Detailed requirements for individual specifications are out of scope except for ## Specification directories - `specs/` contains all project specifications used by depmesh governance rules. -- `specs/backend_architecture/` contains specifications related to backend architecture, database access, entities, tests, and errors. +- `specs/backend_architecture/` contains specifications related to backend architecture, Python conventions, database access, entities, tests, and errors. +- `specs/backend_architecture/modules/` contains specifications for individual backend modules. +- `specs/behavior/` contains specifications for externally visible application behavior. +- `specs/behavior/cli/` contains specifications for individual backend CLI command families. - `specs/frontend_architecture/` contains specifications related to frontend architecture and tests. - `specs/documentation/` contains specifications related to repository documentation artifacts. - `specs/meta/` contains specifications related to requirements for specification documents. @@ -24,11 +27,18 @@ Detailed requirements for individual specifications are out of scope except for - `specs/intro.md` is this file and indexes all specification documents. - `specs/dictionary.md` defines Feeds Fun and dependency metadata terms shared by multiple specifications. - `specs/meta/general.md` defines general rules for project specification documents. +- `specs/meta/backend_modules.md` defines the common structure and dependency metadata requirements for backend module specifications. - `specs/backend_architecture/modules_layout.md` describes backend package layout and ownership boundaries. +- `specs/backend_architecture/python.md` describes language-level implementation conventions for backend Python code. - `specs/backend_architecture/db.md` describes backend database access, transactions, migrations, and database-focused testing practices. - `specs/backend_architecture/entities.md` describes backend entity and data structure architecture. - `specs/backend_architecture/errors.md` describes backend error and warning architecture. - `specs/backend_architecture/tests.md` describes backend pytest test placement. +- `specs/backend_architecture/modules/audit.md` describes append-only audit persistence and its transactional domain interface. +- `specs/backend_architecture/modules/entitlements.md` describes entitlement source ownership, merging, persistence, audit history, and business events. +- `specs/backend_architecture/modules/locks.md` describes collision-free, transaction-scoped logical mutexes backed by PostgreSQL. +- `specs/behavior/cli.md` describes behavior shared by the backend CLI command families. +- `specs/behavior/cli/entitlements.md` describes the CLI command family for managing and inspecting entitlements. - `specs/frontend_architecture/modules_layout.md` describes frontend source layout and ownership boundaries. - `specs/frontend_architecture/tests.md` describes frontend Vitest test placement. - `specs/documentation/readme.md` describes repository README expectations. diff --git a/specs/meta/backend_modules.md b/specs/meta/backend_modules.md new file mode 100644 index 00000000..34e52df4 --- /dev/null +++ b/specs/meta/backend_modules.md @@ -0,0 +1,121 @@ +# Backend module specification requirements + +## Goal of the document + +This document describes the common location, naming, structure, and content conventions for specifications of individual Feeds Fun backend modules. + +## Scope + +This specification applies to module specifications under `specs/backend_architecture/modules/`. + +General specification style, backend-wide architecture, implementation package layout, and requirements that apply uniformly to all backend modules are out of scope. + +## Dictionary + +- `module specification` - a specification dedicated to the responsibilities and behavior of one top-level `ffun` backend module. +- `module-specific section` - a section that describes concepts or behavior unique to the module being specified. + +## Location and identity + +Each module specification MUST be stored as `specs/backend_architecture/modules/.md`, where `` matches the intended or existing top-level Python package name under `ffun/ffun//`. + +A module specification MAY exist before its implementation package is created. + +There MUST be at most one module specification for each top-level backend module. + +The document's `h1` title MUST identify the module and end with `module`, such as `# Audit module` for `ffun.audit`. + +A module specification MUST supplement the backend-wide architecture specifications. It MUST NOT repeat general package, database, entity, error, or test requirements unless it adds a module-specific constraint. + +## Required structure + +In addition to the sections required for every specification by `specs/meta/general.md`, a module specification MUST contain these top-level sections: + +- `Module responsibility`. +- `Domain behavior`. + +Standardized top-level sections MUST appear in this order: + +1. `Goal of the document`. +2. `Scope`. +3. `Dictionary` [can be empty]. +4. `Module responsibility`. +5. `Domain behavior`. +6. `Database schema` [can be empty]. +7. `Domain interface` [can be empty]. +8. `Audit records` [can be empty]. +9. `Business events` [can be empty]. + +A section marked `[can be empty]` MAY contain no module requirements. In that case, the section MUST contain one plain-text sentence explaining why, such as `Module does not require persistent storage.` + +Module-specific sections SHOULD be nested under `Domain behavior`. A module specification MAY use an additional top-level section when the concern is substantial, stable, and does not fit a standardized section. + +## Standard sections + +### `Module responsibility` + +The `Module responsibility` section MUST identify the module's architectural role and the behavior, entities, storage, or integration boundaries it owns. + +The section SHOULD identify important behavior that callers must access through the module boundary. It SHOULD NOT enumerate implementation files or private helpers. + +### `Domain behavior` + +The `Domain behavior` section MUST describe the module-specific concepts, invariants, state transitions, calculations, and workflows needed to implement the module correctly. + +Distinct concepts or workflows SHOULD use nested sections. Transaction and concurrency requirements SHOULD stay with the workflow they constrain. + +### `Database schema` + +A module specification that defines persistent module-owned state MUST include a `Database schema` section. + +The section MUST identify each owned table and its columns, types, nullability, primary keys, foreign keys, uniqueness rules, and required indexes. It MUST distinguish source-of-truth state from derived or historical state when the module owns more than one form. + +Each table SHOULD use a nested section named after the table. + +Each table schema MUST be expressed as PostgreSQL DDL in a fenced `sql` code block. Markdown tables MUST NOT be used to define database schemas. + +The SQL MUST include all specified columns, types, nullability, defaults, keys, foreign keys, uniqueness rules, and indexes. Each column MUST have an adjacent SQL comment that describes its domain meaning. Comments MUST also explain intentionally templated identifiers and structural database constraints whose purpose is not clear from their names. + +Database schemas MUST limit database-enforced constraints to structural storage integrity, such as nullability, primary keys, foreign keys, and uniqueness. Business invariants, including allowed values, cross-column value combinations, and state-transition rules, MUST be defined under `Domain behavior` and enforced by the module's domain or service logic before persistence. Table DDL MUST NOT use `CHECK` constraints or other schema-level validation to enforce business invariants. + +Every `Database schema` section MUST explicitly address secondary indexes. Required secondary indexes MUST be expressed as `CREATE INDEX` or `CREATE UNIQUE INDEX` statements in the corresponding table subsection, next to the table DDL. Each index statement MUST have an adjacent SQL comment that explains the query or invariant it supports. + +When a module requires no secondary indexes, the `Database schema` section MUST contain one plain-text sentence that states this and explains why. + +Prose MAY accompany the SQL to explain ownership, derived-state behavior, or other requirements that cannot be expressed by DDL alone. + +### `Domain interface` + +A module specification SHOULD include a `Domain interface` section when another module needs a stable call contract that is not sufficiently described by the responsibility and behavior sections. + +The section MUST describe public behavior through `ffun..domain`. It MAY define stable operation names, arguments, return values, transaction participation, and failure behavior when those details are part of the cross-module contract. + +The section SHOULD specify supported import paths, public names, call shapes, accepted values, yielded or returned values, errors, transaction behavior, and observable lifecycle semantics when those details form the stable contract. + +The section MUST describe callable interfaces by their observable protocol. It MUST NOT require that a public callable is implemented as a class, function, decorated generator, or callable object unless callers depend on that distinction through type identity, inheritance, instance reuse, introspection, or another explicitly documented contract. + +The section MUST NOT specify private helpers or expose the module's `operations` boundary to callers. + +### `Audit records` + +A module specification that defines concrete durable audit events for its workflows MUST include an `Audit records` section. + +The section MUST define when each record is appended, its event name, actor and subject entity roles, required attributes, transaction boundary, and no-op behavior. + +The section MUST NOT treat business-event logs as durable audit storage. + +### `Business events` + +A module specification that generates business events MUST include a `Business events` section. + +The section MUST define each event name, the condition that generates it, its business-event user, and required attributes. It SHOULD distinguish effective-state events from source or request events when both exist. + +## Dependency metadata + +Depmesh MUST relate each module specification to Python files under the top-level backend module with the matching name. + +The `governed_by` relation for `ffun/ffun//**/*.py` MUST include `specs/backend_architecture/modules/.md` when that specification exists. + +The reverse `governs` relation for a module specification MUST include existing Python files under `ffun/ffun//`. + +All module specifications MUST be governed by this specification through depmesh. These relationships SHOULD use generic name-based rules rather than one rule per module. diff --git a/specs/meta/general.md b/specs/meta/general.md index 59a3b395..0f3dde6e 100644 --- a/specs/meta/general.md +++ b/specs/meta/general.md @@ -107,3 +107,11 @@ When a requirement can be expressed either as an implementation detail or as a g For example, a specification SHOULD require closed sets of named values to use enums instead of raw strings. It SHOULD NOT require a specific enum class name or file location unless that class name or location is itself a stable architectural boundary. Examples in specifications SHOULD illustrate behavior or ownership. Examples SHOULD NOT be treated as a place to enumerate every current implementation file or symbol. + +## Implementation neutrality + +Specifications MUST define required behavior and stable observable contracts without prescribing an implementation mechanism when multiple implementations can satisfy the contract. + +A specification MAY require a concrete mechanism, such as a class, function, decorator, inheritance hierarchy, or specific helper, only when that mechanism is itself an architectural requirement or an observable caller contract. The specification MUST state why the mechanism matters or which required property an alternative implementation would violate. + +Specifications MUST NOT resolve an implementation choice merely to make a requirement or example more concrete. When the implementation choice does not affect the required behavior or stable contract, the specification MUST omit that choice. diff --git a/workflows/inconsistency-check.donna.md b/workflows/inconsistency-check.donna.md index da943f7d..1e03f409 100644 --- a/workflows/inconsistency-check.donna.md +++ b/workflows/inconsistency-check.donna.md @@ -5,7 +5,9 @@ kind = "donna.lib.workflow" start_operation_id = "run_consistency_cycle" ``` -Repeatedly check files changed relative to `main` against their `depmesh`-related artifacts, pausing for the primary agent to repair the first current inconsistency. +Repeatedly discover dependency-ready frontiers among files changed relative to `main`, check one frontier per cycle, +and pause for the primary agent to repair the first current inconsistency in that frontier. Exit code `20` rebuilds the +graph for the next frontier; a fresh cycle with no pending work exits successfully. ## Run Consistency Cycle @@ -37,7 +39,7 @@ id = "fix_first_inconsistency" kind = "donna.lib.request_action" ``` -The consistency checker found the first current inconsistent relation pair. +The consistency checker found the first current inconsistent relation pair in the active dependency-ready frontier. Stdout: diff --git a/workflows/polish.donna.md b/workflows/polish.donna.md index 158ff826..4463798b 100644 --- a/workflows/polish.donna.md +++ b/workflows/polish.donna.md @@ -16,6 +16,7 @@ fsm_mode = "start" save_stdout_to = "backend_tests_output" goto_on_success = "run_frontend_tests" goto_on_failure = "fix_broken_test" +timeout = 300 ``` ```bash donna script