Implement soft delete for pivot records

[OrriMandarin]

Problem

The new data model link management lets users change a link's type (belongs_to ⇄ related). Demoting a belongs_to link must remove its pivot, but pivots are hard-deleted today, and decisions.pivot_id references them with a NO ACTION FK, so deleting any pivot that has produced decisions fails with an FK violation. Hard-deleting also breaks every consumer that resolves a decision's pivot by id (pivot object enrichment on cases, related-cases lookup, scoring).

Solution: soft delete

Pivots get a deleted_at column. Decisions keep referencing their pivot after deletion, so everything reading pivots from historical decisions keeps working. Among the options considered (block deletion forever, soft-delete links too, snapshot pivot metadata), soft delete is the smallest change that preserves all historical data paths.

• Schema: deleted_at on data_model_pivots; the unique index on (organization_id, base_table_id) becomes partial (WHERE deleted_at IS NULL) so a table can get a new pivot after its old one is deleted.
• Link type stays coherent: is_belongs_to is derived from live pivots only, so a demoted link correctly reads as related.
• All user-facing deletion flows (pivot endpoint, link demotion, belongs_to cascade on link deletion, default-pivot replacement) now soft-delete. Hard delete remains only where nothing can reference the pivot (table deletion, org import into an empty org).
• Resurrection: re-creating a pivot with the same definition (e.g. demote then re-promote a link) restores the soft-deleted row instead of inserting a new one, so old and new decisions stay grouped under the same pivot_id.
• Live-only vs. historical reads: ListPivots gains an includeDeleted parameter; only the paths resolving pivots referenced by historical decisions include deleted ones (pivot object enrichment, resurrection lookup). The related-cases CTE intentionally includes them.

Known limitation (deferred by design): links

A soft-deleted pivot's path_link_ids can end up pointing to a hard-deleted link (demote the link, then delete it — nothing references it anymore, so deletion is allowed). The pivot definition then can't be resolved to a table/field, and old decisions lose their entity enrichment.

The problem is deferred, but guarded: ReadPivotObjectsFromValues now (1) detects pivots whose path links no longer resolve and downgrades them to plain-value pivots — skipping ingested-data enrichment instead of querying with an empty table name — and (2) keeps a per-object enrichment failure local: the object is returned without entity data instead of failing the whole batch (previously one broken pivot blanked the entire case view).

Summary by CodeRabbit

• New Features

  • Soft-delete and restore for pivots to preserve historical references.
  • Option to include/exclude deleted pivots when listing pivots.
  • Automatic restoration of matching soft-deleted pivots instead of creating duplicates.

• Improvements

  • Pivot lifecycle now favors soft-deletion to retain identity and history.
  • Decision/export/import flows and navigation options respect soft-deleted pivot state.
  • Pivot enrichment made best-effort (logs warnings and returns partial data when enrichment fails).

• Chores

  • Database migration adds a deleted_at column and adjusts uniqueness to ignore soft-deleted rows.

[linear-code]

MAR-1987

[coderabbitai]

📝 Walkthrough

Walkthrough

Soft-delete support for pivot definitions is added: models, DB schema, repository, and usecases track DeletedAt; delete operations soft-delete pivots; creation may restore matching soft-deleted pivots; ListPivots gains an includeDeleted flag propagated to call sites.

Changes

Soft-Delete Pivot Preservation

Layer / File(s)	Summary
Data model and DB migration models/data_model_pivot.go, repositories/dbmodels/db_data_model_pivot.go, repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql	PivotMetadata and Pivot gain DeletedAt; DbPivot maps deleted_at; migration adds deleted_at and a partial unique index enforcing uniqueness only for non-deleted pivots.
Repository interface and implementation repositories/data_model_repository.go, mocks/data_model_repository.go	ListPivots adds includeDeleted bool and conditionally filters deleted_at IS NULL; cache key updated; GetLinks excludes deleted pivots for belongs-to; new SoftDeletePivot and RestorePivot update deleted_at and invalidate cache; mocks aligned.
Pivot creation and restore usecases/data_model_usecase.go	CreatePivotWithExec restores matching soft-deleted pivots when no live pivot exists using pivotDefinitionMatchesInput; ListPivots call-sites updated for includeDeleted.
Link updates and pivot cascades usecases/data_model_usecase.go	BelongsTo/related link updates and create flows soft-delete replaced pivots instead of hard-deleting them to preserve identity for dependent pivots.
Destroy flows use soft-delete usecases/data_model_destroy_usecase.go	DeletePivot, DeleteLink, and DeleteTable now soft-delete affected pivots; pivot conflict checks use updated ListPivots.
ListPivots parameter propagation usecases/ast_eval/evaluate/*, usecases/decision_usecase.go, usecases/worker_jobs/async_decision_job.go, usecases/org_*.go	Call sites updated to pass the new includeDeleted flag (typically false for operational flows).
Decision rendering / ingested-data reader usecases/ingested_data_reader_usecase.go	ReadPivotObjectsFromValues requests deleted pivots (includeDeleted=true); unresolved object pivots are downgraded to field pivots and enrichment failures are logged but non-fatal.
Tests, docs, mocks usecases/data_model_usecase_test.go, usecases/ast_eval/evaluate/*_test.go, mocks/data_model_repository.go, repositories/case_repository.go	Mocks updated for new signature; new tests verify restore-vs-create behavior and pivotDefinitionMatchesInput; case repo comment clarifies historical inclusion of deleted pivots.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• checkmarble/marble-backend#1644: Overlaps changes to pivot creation paths and related pivot lifecycle edits.

Suggested reviewers

• Pascal-Delange

Lo, pivots spared from final tomb they kept,
Soft timestamps where once hard deletions slept;
When echoes match, the buried rows arise,
Restored to service under kinder skies.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Implement soft delete for pivot records' clearly and concisely summarizes the main objective of the changeset, which introduces soft-deletion support across all pivot-related database operations and code paths.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch leoji/mar-1987-cant-explain-why-a-case-is-mark-as-linked-to-a-client-while

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql`:
- Around line 14-23: Do not hard-delete soft-deleted pivots in the Down
migration: remove the "delete from data_model_pivots where deleted_at is not
null" and the blind "drop column deleted_at" steps; instead either (A) preserve
historical pivots by leaving deleted_at and recreate a unique index that
includes deleted_at (e.g. recreate data_model_pivots_base_table_id_idx on
data_model_pivots (organization_id, base_table_id, deleted_at)) so historical
pivot IDs remain valid for decisions.pivot_id, or (B) if you truly must remove
deleted_at, implement an explicit remapping check first (SELECT id FROM
data_model_pivots WHERE deleted_at IS NOT NULL AND id IN (SELECT pivot_id FROM
decisions)) and abort the migration with a clear error if any referenced
soft-deleted pivots exist, then perform safe remapping before dropping the
column; reference the data_model_pivots table, the index
data_model_pivots_base_table_id_idx, and decisions.pivot_id to locate the
relevant statements.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e1f8a6b1-7bc9-4676-a33e-6247eb654f52

📥 Commits

Reviewing files that changed from the base of the PR and between 35c2ab6 and ec9cf49.

📒 Files selected for processing (17)

• mocks/data_model_repository.go
• models/data_model_pivot.go
• repositories/case_repository.go
• repositories/data_model_repository.go
• repositories/dbmodels/db_data_model_pivot.go
• repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql
• usecases/ast_eval/evaluate/evaluate_entity_annotation_check.go
• usecases/ast_eval/evaluate/evaluate_entity_annotation_check_test.go
• usecases/ast_eval/evaluate/evaluate_record_score.go
• usecases/data_model_destroy_usecase.go
• usecases/data_model_usecase.go
• usecases/data_model_usecase_test.go
• usecases/decision_usecase.go
• usecases/ingested_data_reader_usecase.go
• usecases/org_export_usecase.go
• usecases/org_import_usecase.go
• usecases/worker_jobs/async_decision_job.go

✅ Files skipped from review due to trivial changes (1)

• repositories/case_repository.go

🚧 Files skipped from review as they are similar to previous changes (15)

• usecases/org_export_usecase.go
• usecases/ast_eval/evaluate/evaluate_entity_annotation_check_test.go
• models/data_model_pivot.go
• usecases/worker_jobs/async_decision_job.go
• usecases/decision_usecase.go
• repositories/dbmodels/db_data_model_pivot.go
• usecases/org_import_usecase.go
• mocks/data_model_repository.go
• usecases/ast_eval/evaluate/evaluate_record_score.go
• usecases/ingested_data_reader_usecase.go
• usecases/data_model_destroy_usecase.go
• usecases/data_model_usecase.go
• usecases/data_model_usecase_test.go
• usecases/ast_eval/evaluate/evaluate_entity_annotation_check.go
• repositories/data_model_repository.go

📜 Review details

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Analyze (go)
• GitHub Check: test_backend / Test back-end
• GitHub Check: test_backend / Lint back-end

🧰 Additional context used

📓 Path-based instructions (2)

repositories/migrations/*.sql

📄 CodeRabbit inference engine (CLAUDE.md)

Database migrations must be created using goose and placed in repositories/migrations/ with SQL file format

Files:

• repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql

repositories/**

⚙️ CodeRabbit configuration file

repositories/**: Always use uuid v7 when creating ressources.
Do not let the database create its own ressources, instead generate a uuid v7 if the usecase layer doesn't already do so.

Files:

• repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql

🔇 Additional comments (1)

repositories/migrations/20260611100000_soft_delete_data_model_pivots.sql (1)

1-11: LGTM!