Skip to content

πŸ“ research: ablation plan on consenting hives β€” cross-family reviewer + shadow gates, 30 days each (#12373) - #12400

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
docs/ablation-plan-12373
Oct 16, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
docs/ablation-plan-12373

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Closes #12373. Docs only β€” nothing is switched on; consent is recorded on the issue before any start date.

Part of RFC #12062 (proposal C, Study). cc @mrbobbytables

What

src/docs/research/ablation-plan-2026-10.md (named to match the pre-registration's dated convention; the issue text says 2026-Q4):

  • Auto-merge unchanged everywhere stated up front, with per-arm reasoning: shadow gates are record-only (review.shadow / intent.shadow never hold or weaken require_approval / enforce); the cross-family reviewer changes which model reviews, adds no approval requirement.
  • Consenting hives table (Bluefin / Console / Hive) with consent-link checkboxes and hive-id placeholders.
  • Arms quoted verbatim from pre-reg Β§Experimental arms; each hive runs both arms sequentially, 30 days each, 7-day washout; hash rule (sha256(hive id) first hex digit parity) decides arm order.
  • Schedule table with TBD start/stop placeholders and "who flips it".
  • Exact config diffs citing real hive.yaml leaves (review.shadow, intent.shadow, agents.<reviewer>.review_models.{pool, exclude_author_model, exclude_author_family, fallback}), and a one-revert rollback.
  • Metrics (H3, H7 rows and source definitions) copied verbatim from the pre-registration; where results are read: /api/evidence/summary shadow_gate + reviewer_accuracy_by_model, hub /api/hub/evidence family_false_approve_rate, hivectl evidence export (✨ hivectl: evidence export --since --format csv|jsonl (same schema as the Evidence charts)Β #12367), backfill baseline (✨ research: evidence backfill tooling + runs β€” Bluefin first, Console (L0β†’L6), then Hive/tunaos/Frostyard/Open HorizonΒ #12371).
  • "Predictions we expect to lose" restated (H2, H5, H7 as a result against default gating, H3 possibly underpowered); results reported win or lose on the issue and in the paper.
  • Linked from preregistration-2026-10.md (one navigation line under Tracking, explicitly not an amendment) and review-evidence.md. Changelog fragment added-12373-ablation-plan.md.

Guardrail (RFC #12062 Non-goals)

Record/read-only. Nothing here reads into or alters merge eligibility / auto-merge; no code changes.

Local: check-docs-links.py and check-docs-citations.py clean.

… + shadow gates, 30 days each (#12373)

Docs-only run plan for the pre-registration's two experimental arms:
consent record per hive (Bluefin, Console, Hive), hash-of-hive-id arm order,
30-day periods with placeholder dates, exact hive.yaml diffs
(review.shadow, intent.shadow, review_models.exclude_author_family + pool),
one-revert rollback, metrics quoted verbatim from the pre-registration and
where they are read (/api/evidence/summary shadow_gate and
reviewer_accuracy_by_model, hub family_false_approve_rate, evidence export
and backfill). Auto-merge unchanged everywhere; results reported win or lose.

Linked from preregistration-2026-10.md (navigation only) and
review-evidence.md.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 16, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#12386 ✨ evidence: panel 5 β€” claim vs verdict per week same files src/docs/review-evidence.md
#12395 ✨ hivectl: evidence backfill tooling β€” schema v1 bundles from GitHub history (#12371) same files src/docs/review-evidence.md
#12388 ✨ dashboard: Evidence panel 4 β€” boundary enforcement (attempted vs blocked writes per tier; sentinel hits/week) same files src/docs/review-evidence.md
#12409 ✨ evidence: schedule the monthly complexity/static-analysis snapshot per repo and expose it in the summary API same files src/docs/review-evidence.md
#12389 ✨ evidence: panel 6 β€” work balance (issues opened by kind vs closed) same files src/docs/review-evidence.md
#12390 ✨ intent: per-tier evaluated / would-block counters in the audit log + durable counter file same files src/docs/review-evidence.md

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 30ff020.

@kubestellar-prow
kubestellar-prow Bot deleted the docs/ablation-plan-12373 branch October 16, 2026 13:25
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ“ research: ablation plan on consenting hives β€” cross-family reviewer + shadow-gate accuracy, 30 days each (auto-merge unchanged)

1 participant