Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bacb78d
test(zotero): require complete steward review coverage
seonghobae Sep 4, 2026
65c107d
feat(zotero): verify complete steward review coverage
seonghobae Sep 4, 2026
6e8925b
docs(zotero): define full review completion KPI
seonghobae Sep 4, 2026
59d05d1
test(zotero): preserve complete review validation errors
seonghobae Sep 4, 2026
40c1cc0
fix(zotero): validate reviews before approval boundary
seonghobae Sep 4, 2026
64de274
docs(zotero): order review validation before authority
seonghobae Sep 4, 2026
7d17861
test(zotero): reject invalid full review before approval
seonghobae Sep 4, 2026
ff594c8
chore(zotero): adopt private report review repair
seonghobae Sep 4, 2026
d6bda3c
Merge current private-report parent into review evaluation
seonghobae Sep 4, 2026
bcd850f
Merge repaired write receipt evidence into zotero-complete-review-eva…
seonghobae Sep 4, 2026
c2357e1
chore(zotero): restack complete review evaluation
seonghobae Sep 4, 2026
dfd3e43
Merge remote-tracking branch 'origin/autoresearch/zotero-private-repo…
seonghobae Sep 4, 2026
7d6813d
merge(zotero): adopt current private-report parent
seonghobae Sep 4, 2026
005826d
merge(zotero): adopt current private-report parent and gap baseline
seonghobae Sep 5, 2026
68e3186
merge(research): inherit verified source and proposal approval binding
seonghobae Sep 5, 2026
62966fb
merge(research): inherit canonical local transport repairs into PR #24
seonghobae Sep 5, 2026
fdca08c
merge(research): inherit deterministic transport framing regression i…
seonghobae Sep 5, 2026
ba3a691
merge(zotero): propagate validated approval ordering through PR 24
seonghobae Sep 5, 2026
1e73e15
merge(research): inherit bounded metadata reads into PR #24
seonghobae Sep 6, 2026
b4c16a4
merge(research): preserve PR24 coverage and verified PR23 source boun…
seonghobae Sep 6, 2026
8c0ef44
test(research): reject pending sources as complete review evidence
seonghobae Sep 6, 2026
ac59477
fix(research): require resolved source scope before completion
seonghobae Sep 6, 2026
b868f39
test(research): retain pending ancestry validation at completion boun…
seonghobae Sep 6, 2026
5b2282a
fix(research): clarify complete review scope and reuse owned fixture
seonghobae Sep 6, 2026
35c57ca
docs(research): record PR24 pending-source RED and verified scope
seonghobae Sep 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ All notable changes to ConceptWeave are documented here.
- Read-only delayed reconciliation receipts for indeterminate Zotero rollback operations.
- Minimal, nonduplicated local abstract context for Zotero items that require steward classification.
- Owner-only file permissions for sensitive local Zotero classification reports.
- A complete-review evaluator that rejects partial steward labels as full reclassification evidence.

### Security

Expand Down
27 changes: 27 additions & 0 deletions crates/conceptweave-zotero/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1241,6 +1241,8 @@ pub enum EvaluationError {
UnknownItem,
/// A reviewed key occurs more than once.
DuplicateItem,
/// Bibliographic labels are incomplete or source records remain unresolved.
IncompleteReview,
}

impl fmt::Display for EvaluationError {
Expand All @@ -1254,12 +1256,37 @@ impl fmt::Display for EvaluationError {
}
Self::UnknownItem => "golden set contains an item absent from the report",
Self::DuplicateItem => "golden set contains a duplicate item",
Self::IncompleteReview => {
"complete review must label every bibliographic item exactly once and resolve all pending sources"
}
})
}
}

impl std::error::Error for EvaluationError {}

/// Evaluates a review covering every bibliographic item with no unresolved sources.
///
/// Standalone sources, orphan trees and disconnected cycles must be resolved
/// before completion. Sampled quality evaluation remains available separately.
/// Success proves reviewed metadata coverage, not a Zotero write or full-text approval.
pub fn evaluate_complete_reviewed_classification<F>(
report: &ClassificationReport,
golden: &ReviewedGoldenSet,
verify_approval: F,
) -> Result<GoldenSetEvaluation, EvaluationError>
where
F: FnOnce(&ReviewedGoldenSet) -> bool,
{
if golden.labels.len() != report.classified_items.len()
|| !report.pending_source_item_keys.is_empty()
{
return Err(EvaluationError::IncompleteReview);
}
let evaluation = evaluate_reviewed_golden_set(report, golden, verify_approval)?;
Ok(evaluation)
}

/// Checks that every observed item belongs to exactly one report partition.
///
/// Child links and unresolved source keys are recomputed from preserved metadata.
Expand Down
132 changes: 131 additions & 1 deletion crates/conceptweave-zotero/tests/golden_set_evaluation.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
use conceptweave_zotero::{
Disposition, EvaluationError, GoldenLabel, GoldenSetApproval, ItemData, ReviewedGoldenSet,
SnapshotItemRevision, ZoteroItem, classification_proposal_digest,
classification_snapshot_digest, classify_snapshot, evaluate_reviewed_golden_set,
classification_snapshot_digest, classify_snapshot, evaluate_complete_reviewed_classification,
evaluate_reviewed_golden_set,
};

fn item(key: &str, title: &str) -> ZoteroItem {
Expand All @@ -21,6 +22,134 @@ fn item(key: &str, title: &str) -> ZoteroItem {
}
}

#[test]
fn complete_review_requires_one_steward_label_per_bibliographic_item() {
use std::cell::Cell;

let report = report();
let verifier_calls = Cell::new(0);
assert_eq!(
evaluate_complete_reviewed_classification(&report, &golden(vec![]), |_| {
verifier_calls.set(verifier_calls.get() + 1);
true
},),
Err(EvaluationError::IncompleteReview)
);
assert_eq!(
evaluate_complete_reviewed_classification(
&report,
&golden(vec![
GoldenLabel::new("A", Disposition::Generation),
GoldenLabel::new("B", Disposition::EvaluationGovernance),
]),
|_| {
verifier_calls.set(verifier_calls.get() + 1);
true
},
),
Err(EvaluationError::IncompleteReview)
);
assert_eq!(verifier_calls.get(), 0);
assert_eq!(
evaluate_complete_reviewed_classification(
&report,
&golden(vec![
GoldenLabel::new("A", Disposition::Generation),
GoldenLabel::new("A", Disposition::Generation),
GoldenLabel::new("B", Disposition::EvaluationGovernance),
]),
|_| {
verifier_calls.set(verifier_calls.get() + 1);
true
},
),
Err(EvaluationError::DuplicateItem)
);
assert_eq!(verifier_calls.get(), 0);

let evaluation = evaluate_complete_reviewed_classification(
&report,
&golden(vec![
GoldenLabel::new("A", Disposition::Generation),
GoldenLabel::new("B", Disposition::EvaluationGovernance),
GoldenLabel::new("C", Disposition::OutOfScope),
]),
verify_synthetic_approval,
)
.unwrap();
assert_eq!(evaluation.reviewed_count, 3);
}

#[test]
fn complete_review_rejects_pending_sources_without_blocking_sampled_evaluation() {
use std::cell::Cell;

for parent_key in ["", "missing", "source", "A"] {
let mut source_item = item("source", "synthetic attachment");
source_item.data.item_type = "attachment".into();
source_item.data.parent_item = parent_key.into();
let report = classify_snapshot(
"9.0.6".into(),
None,
42,
vec![item("A", "ontology learning"), source_item],
);
let mut reviewed = golden(vec![GoldenLabel::new("A", Disposition::Generation)]);
reviewed.approval.snapshot_digest = classification_snapshot_digest(&report);
reviewed.approval.proposal_digest = classification_proposal_digest(&report);
reviewed.approval.snapshot_items = report.snapshot_items.clone();
let issued_review = reviewed.clone();
assert!(
evaluate_reviewed_golden_set(&report, &reviewed, |value| { value == &issued_review })
.is_ok()
);

let verifier_calls = Cell::new(0);
let result = evaluate_complete_reviewed_classification(&report, &reviewed, |value| {
verifier_calls.set(verifier_calls.get() + 1);
value == &issued_review
});
if parent_key == "A" {
assert_eq!(result.unwrap().reviewed_count, 1);
assert_eq!(verifier_calls.get(), 1);
} else {
assert_eq!(result, Err(EvaluationError::IncompleteReview));
assert_eq!(verifier_calls.get(), 0);
let mut forged_report = report;
forged_report.pending_source_item_keys.clear();
reviewed.approval.proposal_digest = classification_proposal_digest(&forged_report);
assert_eq!(
evaluate_complete_reviewed_classification(&forged_report, &reviewed, |_| {
verifier_calls.set(verifier_calls.get() + 1);
true
}),
Err(EvaluationError::InvalidReview)
);
assert_eq!(verifier_calls.get(), 0);
}
}
}

#[test]
fn invalid_local_review_never_reaches_the_approval_verifier() {
use std::cell::Cell;

let verifier_calls = Cell::new(0);
let result = evaluate_reviewed_golden_set(
&report(),
&golden(vec![
GoldenLabel::new("A", Disposition::Generation),
GoldenLabel::new("A", Disposition::Generation),
]),
|_| {
verifier_calls.set(verifier_calls.get() + 1);
true
},
);
assert_eq!(result, Err(EvaluationError::DuplicateItem));
assert_eq!(verifier_calls.get(), 0);
}

fn report() -> conceptweave_zotero::ClassificationReport {
classify_snapshot(
"9.0.6".into(),
Expand Down Expand Up @@ -236,6 +365,7 @@ fn reviewed_golden_set_rejects_stale_unknown_and_duplicate_labels() {
(EvaluationError::InvalidExpectedDisposition, "abstention"),
(EvaluationError::UnknownItem, "absent"),
(EvaluationError::DuplicateItem, "duplicate"),
(EvaluationError::IncompleteReview, "every bibliographic"),
] {
assert!(error.to_string().contains(fragment));
}
Expand Down
2 changes: 2 additions & 0 deletions docs/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ PR #21 retains validated delayed reads without writes and complete observed meta
The Zotero 10+ adapter can accept a caller-owned API key and server identity at runtime or consume one successful, user-approved Local API authorization. Authorization sends one bounded application name and the expected server identity to the fixed loopback endpoint; only a same-server bounded response that explicitly reports denial is classified as the user's decision. Denial and rate limiting return immediately without another prompt or automatic retry. The private 32-character key is neither serializable nor printable. Authorization, read, and write responses bind to the expected server before status classification; writes name expired authorization and matching-server stale preconditions separately. Thin public execution boundaries connect the adapter to the reviewed write and rollback cores without duplicating mutation logic. Synthetic transport evidence does not satisfy AC6's approved live Zotero 10 authorization, write, and rollback requirement.

Evaluate classifier quality only against a steward-reviewed local golden set whose governance receipt is externally verified and binds both the complete source/classifier-input snapshot and every current proposal field, in addition to the item-key/item-version coordinates. Same-version changes to unmodeled provider metadata, absent/default fields, classifier inputs, predictions or supporting evidence must invalidate the corresponding binding. Evaluation recomputes proposal identity before contacting governance; a locally changed digest cannot renew an approval. Legacy unbound approvals require reissuance, never automatic backfill. Abstention is a prediction outcome, never an approved truth label. Evaluation emits the verified library revision, rule revision, opaque snapshot and proposal digests, and aggregate counts for exact matches, abstentions, and per-disposition true-positive/predicted/expected totals; it must not copy Zotero keys, reviewer identity, or bibliographic text into the result.

A complete metadata-review result additionally requires exactly one non-abstention steward label for every top-level bibliographic item and no unresolved source records. Standalone sources, orphan trees and disconnected cycles must be resolved before completion; clearing their reported list cannot bypass inventory validation. A sampled golden set remains valid for quality measurement but cannot prove completion. Neither result proves full-text approval or an applied Zotero reclassification.
Every successful classification report includes aggregate evidence for snapshot coverage, proposal coverage, provenance completeness, abstentions, duplicate candidates, disposition totals, and zero unreported failures.

## 6. First vertical slice
Expand Down
2 changes: 2 additions & 0 deletions docs/TRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ The proposal-only v1 format above describes the previous receipt contract. The c
Structural, source, proposal, and label checks precede the external verifier. Blank, duplicate, unknown, stale, content-mismatched, prediction-mismatched, label-mismatched, or abstention-as-truth inputs fail closed. The aggregate result retains the verified library version, rule revision, and opaque snapshot/proposal digests, but contains no item keys, reviewer identity, or bibliographic text. Production authorization remains Keyverse/governance-owned; this crate passes the complete reviewed labels and approval bindings to that boundary instead of minting authority.

Provider deserialization captures each complete JSON object before projecting metadata. Snapshot hashing serializes the domain marker `conceptweave-zotero-snapshot-v2` followed by key-ordered pairs of that canonical source JSON and the actual typed classifier input. Unknown nested fields, array order, and omitted-versus-explicit default fields remain bound; changing a typed input after decoding also changes the digest. Synthetic offline typed items have no captured provider object and bind an explicit absent-source value alongside their typed input. Earlier reduced-content digests remain historical evidence and cannot establish this complete-content contract; regenerate the report and review artifacts and obtain fresh approval before any release or approved write.

The complete metadata-review evaluator rejects unequal label cardinality or nonempty `pending_source_item_keys` with `IncompleteReview` before governance. The shared evaluator then recomputes the complete inventory and pending ancestry, so clearing pending keys and rewriting the proposal digest still fails local validation. Because shared validation rejects blank, duplicate, and unknown keys, equal cardinality proves bibliographic label coverage. Sampled evaluation still supports pending sources; completion does not prove a Zotero mutation or full-text approval.
A successful classification report carries an `audit_summary` whose snapshot, bibliographic, proposed-disposition, provenance-complete, abstention, duplicate-candidate, failure, and per-disposition counts are derived from the same in-memory immutable snapshot. Zotero item version zero remains a valid observed coordinate for never-synced Zotero 9 records; provenance completeness rejects a missing item key rather than inventing a positive-only version invariant. Reader failures return an error instead of a partial report; therefore a returned report records `failure_count=0` rather than hiding partial failures.

The local report can contain titles, tags, matched metadata, and abstention abstracts. It is sensitive steward-review material, remains outside the repository, and is not a publication artifact.
Expand Down
2 changes: 2 additions & 0 deletions docs/UML.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,12 @@ sequenceDiagram
Intake->>Report: write proposals, complete inventory and unresolved source keys
Note over Report,Steward: Pending sources prevent a whole-library completion claim; inventory is not approval
Report->>Steward: review dispositions and merge candidates
Steward->>Intake: verified labels for every bibliographic item
Steward->>Intake: reviewed labels and independently issued receipt
Intake->>Intake: validate complete partitions and recompute pending ancestry
Intake->>Intake: verify v2 proposal and retained-source binding
Note over Intake,Steward: Only locally valid reports reach independent governance verification
Intake-->>Steward: aggregate bibliographic review evidence or incomplete-review failure
Steward->>Intake: verified canonical-item decisions
Intake->>Report: before/after/rollback identity manifest
Report-->>Steward: reversible local mapping; source records preserved
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0006-zotero-research-intake.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Matched metadata values are copied into the local-only evidence receipt for repl

Duplicate candidates become canonical references only through externally verified steward decisions bound to the raw digest, complete item-key/item-version snapshot, and exact candidate membership. Overlapping candidates form one connected component and must select one component-level canonical item. Every resulting operation retains all component source revisions and complete before/after/rollback key mappings. It changes downstream identity resolution only; classification does not merge, delete, or mutate Zotero source records.

Classifier quality is measured only against local steward-reviewed labels whose complete reviewed set is verified outside this crate and bound to the exact library version, rule revision, canonical SHA-256 raw-snapshot digest, and every observed parent/child item-key/item-version coordinate. `NeedsStewardReview` is an abstention prediction and cannot be approved truth. Evaluation returns the verified revisions and opaque digest with aggregate integer evidence; Zotero keys, reviewer identity, and bibliographic text are omitted. Missing, stale, content- or label-mismatched, unverified, unknown, duplicate, or invalid review identities fail closed.
Classifier quality is measured only against local steward-reviewed labels whose reviewed set is verified outside this crate and bound to the exact library version, rule revision, canonical SHA-256 raw-snapshot digest, and every observed parent/child item-key/item-version coordinate. `NeedsStewardReview` is an abstention prediction and cannot be approved truth. Sampled labels may measure classifier quality, but a full-reclassification completion result requires exactly one approved label for every classified bibliographic item. Cardinality, snapshot, key, disposition, and duplicate checks run before the external approval verifier so invalid local input cannot consume approval authority. Evaluation returns the verified revisions and opaque digest with aggregate integer evidence; Zotero keys, reviewer identity, and bibliographic text are omitted. Missing, incomplete, stale, content- or label-mismatched, unverified, unknown, duplicate, or invalid review identities fail closed at the applicable completion boundary.
Every successful report includes an aggregate audit summary computed from the same captured snapshot. Zotero 9 item version zero is preserved as a valid never-synced source coordinate, not treated as missing provenance. Partial reads never produce a report, so successful output explicitly records zero failures alongside snapshot, proposal, provenance, abstention, duplicate, and per-disposition totals.

The reader fails closed above 50,000 items or 256 MiB of cumulative response bodies, while retaining the 8 MiB per-page bound, finite request timeouts, redirect denial, total-count checks, snapshot-version checks, and duplicate-key detection. Pagination, consistency, resource-budget, and provider-contract behavior are separated from the narrow `ureq` transport so deterministic tests exercise the production reader core rather than excluding the entire reader from coverage.
Expand Down
Loading