diff --git a/CHANGELOG.md b/CHANGELOG.md index 92a2d3f9..ce01ded9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ All notable changes to ConceptWeave are documented here. - Draft 2020-12 JSON Schema for the semantic-candidate public contract. - Standards and research doctoring covering stable W3C ontology standards, 2026 RDF/SHACL work in progress, Apache Ossie, and recent LLM ontology-engineering research. - Read-only delayed reconciliation receipts for indeterminate Zotero rollback operations. +- Minimal, nonduplicated local abstract context for Zotero items that require steward classification. ### Security diff --git a/SECURITY.md b/SECURITY.md index 548c11e5..a0a98d12 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,6 +4,8 @@ All source artifacts, generated candidate payloads, external ontology files, model outputs, and future web-retrieved content are untrusted input. +Local Zotero classification reports can contain bibliographic titles, tags, matched values, and abstention abstracts. They are sensitive review material, remain outside the repository, and are not publication artifacts. + ## Required controls - source size, type, nesting, archive/decompression, and parser-time bounds; @@ -38,4 +40,4 @@ Zotero 10+ write authorization and mutation use the provider-defined loopback HT A hostile same-host process capable of binding, observing, or interposing on the loopback endpoint therefore remains an unresolved credential-confidentiality threat. The currently documented Zotero Local API does not provide an HTTPS or OS-authenticated IPC write endpoint that ConceptWeave can substitute. Consequently, mock/local orchestration may be tested, but enterprise-secure live write-back remains fail closed. It may become release-eligible only if Zotero provides a protected transport or an explicit product-security/governance decision narrows the supported threat model and accepts the residual same-host risk. The detailed actor, asset, residual-risk, and release decision is maintained in `THREAT_MODEL.md`, and `docs/TRD.md` carries the same technical boundary. -Security findings become tests before the related runtime capability can be marked release-ready. \ No newline at end of file +Security findings become tests before the related runtime capability can be marked release-ready. diff --git a/crates/conceptweave-zotero/src/lib.rs b/crates/conceptweave-zotero/src/lib.rs index 17729a1e..9d871a0f 100644 --- a/crates/conceptweave-zotero/src/lib.rs +++ b/crates/conceptweave-zotero/src/lib.rs @@ -166,6 +166,9 @@ pub struct ClassifiedItem { pub item_type: String, /// Human-readable title retained in the local report only. pub title: String, + /// Nonempty abstract retained only when a steward must classify the item. + #[serde(skip_serializing_if = "Option::is_none")] + pub review_abstract_note: Option, /// Collection keys observed with the item. pub collection_keys: Vec, /// Tag text observed with the item. @@ -2731,11 +2734,17 @@ fn classify_item(item: &ZoteroItem, child_item_keys: Vec) -> ClassifiedI ), }; + let review_abstract_note = (proposed_disposition == Disposition::NeedsStewardReview + && !item.data.abstract_note.trim().is_empty() + && !field_values.contains_key("abstract_note")) + .then(|| item.data.abstract_note.clone()); + ClassifiedItem { item_key: item.key.clone(), item_version: item.version, item_type: item.data.item_type.clone(), title: item.data.title.clone(), + review_abstract_note, collection_keys: item.data.collections.clone(), tags: item.data.tags.clone(), proposed_disposition, diff --git a/crates/conceptweave-zotero/tests/steward_review_context.rs b/crates/conceptweave-zotero/tests/steward_review_context.rs new file mode 100644 index 00000000..73c27a94 --- /dev/null +++ b/crates/conceptweave-zotero/tests/steward_review_context.rs @@ -0,0 +1,171 @@ +use conceptweave_zotero::{ + Disposition, EvaluationError, GoldenLabel, GoldenSetApproval, ItemData, ReviewedGoldenSet, + ZoteroItem, classification_proposal_digest, classify_snapshot, evaluate_reviewed_golden_set, +}; + +fn item(key: &str, title: &str, abstract_note: &str) -> ZoteroItem { + ZoteroItem { + source_record: None, + key: key.into(), + version: 7, + data: ItemData { + item_type: "journalArticle".into(), + title: title.into(), + abstract_note: abstract_note.into(), + doi: String::new(), + parent_item: String::new(), + collections: vec![], + tags: vec![], + }, + } +} + +#[test] +fn abstentions_retain_only_the_abstract_needed_for_local_steward_review() { + let review_abstract = "A domain-specific vocabulary outside the deterministic rules."; + let report = classify_snapshot( + "9.0.6".into(), + None, + 42, + vec![ + item("REVIEW01", "Unmatched domain study", review_abstract), + item( + "DECIDED1", + "Ontology learning", + "This abstract must not be copied into review-only context.", + ), + item("EMPTY001", "Unmatched title", ""), + item("SPACE001", "Unmatched title", " \n\t "), + item("CJK00001", "Unmatched title", "地域固有の語彙を調査する。"), + item( + "CONFLICT", + "Unmatched title", + "Ontology learning and ontology matching are compared.", + ), + ], + ); + + let review = report + .classified_items + .iter() + .find(|item| item.item_key == "REVIEW01") + .unwrap(); + assert_eq!(review.proposed_disposition, Disposition::NeedsStewardReview); + assert_eq!( + review.review_abstract_note.as_deref(), + Some(review_abstract) + ); + + let decided = report + .classified_items + .iter() + .find(|item| item.item_key == "DECIDED1") + .unwrap(); + assert_eq!(decided.proposed_disposition, Disposition::Generation); + assert!(decided.review_abstract_note.is_none()); + + let empty = report + .classified_items + .iter() + .find(|item| item.item_key == "EMPTY001") + .unwrap(); + assert_eq!(empty.proposed_disposition, Disposition::NeedsStewardReview); + assert!(empty.review_abstract_note.is_none()); + + let whitespace = report + .classified_items + .iter() + .find(|item| item.item_key == "SPACE001") + .unwrap(); + assert!(whitespace.review_abstract_note.is_none()); + assert!( + serde_json::to_value(whitespace) + .unwrap() + .get("review_abstract_note") + .is_none() + ); + let non_english = report + .classified_items + .iter() + .find(|item| item.item_key == "CJK00001") + .unwrap(); + assert_eq!( + non_english.proposed_disposition, + Disposition::NeedsStewardReview + ); + assert_eq!( + non_english.review_abstract_note.as_deref(), + Some("地域固有の語彙を調査する。") + ); + + let conflict = report + .classified_items + .iter() + .find(|item| item.item_key == "CONFLICT") + .unwrap(); + assert_eq!( + conflict.proposed_disposition, + Disposition::NeedsStewardReview + ); + assert!(conflict.evidence.field_values.contains_key("abstract_note")); + assert!(conflict.review_abstract_note.is_none()); + let serialized = serde_json::to_string(conflict).unwrap(); + assert_eq!(serialized.matches("Ontology learning").count(), 1); +} + +#[test] +fn changed_review_context_invalidates_prior_approval_before_verification() { + for (original, replacement) in [ + ( + "Unmatched original abstract.", + Some("Changed review context."), + ), + ("Unmatched original abstract.", None), + ("", Some("Added review context.")), + ] { + let mut report = classify_snapshot( + "10.0.1".into(), + Some("synthetic-server".into()), + 42, + vec![item("REVIEW01", "Unmatched domain study", original)], + ); + let approved = ReviewedGoldenSet { + approval: GoldenSetApproval { + receipt_id: "synthetic-receipt".into(), + reviewer_subject: "synthetic-steward".into(), + library_version: report.library_version, + rule_revision: report.rule_revision.into(), + snapshot_digest: report.snapshot_digest.clone(), + proposal_digest: classification_proposal_digest(&report), + snapshot_items: report.snapshot_items.clone(), + }, + labels: vec![GoldenLabel::new("REVIEW01", Disposition::Generation)], + }; + let result = + evaluate_reviewed_golden_set(&report, &approved, |set| set == &approved).unwrap(); + let aggregate = serde_json::to_string(&result).unwrap(); + for private_value in ["REVIEW01", "synthetic-steward", "Unmatched domain study"] { + assert!(!aggregate.contains(private_value)); + } + if !original.is_empty() { + assert!(!aggregate.contains(original)); + } + report.classified_items[0].review_abstract_note = replacement.map(str::to_owned); + assert_ne!( + classification_proposal_digest(&report), + approved.approval.proposal_digest + ); + assert_eq!( + evaluate_reviewed_golden_set(&report, &approved, |_| { + panic!("changed review context must fail before governance") + }), + Err(EvaluationError::SnapshotMismatch) + ); + let mut rewritten = approved.clone(); + rewritten.approval.proposal_digest = classification_proposal_digest(&report); + assert_eq!( + evaluate_reviewed_golden_set(&report, &rewritten, |set| set == &approved), + Err(EvaluationError::UnverifiedApproval) + ); + } +} diff --git a/docs/PRD.md b/docs/PRD.md index febeb7f3..5b094f16 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -72,7 +72,7 @@ Preserve every observed source, including standalone files and notes outside the Library reads must finish within a bounded observation window or fail visibly without returning a partial classification. Slowly arriving pages cannot keep a run open indefinitely, and missing time budget must not be handled by silently dropping papers. -Read a complete Zotero Local API observation with one consistent library version and propose exactly one research disposition for every top-level bibliographic item. This consistency check does not establish an atomic provider snapshot. A record claiming a revision newer than the observed library invalidates the complete read; it must not be omitted or assigned a different revision to make the read pass. Each proposal retains the item key/version, exact matched metadata values, rule revision, linked child records, and any model receipt. Weak evidence and evidence that matches multiple specific disposition families must abstain into steward review. Duplicate DOI/title identities are review candidates only: intake never merges, deletes, or silently mutates Zotero records. +Read a complete Zotero Local API observation with one consistent library version and propose exactly one research disposition for every top-level bibliographic item. This consistency check does not establish an atomic provider snapshot. A record claiming a revision newer than the observed library invalidates the complete read; it must not be omitted or assigned a different revision to make the read pass. Each proposal retains the item key/version, exact matched metadata values, rule revision, linked child records, and any model receipt. Weak evidence and evidence that matches multiple specific disposition families must abstain into steward review. A local abstention retains its nonempty abstract exactly once, as matched evidence when applicable or otherwise as review context; decided items omit the review-only copy. Duplicate DOI/title identities are review candidates only: intake never merges, deletes, or silently mutates Zotero records. For every connected duplicate component, accept externally verified steward decisions selecting one component-level canonical item. Produce a local-only manifest that binds decisions to the raw snapshot, complete item revisions, exact duplicate membership, current proposals and retained source metadata. Reject missing or inconsistent source inventory and invalid decisions before requesting approval. Changed retained evidence requires fresh independent approval, even when duplicate members are unchanged. Record every component source revision plus before, after, and rollback canonical-key mappings. Classification preserves every Zotero source record. diff --git a/docs/TRD.md b/docs/TRD.md index ea43705d..83b1d44c 100644 --- a/docs/TRD.md +++ b/docs/TRD.md @@ -94,7 +94,7 @@ One monotonic five-minute budget covers page admission and complete-report accep After count/byte validation and before accumulating each metadata page, every returned object's revision must be less than or equal to that page's library revision. This includes attachments, notes and annotations, not only bibliographic records. A higher revision returns the existing snapshot-consistency error immediately, with no next-page request or partial report. Zero, lower and equal revisions remain valid and are preserved exactly, including the unsigned maximum. Compare only within this metadata read of one local instance: Zotero 9's synced revisions and Zotero 10's local revisions are not interchangeable, and this condition is not a full-text endpoint contract or proof of atomicity. See the [revision admission evidence](doctoring/zotero_item_revision.md). -Every top-level bibliographic record receives exactly one proposed disposition. `NeedsStewardReview` also records a deterministic abstention reason so missing classification metadata, vocabulary unsupported by the current deterministic rules, and present-but-unmatched metadata are distinguishable. DOI duplicate identity normalization treats bare DOI values, `doi:`, `doi.org`, and legacy `dx.doi.org` resolver forms as the same identity when their normalized DOI is equal. +Every top-level bibliographic record receives exactly one proposed disposition. `NeedsStewardReview` also records a deterministic abstention reason so missing classification metadata, vocabulary unsupported by the current deterministic rules, and present-but-unmatched metadata are distinguishable. Its nonempty abstract is retained exactly once in the local report: an abstract that triggered conflicting rules remains in matched evidence, while other abstention abstracts use the review-only field. Non-abstained items omit that field. DOI duplicate identity normalization treats bare DOI values, `doi:`, `doi.org`, and legacy `dx.doi.org` resolver forms as the same identity when their normalized DOI is equal. Duplicate decisions are independent of subject labels but share the complete source-evidence admission boundary. A reviewed set must match the raw-snapshot digest, item revisions, exact candidate membership and required v2 `proposal_digest`. Receipt comparison retains `SnapshotMismatch` precedence; shared structural/audit admission and all component/decision checks then run before the external verifier. The verifier authenticates the entire independently issued set, not a locally recomputed digest. Missing scope bindings fail deserialization and blank bindings fail admission; no legacy default or automatic reapproval exists. The manifest retains the verified proposal digest. Every operation records all component item revisions, before/after identity maps and exact rollback. Zotero records remain unchanged. @@ -107,6 +107,8 @@ Structural, source, proposal, and label checks precede the external verifier. Bl Provider deserialization captures each complete JSON object before projecting metadata. Snapshot hashing serializes the domain marker `conceptweave-zotero-snapshot-v2` followed by key-ordered pairs of that canonical source JSON and the actual typed classifier input. Unknown nested fields, array order, and omitted-versus-explicit default fields remain bound; changing a typed input after decoding also changes the digest. Synthetic offline typed items have no captured provider object and bind an explicit absent-source value alongside their typed input. Earlier reduced-content digests remain historical evidence and cannot establish this complete-content contract; regenerate the report and review artifacts and obtain fresh approval before any release or approved write. A successful classification report carries an `audit_summary` whose snapshot, bibliographic, proposed-disposition, provenance-complete, abstention, duplicate-candidate, failure, and per-disposition counts are derived from the same in-memory immutable snapshot. Zotero item version zero remains a valid observed coordinate for never-synced Zotero 9 records; provenance completeness rejects a missing item key rather than inventing a positive-only version invariant. Reader failures return an error instead of a partial report; therefore a returned report records `failure_count=0` rather than hiding partial failures. +The local report can contain titles, tags, matched metadata, and abstention abstracts. It is sensitive steward-review material, remains outside the repository, and is not a publication artifact. + The report is local JSON and contains proposals rather than governance decisions. CLI output is restricted to a new direct child of canonical `/tmp` or the operating system temporary directory; relative paths, nested paths, existing paths, and symlinks are rejected, and create-new file semantics prevent overwrite/path-swap writes. Reviewed collection/tag changes can produce a pure local plan whose default mode is dry-run. The plan requires exact report and item preconditions, complete before/after/rollback arrays, externally verified authority, and preserved Zotero tag types; its fields are externally read-only after validation. Zotero 9 execute mode fails closed. Every receipt copies the plan's review, authority, server, Zotero version, library, rule, snapshot and proposal coordinates; dry-run reports every operation as not attempted and makes no Local API call. Execute mode preflights every item before the first write, advances the library precondition only from a directly verified write response, stops on the first adapter or response failure, and re-reads that item through the same boundary as observation only. Failed writes remain indeterminate regardless of observed metadata; no inverse is issued for them. Prior directly verified operations retain their inverse coordinates. The API key remains adapter-owned and absent from serializable structures. PR #20 retains its rollback core and adapter: mixed-server rejection precedes reads; complete current-state checks precede inverse writes; only directly verified responses advance the library version. Every failed or invalid inverse response now remains indeterminate, retaining the complete operation, exact submitted request (including its library precondition), and optional complete readback. Matching restored or unchanged metadata does not prove causal completion or termination, and the failed inverse is absent from remaining work. Earlier directly verified restorations remain recorded; remaining operations are untouched only, not automatic retry authority. The operation-slice API still lacks original-write scope and independent authority; authoritative consumer adoption remains an open gate, including empty-slice and delayed-reconciliation handling. diff --git a/docs/adr/0006-zotero-research-intake.md b/docs/adr/0006-zotero-research-intake.md index 22584a05..b27b43ee 100644 --- a/docs/adr/0006-zotero-research-intake.md +++ b/docs/adr/0006-zotero-research-intake.md @@ -17,7 +17,7 @@ ConceptWeave owns a small read-only Anti-Corruption Layer from Zotero into resea The adapter links child records, emits exactly one deterministic proposed disposition per top-level bibliographic item, and abstains when evidence is weak or ambiguous. Every abstention preserves a deterministic reason distinguishing missing classification metadata, vocabulary outside the current deterministic rules, present-but-unmatched metadata, and conflicting specific disposition families. Specific rule families are evaluated together rather than by first-match priority. When evidence matches multiple families, the proposal becomes `NeedsStewardReview` and all matching evidence is retained. -Matched metadata values are copied into the local-only evidence receipt for replay. This is necessary for abstract-only matches because a later Zotero revision cannot reconstruct the exact text used for an earlier proposal from item key/version alone. DOI/title matches remain reversible duplicate candidates, including legacy `dx.doi.org` resolver forms. +Matched metadata values are copied into the local-only evidence receipt for replay. This is necessary for abstract-only matches because a later Zotero revision cannot reconstruct the exact text used for an earlier proposal from item key/version alone. An abstention likewise retains its nonempty abstract so a steward can resolve unsupported or unmatched vocabulary from the same immutable report. If matched evidence already contains the abstract, the review-only field is omitted so sensitive text appears once; decided items also omit that extra copy. The report remains sensitive local material. DOI/title matches remain reversible duplicate candidates, including legacy `dx.doi.org` resolver forms. Duplicate candidates become canonical references only through externally verified steward decisions bound to the raw digest, complete item-key/item-version snapshot, and exact candidate membership. Overlapping candidates form one connected component and must select one component-level canonical item. Every resulting operation retains all component source revisions and complete before/after/rollback key mappings. It changes downstream identity resolution only; classification does not merge, delete, or mutate Zotero source records. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 205fddf8..a2778345 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -89,6 +89,40 @@ Remaining work: mandatory adoption by restoration, worksheet, duplicate and writ ## DDD fitness constraints +### PR #22 steward-context binding verification (2026-09-06) + +Untouched `7179d13b45d160682e4cce1473c145d465fe657b` passed 120 tests/23 suites. +Normal merge `49daf4c` retains that delta and parent PR #21 `425d8df`; integrated +tests passed 143/23. Documentation conflicts preserve both private abstract +minimization and corrected source-version/uncertainty contracts. The original +PR #22 production implementation is unchanged: an abstention retains a nonblank +abstract only when it is not already in matched evidence; decided items omit it. + +Independent review identified missing binding regression evidence, not a runtime +defect. Test `ebcf2a3` covers insertion, replacement and removal of review context: +each changes the inherited v2 proposal digest and rejects the original approval +before contacting governance. `b44d603` also rejects locally rewritten digests +against the independent original receipt, preserves whitespace omission and +verbatim Japanese context, and retains all four original minimization cases. +Evaluation aggregates exclude synthetic item/reviewer/title/abstract values. +This is context retention, not multilingual classification or real approval. + +Final 144 tests/23 suites including three doctests, strict Clippy, warnings-denied +rustdoc, format/CI-contract/diff and unchanged coverage pass. Coverage is 269/269 +functions, 2369/2369 normalized regions, 404/404 normalized branches; raw LLVM is +3083/3143 lines, 4625/4722 regions, 360/404 branches, not 100%. Logs use +`/tmp/conceptweave-pr22-scope-` with `baseline.log`, `integration.log`, +`complete.log`, `clippy-complete.log`, `rustdoc.log`, `coverage-complete.log`. +Independent final review found no remaining issue in the reviewed delta. + +Sensitive report context stays private; no fresh real-data capture or visual +evidence was collected. Latest native visual attempt encountered the locked Mac; +historical 3,719 displayed items are not classification evidence. Real decisions +and approvals remain 0/3,715 plus four unresolved sources. No real mutation, +authorization, recovery, protected merge or release occurred. Next successor is +PR #23 owner-only report permissions `2a3619f52e1d3e4f699c91be1fc2d0e9a6e234c8`; +root and authoritative full-text envelope adoption remain outstanding. + ### PR #21 delayed observation authority repair (2026-09-06) Untouched `09c84e4cdb1393a5e450f5200b87f292eeea956f` passed 119 tests/22 suites.