Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0027d7d
test: require capture-bound private full-text review context
seonghobae Sep 5, 2026
e426c2b
feat: build bounded capture-bound full-text review views
seonghobae Sep 5, 2026
3fb930b
test: preserve full-text review selection and resource boundaries
seonghobae Sep 5, 2026
fcba236
test: verify distinct text parents and source version spaces
seonghobae Sep 5, 2026
25a1005
test(zotero): reproduce missing private full-text review CLI
seonghobae Sep 5, 2026
db1d2b9
docs: define private full-text review view and approval boundary
seonghobae Sep 5, 2026
045031c
feat(zotero): add bounded private full-text review CLI
seonghobae Sep 5, 2026
ed308bd
test(zotero): keep streaming reader checks warning-free
seonghobae Sep 5, 2026
b86fe86
merge: integrate bounded private full-text review CLI
seonghobae Sep 5, 2026
26a3e6a
test(zotero): verify streaming read ceiling without unreachable arm
seonghobae Sep 5, 2026
54383ea
merge: retain bounded reader coverage regression
seonghobae Sep 5, 2026
a1b4939
docs: record offline full-text review evidence and live review gates
seonghobae Sep 5, 2026
6b9f2ef
docs: distinguish review rows and confirmed owner scope
seonghobae Sep 5, 2026
bdeaa19
docs: link private review view successor PR
seonghobae Sep 5, 2026
a41306a
Merge verified private artifact boundary repair into PR 37
seonghobae Sep 5, 2026
692cb58
merge(zotero): propagate validated approval ordering through PR 37
seonghobae Sep 5, 2026
2688b50
merge(research): inherit bounded metadata reads into PR #37
seonghobae Sep 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ ConceptWeave owns automatic, evidence-bound **Semantic Model Engineering**. Do n
- New database objects, when introduced, use descriptive two-or-more-word `snake_case` names and 3NF by default.
- Preserve source evidence, truth status, and publication state separately.
- Keep Zotero full-text captures separate from metadata reports and approval receipts; restored captures require bounded verification, and local HTTP continuity is not peer authentication.
- Full-text review views are read-only evidence; never strip their outer binding into a metadata-only patch and claim full-text decision or approval provenance.
- Published semantic truth is immutable; correction uses supersession/new release.
- Public Rust APIs require beginner-readable documentation.
- Owned production coverage target is 100% line/function/region/branch where tooling exposes it.
Expand Down
2 changes: 2 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ flowchart LR

Research Intake's Zotero adapter retains optional full-text observations in a separate private artifact bound to the original metadata report. It remains inside ConceptWeave: acquisition is supporting evidence work, not a publication authority or another research system of record. Provider version counters remain opaque at this Anti-Corruption Layer; downstream classification and review must explicitly adopt new content under fresh evidence bindings. See [ADR 0006](docs/adr/0006-zotero-research-intake.md).

Its Full-Text Review View reuses capture verification and canonical pending selection to make exact text inspectable without changing earlier proposals. This bounded read projection is not an aggregate or a decision-application API; the metadata-only approval chain does not acquire full-text provenance from it.

| Context | Type | Owns | Does not own |
| --- | --- | --- | --- |
| Source Observation | Supporting | immutable observations, parser receipts, evidence locations | source-system business truth |
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ All notable changes to ConceptWeave are documented here.

### Added

- Private inspection of pending papers alongside their saved text, preserving missing material and leaving previous reports and decisions unchanged.

- Private, replayable paper-text capture for later research review, preserving unavailable material and leaving earlier reports and approvals unchanged.

- Full-library research-source audit separating available text, incomplete indexing and missing material from reviewed classification; no paper is excluded because its abstract or text is unavailable.
Expand Down
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ ConceptWeave's core invariant is: **inference is not authority**. Every generate
Keep domain logic in bounded domain modules, LLM/provider logic behind ports/adapters, and source/consumer systems independent. Prefer deterministic validation and explicit abstention over plausible unsupported output.

Zotero source capture must not alter the metadata report or renew its approval. Preserve private-file protections and the full bibliographic denominator, including missing and partial text.

The separate full-text review view does not authorize decisions or writes. Keep its evidence binding intact; existing metadata-only apply/finalization cannot establish full-text-reviewed approval.
2 changes: 2 additions & 0 deletions THREAT_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,4 +75,6 @@ PR #30 commit `9733d28` reproduced the inode-preserving final-component symlink

## Release gate

The optional full-text review view must not be confused with the metadata-only decision or approval artifacts. A required versioned outer envelope and strict legacy parsers prevent silent direct application; an intentional caller can still strip fields, so no full-text-reviewed approval is claimed. The complete capture is reverified before selecting text, unrelated parent text is excluded, and output serialization itself is bounded. The larger serialized-capture input ceiling is separate from the unchanged report/worksheet limit. Source text is untrusted data, never instructions, and source content must not be included in input-error diagnostics.

A capability is not release-ready while a valid security finding lacks a deterministic test or equivalent machine-verifiable contract, while required exact-head checks are non-terminal, or while the implemented transport cannot satisfy the advertised security claim. Documentation must describe residual risk without upgrading provider guarantees by inference.
78 changes: 78 additions & 0 deletions crates/conceptweave-zotero/src/full_text_capture.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,84 @@ impl fmt::Display for FullTextError {

impl std::error::Error for FullTextError {}

/// Builds a bounded private evidence view for the next pending review rows.
///
/// This verifies the complete capture against the unchanged report, then selects
/// the same 1–100 pending rows as the metadata review command. Each selected paper
/// keeps an attachment list, even when no text was captured. Exact response bodies
/// preserve missing, empty and partially indexed content without interpreting it.
/// The serialized output is limited to 16 MiB, including JSON escaping; an
/// oversized view fails without truncation. Callers must separately bound private
/// capture-file deserialization before passing the restored capture here.
///
/// The returned JSON is not a decision patch or an approval. It must not replace
/// the original report or worksheet, and it is rejected by legacy apply commands.
pub fn build_full_text_review_json(
report: &ClassificationReport,
worksheet: &crate::StewardReviewWorksheet,
capture: &FullTextCapture,
limit: usize,
) -> Result<Vec<u8>, FullTextError> {
#[derive(Serialize)]
struct AttachmentEvidence<'a> {
item_key: &'a str,
metadata_version: Option<u64>,
content_response: &'a CapturedResponse,
}
#[derive(Serialize)]
struct ReviewView<'a> {
view_kind: &'static str,
capture_digest: &'a str,
metadata_report_digest: &'a str,
proposal_digest: String,
bibliographic_item_count: usize,
review_batch: crate::StewardReviewBatch,
attachment_evidence: BTreeMap<String, Vec<AttachmentEvidence<'a>>>,
}

let review_batch = crate::build_steward_review_batch(report, worksheet, limit)
.map_err(|_| FullTextError("full-text review requires a valid pending batch"))?;
verify_full_text_capture(capture, report)?;
let parent_by_key: BTreeMap<_, _> = report
.snapshot_items
.iter()
.map(|item| (item.item_key.as_str(), item.parent_item_key.as_deref()))
.collect();
let mut attachment_evidence: BTreeMap<_, Vec<_>> = review_batch
.decisions
.iter()
.map(|decision| (decision.item_key.clone(), Vec::new()))
.collect();
for record in &capture.capture_evidence.records {
if let Some(rows) = parent_by_key[record.item_key.as_str()]
.and_then(|parent| attachment_evidence.get_mut(parent))
{
rows.push(AttachmentEvidence {
item_key: &record.item_key,
metadata_version: record.metadata_response.version,
content_response: &record.content_response,
});
}
}
let view = ReviewView {
view_kind: "full_text_review_view_v1",
capture_digest: &capture.capture_digest,
metadata_report_digest: &capture.capture_evidence.metadata_report_digest,
proposal_digest: crate::classification_proposal_digest(report),
bibliographic_item_count: report.classified_items.len(),
review_batch,
attachment_evidence,
};
// A fixed slice bounds serialization itself, including JSON escaping.
let mut bytes = vec![0; 16 * 1024 * 1024];
let mut writer = std::io::Cursor::new(bytes.as_mut_slice());
serde_json::to_writer(&mut writer, &view)
.map_err(|_| FullTextError("full-text review exceeds the 16 MiB output limit"))?;
let length = writer.position() as usize;
bytes.truncate(length);
Ok(bytes)
}

/// Reads every full-text manifest entry through the fixed loopback API.
///
/// The report remains unchanged. Missing content is retained explicitly, while
Expand Down
267 changes: 267 additions & 0 deletions crates/conceptweave-zotero/src/full_text_capture_tests.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,273 @@
use super::*;
use crate::{ZoteroItem, classify_snapshot};

#[test]
fn review_view_binds_exact_capture_and_keeps_missing_parents_without_decisions() {
let report = report_fixture();
let worksheet = build_steward_review_worksheet(&report).unwrap();
let before_report = serde_json::to_vec(&report).unwrap();
let before_worksheet = worksheet.clone();
let capture = capture_with(&report, 4096, &mut |request_path, _| {
Ok(response_fixture(request_path))
})
.unwrap();
let bytes = build_full_text_review_json(&report, &worksheet, &capture, 2).unwrap();
let view: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_eq!(view["view_kind"], "full_text_review_view_v1");
assert_eq!(view["capture_digest"], capture.capture_digest);
assert_eq!(
view["proposal_digest"],
crate::classification_proposal_digest(&report)
);
assert_eq!(
view["metadata_report_digest"],
capture.capture_evidence.metadata_report_digest
);
assert_eq!(view["bibliographic_item_count"], 2);
assert_eq!(view["review_batch"]["remaining_count"], 2);
assert_eq!(
view["review_batch"]["decisions"].as_array().unwrap().len(),
2
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"]
.as_array()
.unwrap()
.len(),
2
);
assert_eq!(
view["attachment_evidence"]["DEFG5678"],
serde_json::json!([])
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"][0]["item_key"],
"BCDE3456"
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"][0]["metadata_version"],
1
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"][0]["content_response"]["version"],
12403
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"][1]["metadata_version"],
0
);
assert!(view["attachment_evidence"]["ABCD2345"][1]["content_response"]["version"].is_null());
assert_eq!(
view["attachment_evidence"]["ABCD2345"][0]["content_response"]["body"],
r#"{"content":"fixture text 한글","indexedPages":2,"totalPages":2,"providerExtra":{"retained":true}}"#
);
assert_eq!(
view["attachment_evidence"]["ABCD2345"][1]["content_response"]["status"],
404
);
assert!(
view["review_batch"]["decisions"]
.as_array()
.unwrap()
.iter()
.all(|row| row["reviewed_disposition"].is_null())
);
assert!(serde_json::from_slice::<crate::StewardReviewBatch>(&bytes).is_err());
assert!(serde_json::from_slice::<crate::StewardDecisionPatch>(&bytes).is_err());
assert_eq!(serde_json::to_vec(&report).unwrap(), before_report);
assert_eq!(worksheet, before_worksheet);
assert_eq!(
build_full_text_review_json(&report, &worksheet, &capture, 2).unwrap(),
bytes
);
}

#[test]
fn review_view_selects_only_pending_parents_and_never_copies_unrelated_text() {
let report = report_fixture();
let mut worksheet = build_steward_review_worksheet(&report).unwrap();
let capture = capture_with(&report, 4096, &mut |request_path, _| {
Ok(response_fixture(request_path))
})
.unwrap();
let first: serde_json::Value = serde_json::from_slice(
&build_full_text_review_json(&report, &worksheet, &capture, 1).unwrap(),
)
.unwrap();
assert_eq!(
first["review_batch"]["decisions"].as_array().unwrap().len(),
1
);
assert!(first["attachment_evidence"].get("DEFG5678").is_none());
worksheet.decisions[0].reviewed_disposition = Some(crate::Disposition::OutOfScope);
let next = build_full_text_review_json(&report, &worksheet, &capture, 2).unwrap();
let view: serde_json::Value = serde_json::from_slice(&next).unwrap();
assert_eq!(view["review_batch"]["remaining_count"], 1);
assert_eq!(view["bibliographic_item_count"], 2);
assert_eq!(
view["attachment_evidence"],
serde_json::json!({"DEFG5678":[]})
);
assert!(!String::from_utf8(next).unwrap().contains("fixture text"));
}

#[test]
fn review_view_separates_two_text_parents_and_excludes_standalone_attachments() {
let source: Vec<ZoteroItem> = serde_json::from_value(serde_json::json!([
{"key":"ABCD2345","version":2,"data":{"itemType":"book","title":"first paper"}},
{"key":"BCDE3456","version":1,"data":{"itemType":"attachment","parentItem":"ABCD2345"}},
{"key":"DEFG5678","version":2,"data":{"itemType":"book","title":"second paper"}},
{"key":"EFGH6789","version":1,"data":{"itemType":"attachment","parentItem":"DEFG5678"}},
{"key":"FGHI789A","version":1,"data":{"itemType":"attachment"}}
]))
.unwrap();
let mut report = classify_snapshot("10.0.1".into(), Some("fixture-server".into()), 2, source);
report.api_version = Some(3);
report.schema_version = Some(44);
let mut worksheet = build_steward_review_worksheet(&report).unwrap();
let capture = capture_with(&report, 4096, &mut |request_path, _| {
Ok(match request_path {
"fulltext?since=0" => CapturedResponse {status:200,version:None,
body:r#"{"BCDE3456":12403,"EFGH6789":9,"FGHI789A":0}"#.into()},
"items/EFGH6789" => CapturedResponse {status:200,version:Some(1),
body:r#"{"key":"EFGH6789","version":1,"data":{"itemType":"attachment","parentItem":"DEFG5678"}}"#.into()},
"items/EFGH6789/fulltext" => CapturedResponse {status:200,version:Some(9),
body:r#"{"content":"second parent evidence"}"#.into()},
"items/FGHI789A" => CapturedResponse {status:200,version:Some(1),
body:r#"{"key":"FGHI789A","version":1,"data":{"itemType":"attachment"}}"#.into()},
"items/FGHI789A/fulltext" => CapturedResponse {status:200,version:Some(0),
body:r#"{"content":"standalone evidence"}"#.into()},
_ => response_fixture(request_path),
})
}).unwrap();
let both: serde_json::Value = serde_json::from_slice(
&build_full_text_review_json(&report, &worksheet, &capture, 2).unwrap(),
)
.unwrap();
assert_eq!(both["bibliographic_item_count"], 2);
assert_eq!(
both["attachment_evidence"]["ABCD2345"]
.as_array()
.unwrap()
.len(),
1
);
assert_eq!(
both["attachment_evidence"]["DEFG5678"]
.as_array()
.unwrap()
.len(),
1
);
assert_eq!(
both["attachment_evidence"]["DEFG5678"][0]["item_key"],
"EFGH6789"
);
assert_eq!(
both["attachment_evidence"]["DEFG5678"][0]["content_response"]["version"],
9
);
let first =
String::from_utf8(build_full_text_review_json(&report, &worksheet, &capture, 1).unwrap())
.unwrap();
assert!(first.contains("fixture text"));
assert!(!first.contains("second parent evidence"));
assert!(!first.contains("standalone evidence"));
worksheet.decisions[0].reviewed_disposition = Some(crate::Disposition::OutOfScope);
let second =
String::from_utf8(build_full_text_review_json(&report, &worksheet, &capture, 1).unwrap())
.unwrap();
assert!(second.contains("second parent evidence"));
assert!(!second.contains("fixture text"));
assert!(!second.contains("standalone evidence"));
}

#[test]
fn review_view_rejects_changed_capture_report_and_invalid_pending_work() {
let report = report_fixture();
let worksheet = build_steward_review_worksheet(&report).unwrap();
let capture = capture_with(&report, 4096, &mut |request_path, _| {
Ok(response_fixture(request_path))
})
.unwrap();
for field in ["body", "status", "version"] {
let mut saved = serde_json::to_value(&capture).unwrap();
saved["capture_evidence"]["records"][0]["content_response"][field] = if field == "body" {
serde_json::json!("private changed text")
} else {
serde_json::json!(999)
};
let changed: FullTextCapture = serde_json::from_value(saved).unwrap();
assert!(build_full_text_review_json(&report, &worksheet, &changed, 2).is_err());
}
let mut changed_report = report_fixture();
changed_report.classified_items[0].title = "different report evidence".into();
assert!(build_full_text_review_json(&changed_report, &worksheet, &capture, 2).is_err());
for limit in [0, 101] {
assert!(build_full_text_review_json(&report, &worksheet, &capture, limit).is_err());
}
let mut invalid = worksheet.clone();
invalid.decisions[0].reviewed_disposition = Some(crate::Disposition::NeedsStewardReview);
assert!(build_full_text_review_json(&report, &invalid, &capture, 2).is_err());
for decision in &mut invalid.decisions {
decision.reviewed_disposition = Some(crate::Disposition::OutOfScope);
}
assert!(build_full_text_review_json(&report, &invalid, &capture, 2).is_err());
}

#[test]
fn review_view_retains_empty_partial_and_unknown_provider_evidence() {
for body in [
r#"{"content":""}"#,
r#"{"content":"partial text","indexedPages":1,"totalPages":2}"#,
r#"{"content":"unknown completeness","providerExtra":{"original":true}}"#,
] {
let report = report_fixture();
let worksheet = build_steward_review_worksheet(&report).unwrap();
let capture = capture_with(&report, 4096, &mut |request_path, _| {
let mut response = response_fixture(request_path);
if request_path == "items/BCDE3456/fulltext" {
response.body = body.into();
}
Ok(response)
})
.unwrap();
let view: serde_json::Value = serde_json::from_slice(
&build_full_text_review_json(&report, &worksheet, &capture, 2).unwrap(),
)
.unwrap();
assert_eq!(
view["attachment_evidence"]["ABCD2345"][0]["content_response"]["body"],
body
);
assert_eq!(view["bibliographic_item_count"], 2);
}
}

#[test]
fn review_view_rejects_json_expansion_without_truncating_valid_captured_text() {
let report = report_fixture();
let worksheet = build_steward_review_worksheet(&report).unwrap();
let capture = capture_with(&report, MAX_SNAPSHOT_BYTES, &mut |request_path, _| {
let mut response = response_fixture(request_path);
if request_path.ends_with("/fulltext") {
response.status = 200;
response.version = Some(if request_path.contains("BCDE3456") {
12403
} else {
0
});
response.body = serde_json::json!({"content":"\"".repeat(3 * 1024 * 1024)}).to_string();
}
Ok(response)
})
.unwrap();
verify_full_text_capture(&capture, &report).unwrap();
let error = build_full_text_review_json(&report, &worksheet, &capture, 1).unwrap_err();
assert!(error.to_string().contains("16 MiB output limit"));
assert_eq!(worksheet.decisions.len(), 2);
}

fn report_fixture() -> ClassificationReport {
let items: Vec<ZoteroItem> = serde_json::from_value(serde_json::json!([
{"key":"ABCD2345","version":2,"data":{"itemType":"journalArticle","title":"fixture paper"}},
Expand Down
Loading