Add Zotero integration (import, sync, add)

[katosh]

Summary

Adds full Zotero support to bip. Zotero is a free, open-source reference manager widely used in academia — the open-source counterpart to Paperpile. It stores papers locally with optional cloud sync, provides browser-based paper capture, and exposes both a Web API and a local SQLite database for programmatic access.

Three access methods

• bip import --format zotero export.json — import from Zotero's CSL-JSON export format, with optional --zotero-db flag for PDF path resolution from Zotero's local SQLite database
• bip zotero sync — pull all items from Zotero's Web API into bip, with skip-if-unchanged deduplication
• bip zotero add DOI:10.xxx — add paper to both bip and Zotero simultaneously; fetches metadata from S2 with automatic CrossRef fallback when rate-limited

How this differs from the Paperpile integration

The Paperpile and Zotero backends take different approaches due to the different capabilities each tool exposes.

Paperpile embeds PDF attachment paths directly in its JSON export, so bip import --format paperpile gets both metadata and file locations in one step. However, this requires a manual export-download-import cycle each time the library changes.

Zotero's CSL-JSON export does not include file paths, since Zotero manages PDFs in its own storage directory (~/Zotero/storage/<key>/<file>.pdf). To bridge this gap, the --zotero-db flag reads Zotero's local SQLite database to resolve PDF paths as a separate enrichment step after parsing.

Beyond the file import, Zotero's Web API enables live sync (bip zotero sync) and bidirectional writes (bip zotero add), which aren't available in the current Paperpile integration because it relies on file exports rather than an API. The same API-based approach could be added for Paperpile in the future.

Both importers produce identical Reference structs and feed into the same deduplication pipeline — only Source.Type ("paperpile" vs "zotero") differs. A mixed library with papers from both sources works without issues.

New packages/files

• internal/importer/zotero.go — CSL-JSON parser (ParseZotero), follows ParsePaperpile pattern exactly
• internal/importer/zotero_pdf.go — PDF path resolution from Zotero's SQLite DB (~/Zotero/zotero.sqlite)
• internal/zotero/ — Web API v3 client (rate-limited, paginated), bidirectional mapper (Reference ↔ ZoteroItemData), CrossRef DOI resolver, error types
• testdata/zotero_csl_sample.json — 5-entry test fixture

Modified

• cmd/bip/import.go — --format zotero dispatch + --zotero-db flag
• internal/config/global.go — zotero_api_key, zotero_user_id config fields
• skills/bip.lit.import/SKILL.md, skills/bip.lit/SKILL.md — Zotero workflow docs

Config

# ~/.config/bip/config.yml
zotero_api_key: your-key   # from https://www.zotero.org/settings/keys
zotero_user_id: "12345"    # numeric ID shown on the same page

For Zotero users, set pdf_root: ~/Zotero/storage in .bipartite/config.yml since Zotero stores PDFs as ~/Zotero/storage/<8-char-key>/<filename>.pdf.

Test plan

• go test ./... — all 28 packages pass
• go vet ./... — clean
• 12 unit tests for CSL-JSON parser (valid entries, missing fields, partial dates, institutional authors, real fixture)
• 14 unit tests for Zotero API package (mapper round-trip, date parsing, Extra field extraction, error types)
• End-to-end: synced 100 papers from a real Zotero library
• Re-sync correctly skips unchanged refs (103 skipped, 3 actual updates)
• bip zotero add DOI:... — added to both bip and Zotero via CrossRef fallback
• Existing Paperpile import and all other tests unaffected

[matsen]

Very nice! Zotero has an MCP, is this better than just using that?

[matsen]

🤖 Pushed a fixup commit (94160b9) that addresses:

• CI lint fix: gofmt on all new files (alignment in struct tags, etc.)
• Exit code cleanup: Replaced ExitZoteroNotConfigured/APIError/Duplicate (1/2/3) with the canonical ExitConfigError/ExitError/ExitDataError from exitcodes.go — avoids shadowing the shared constants
• Broader duplicate check: bip zotero add now checks by ID in addition to DOI before adding
• HTML entity handling: stripHTMLTags now calls html.UnescapeString so CrossRef abstracts with & etc. render correctly
• Rate limiter burst: Increased from 1 to 3 for smoother request flow during paginated fetches

All tests pass (go test ./..., go vet ./..., gofmt clean). Please check that this still works with your Zotero library.

[katosh]

Thanks! I found since this is an interaction of the bip tool with Zotero instead of a direct interaction between the agent and Zotero, the classic API seemed right. Would an additional MCP interface for the agent also help?

[matsen]

I guess I'm just asking, if you're using Zotero, why would you need to use our tool at all and why not just use their MCP using your agent?

[katosh]

Good point. I assumed that using the bip interface is part of the agents workflow and might be less distracting for it than interacting with an MCP server. Is the agent actually instructed to use bip and if so, do you think it might be better to just replace the instructions when working with Zotero?

I am really just starting to understand this project would totally understand if this PR does not make much sense!

[matsen]

Well, I would love it if you would try this out and show that Zotero works using this package. I was just curious.

[katosh]

Note: This comment was generated by an AI agent (Claude Code) at @katosh's request, summarizing an exploration of replacing bip's paper management with a Zotero MCP server.

Zotero MCP Server as an Alternative to bip's Paper Management

We tested the 54yyyu/zotero-mcp server (~1.6k stars, actively maintained) as a direct MCP integration for Zotero, bypassing bip entirely for paper lookups. The server connects via Zotero's Web API and exposes ~35 tools: search, metadata retrieval, annotation extraction, collection management, tagging, citation export, semantic search, and even retraction checking via scite.ai.

What Would Need to Change

Skills (high effort):

• bip.lit/ — The core literature skill would need to be rewritten to call MCP tools (zotero_search_items, zotero_get_item_metadata, etc.) instead of bip search, bip get, bip s2 add
• bip.lit.import/ — Becomes unnecessary; Zotero handles imports natively through its desktop app and browser connector
• bip.lit.extract/ and bip.lit.edges/ — The knowledge graph (concepts, edges) has no Zotero equivalent. These would need a separate persistence layer or be reimplemented as Zotero tags/collections (lossy)

CLAUDE.md instructions:

• The "Paper Lookups (nexus)" section assumes grep on .bipartite/refs.jsonl as the first step. This would become zotero_search_items MCP calls
• The "always search locally first" policy becomes "always search Zotero first" (still valid, just different target)
• Database location and rebuild instructions become irrelevant

CLI commands (medium effort):

• bip search → zotero_search_items / zotero_advanced_search / zotero_semantic_search
• bip get → zotero_get_item_metadata + zotero_get_annotations
• bip s2 add → zotero_add_by_doi (Zotero resolves metadata itself via translators)
• bip s2 citations/references → No Zotero equivalent; would still need S2/ASTA for citation graph traversal
• bip rebuild → Unnecessary (Zotero is always live)
• bip export --bibtex → zotero_search_items returns citation keys; Zotero's own export is richer
• bip concept/edge → No equivalent in Zotero MCP (see cons below)

Pros of Zotero MCP

1. PDF annotation extraction — Pull highlights, notes, and margin comments directly from Zotero-managed PDFs. This is a capability bip doesn't have at all, and it's valuable for literature review workflows.
2. No rebuild step — Zotero is always live. No JSONL→SQLite cache invalidation, no bip rebuild after imports.
3. Richer import ecosystem — Zotero's browser connector, desktop app, and translator framework handle import from virtually any source. No need for format-specific parsers like ParsePaperpile or ParseZotero.
4. Collection/tag management — Hierarchical collections and tags managed through the same MCP interface.
5. Semantic search built in — The MCP server includes embedding-based search out of the box.
6. One fewer binary to build — No go build step for paper operations; the MCP server is uvx-installable.
7. Retraction checking — Built-in scite.ai integration for checking paper retractions.

Cons of Zotero MCP

1. Rate limiting is real — Zotero's free API has aggressive rate limits. During our testing, repeated auth failures with an invalid key triggered a prolonged lockout (~20 minutes). Normal usage with a valid key should be fine for typical library sizes, but the MCP server's startup indexing and semantic search features add API pressure.
2. No knowledge graph — bip's concept/edge system (linking papers to concepts with typed relationships and summaries) has no Zotero equivalent. Zotero tags are flat strings, not a graph. This is the biggest architectural gap — the edges table is central to bip's value proposition as a bipartite graph tool.
3. No citation graph traversal — Zotero doesn't know about citation networks. S2/ASTA would still be needed for citations, references, and gaps commands. So bip can't be fully replaced.
4. No offline/local-first — Zotero MCP goes through the cloud API (local API requires Zotero 7 desktop running, not feasible on HPC). Every search is a network round-trip. bip's SQLite is instant.
5. No JSONL portability — bip's JSONL is diffable, greppable, and version-controllable. Zotero's storage is opaque from the outside.
6. Dependency on external service — Zotero's API availability becomes a single point of failure. The MCP server itself adds another dependency layer.

Recommendation

The Zotero MCP server is a valuable complement to bip, not a replacement. The sweet spot is:

• Use Zotero MCP for: annotation extraction, browsing the reading collection, and adding papers via DOI
• Use bip for: the knowledge graph (concepts/edges), citation traversal (S2/ASTA), local-first search, and as the durable source of truth
• Use this PR's sync feature for: bridging the two — pulling Zotero papers into bip so they join the knowledge graph

The Zotero MCP's annotation extraction is the most compelling unique capability. Being able to ask "what did I highlight in this paper?" directly in a Claude conversation is genuinely useful and not something bip provides today.

[matsen]

Thanks, Dominik! I encourage you to do with this PR what you like-- I would love it if you incorporated anything that you actually use so that it's dog-fooded. I can't do this work because I don't use Zotero.