diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index f6edeac6..8550bb01 100644 --- a/.github/workflows/docs.yaml +++ b/.github/workflows/docs.yaml @@ -43,6 +43,8 @@ jobs: - name: Build docs shell: bash -el {0} id: build + env: + PLINDER_SCORE_EXAMPLE_DATA_DIR: ${{ github.workspace }}/tests/test_data/docs_score_release run: sphinx-build docs build/docs - name: Upload build artifact uses: actions/upload-pages-artifact@v3 diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 7657eeb0..55f39ff1 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -96,6 +96,8 @@ jobs: - name: Install plinder run: UV_PROJECT_ENVIRONMENT="$CONDA_PREFIX" uv sync --inexact --extra docs - name: Build the documentation + env: + PLINDER_SCORE_EXAMPLE_DATA_DIR: ${{ github.workspace }}/tests/test_data/docs_score_release run: sphinx-build --fail-on-warning docs build/docs - name: Show notebook errors if: failure() diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..701a5f31 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,32 @@ +# Changelog + +- WIP (Current — unreleased): + - **Public downloads from Cameo**: releases download directly from `https://cameo3d.org/plinder/PLINDER-/` (currently `2026-09`); `PLINDER_MIRROR_URL` selects another HTTP file server with the same directory layout. Downloads stream to disk, validate HTTP Content-Length, and replace cached files atomically. Interrupted transfers retry from the start. Cached files are reused when their size and Last-Modified time match the server, so data hotfixes are fetched on the next access. Google storage dependencies are optional: install `plinder[data]` for the GCS data-generation utilities. + - **Major backend refactor**: replaced OST, gemmi, plip, openbabel with biotite + peppr for data generation; removed 6 dependencies from ingest pipeline + - **Nucleic acid support**: DNA/RNA chains now correctly included as receptor neighbors, mainchain/sidechain detection works for both protein and nucleic acids ([#61](https://github.com/plinder-org/plinder/issues/61)) + - **Custom CIF support**: new `Entry.from_custom_cif_file` for structure-prediction outputs (Boltz, AlphaFold3, Chai-1) that ship CIFs without `_chem_comp_bond` ([#117](https://github.com/plinder-org/plinder/issues/117)). Bond orders come from `ligand_smiles_dict` via positional atom-order match (the convention these tools follow); element/count mismatches raise with the offending position, `force_substructure_match=True` opts into substructure matching when atom order isn't preserved. User SMILES win over CCD for both `smiles` and `resolved_stereo_matches_template` — closes a silent gap where biotite's `LIG` placeholder would pass any 3D conformer. Input CIFs are never mutated; optional `save_fixed_cif` persists the enriched copy. + - **Stricter CIF ingest**: H/D filtered consistently (biotite's `filter_heavy`); multi-model CIFs warn and use model 1; multi-instance custom comp_ids must share heavy-atom naming (since `_chem_comp_bond` is comp_id-keyed); silent `connect_via_residue_names` and half-sanitized substructure fallbacks replaced with `ValueError` so corrupt inputs fail loudly. + - **Stereochemistry**: CCD ideal 3D coordinates used as stereo ground truth; new `resolved_stereo_matches_template` flag validates resolved structure chirality against CCD template (handles partial resolution via MCS trimming) + - **Interactions**: water bridge and metal bridge detection via peppr; halogen bond sidechain flag now computed (was hardcoded) + - **Binding affinity**: fixed BindingDB matching — target sequence now validated against PDB SEQRES with 100% core identity, terminal tags/truncations tolerated ([#94](https://github.com/plinder-org/plinder/issues/94)); updated code to get the latest BindingDB release + - **Optional eval**: `pip install plinder[eval]` adds PoseBusters ligand validation; OpenStructure-backed metrics use the Conda-only `openstructure` package; PoseBusters no longer runs during ingest + - **PlinderSystem API**: new `receptor_structure` and `ligand_structures` (Biotite AtomArray) plus `ligand_mols` (RDKit Mol) properties; OpenStructure is confined to the evaluation implementation + - **Chain type support**: `Chain.from_cif_data` now assigns proper one-letter codes and chem_types for nucleotides (`RNA Linking`, `DNA Linking`); new `Residue.is_modified` property covers both protein PTMs and modified nucleotide bases + - **Save utils**: receptor/ligand chain naming generalized (`PDB_RECEPTOR_CHAINS`); system saving works for protein, NA, and mixed complexes + - **System definition**: unified `min_polymer_size=12` replaces separate `min_polymer_size`/`max_non_small_mol_ligand_length` — polymers ≥ 12 residues are receptor, shorter are ligands (threshold matches minimum MMseqs2/Foldseek search length); molecules with BIRD annotation are ligands irrespective of size; ligand chains no longer appear in both receptor and ligand parts of system IDs. + - **System grouping**: pocket-based grouping (≥ 3 shared receptor residues on the same chain instance) for adjacent binding sites (e.g. orthosteric + allosteric, cofactor + substrate in same active site); artifacts attach only via 4 Å proximity + - **Dead code removal**: removed unused OST-based functions, PDB string roundtrips, duplicate SMILES derivation paths, v1 template matching (consolidated to Rascal MCES `get_matched_template`) + - **License**: changed from GPL-2.0 to Apache-2.0 (GPL was only required by PLIP, now removed) + +- 2024-06/v2: + - New systems added based on the 2024-06 RCSB sync + - Updated system definition to be more stable and depend only on ligand distance rather than PLIP + - Added annotations for crystal contacts + - Improved ligand handling and saving to fix some bond order issues + - Improved covalency detection and annotation to reference each bond explicitly + - Added linked apo/pred structures to v2/links and v2/linked_structures + - Added binding affinity annotations from [BindingDB](https://bindingdb.org) (see known bugs!) + - Added statistics requirement and other changes in the split to enrich test set diversity + +- 2024-04/v1: Version described in the preprint, with updated redundancy removal by protein pocket and ligand similarity. +- 2024-04/v0: Version used to re-train DiffDock in the paper, with redundancy removal based on \\_\ diff --git a/README.md b/README.md index b3f0534e..949998d0 100644 --- a/README.md +++ b/README.md @@ -1,169 +1,123 @@ ![plinder](https://github.com/user-attachments/assets/05088c51-36c8-48c6-a7b2-8a69bd40fb44) -
-

Protein & Ligand INteraction Dataset and Evaluation Resource

-
+# PLINDER ---- +Protein & Ligand INteraction Dataset and Evaluation Resource -[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/plinder-org/plinder/blob/master/LICENSE.txt) +[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt) [![publish](https://github.com/plinder-org/plinder/actions/workflows/main.yaml/badge.svg)](https://pypi.org/project/plinder/) -[![website](https://img.shields.io/badge/website-plinder-blue.svg)](https://www.plinder.sh/) -[![bioRxiv](https://img.shields.io/badge/bioRxiv-2024.07.17.603955-blue.svg)](https://www.biorxiv.org/content/10.1101/2024.07.17.603955) [![docs](https://github.com/plinder-org/plinder/actions/workflows/docs.yaml/badge.svg)](https://plinder-org.github.io/plinder/) -[![coverage](https://github.com/plinder-org/plinder/raw/python-coverage-comment-action-data/badge.svg)](https://github.com/plinder-org/plinder/tree/python-coverage-comment-action-data) +[![bioRxiv](https://img.shields.io/badge/bioRxiv-2024.07.17.603955-blue.svg)](https://doi.org/10.1101/2024.07.17.603955) -![overview](https://github.com/user-attachments/assets/39d251b1-8114-4242-b9fc-e0cce900d22f) - -# 📚 About +PLINDER brings together protein and nucleic-acid monomers, protein–ligand systems, +protein–protein interfaces, protein–nucleic-acid complexes, and their +similarities for training and evaluating structure prediction and cofolding models. -**PLINDER** is the **Protein & Ligand INteraction Dataset and Evaluation -Resource**: a comprehensive, annotated, high-quality resource for training and -evaluating protein-ligand and protein-protein structure models. +The September 2026 release contains: -- \> 400k PLI systems across > 11k SCOP domains and > 50k unique small molecules -- Ligand-level annotations plus compact entry, chain, interface, and representative tables -- Automated curation pipeline to keep up with the PDB -- Reusable ligand, pocket, and protein-interface similarities and cluster assignments -- Deposited apo protein chains linked to compatible _holo_ systems -- Directed-cover assignments for choosing diverse training representatives -- Robust evaluation harness to simplify and standard performance comparison between - models. + + +| Release contents | Count | +| --- | ---: | +| Ligand systems | 611,930 | +| Proper ligands | 803,190 | +| Protein–protein interfaces | 3,811,623 | +| Protein-monomer assemblies | 161,654 | +| Nucleic-acid-monomer assemblies | 2,208 | +| Protein–nucleic-acid assemblies | 16,766 | -The *PLINDER* project is a community effort, launched by the University of Basel, -SIB Swiss Institute of Bioinformatics, Proxima (formerly VantAI), NVIDIA, MIT CSAIL, -and will be regularly updated. +Nucleic-acid monomers: 600 DNA, 1,587 RNA, and 21 DNA/RNA hybrids. + -PLINDER set a new standard for the Protein-Ligand interaction datasets. It was first introduced as part of the 2024 Machine Learning in Structural Biology (MLSB) [Workshop challenge](https://www.mlsb.io/index_2024.html#challenge) at NeurIPS, one of the field's premiere academic gatherings. -More details about the competition and other helpful practical tips can be found at our recent workshop repo: -[Moving Beyond Memorization](https://github.com/plinder-org/moving_beyond_memorisation). +## Start with the Python API -### 👋 [Join the P(L)INDER user group Discord Server!](https://discord.gg/KgUdMn7TuS) - - -## 🔢 Plinder versions - -We version the `plinder` dataset by the month stamp (`YYYY-MM`) of the RCSB -sync it was built from, set with `PLINDER_RELEASE`. The current release, -`2026-09`, is the default. - -We version the `plinder` application using an automated semantic -versioning scheme based on the `git` commit history. -The `plinder.data` package is responsible for generating a dataset -release and the `plinder.core` package makes it easy to interact -with the dataset. - -#### Changelog: - -- WIP (Current — unreleased): - - **Public downloads from Cameo**: releases download directly from `https://cameo3d.org/plinder/PLINDER-/` (currently `2026-09`); `PLINDER_MIRROR_URL` selects another HTTP file server with the same directory layout. Downloads stream to disk, validate HTTP Content-Length, and replace cached files atomically. Interrupted transfers retry from the start. Cached files are reused when their size and Last-Modified time match the server, so data hotfixes are fetched on the next access. Google storage dependencies are optional: install `plinder[data]` for the GCS data-generation utilities. - - **Major backend refactor**: replaced OST, gemmi, plip, openbabel with biotite + peppr for data generation; removed 6 dependencies from ingest pipeline - - **Nucleic acid support**: DNA/RNA chains now correctly included as receptor neighbors, mainchain/sidechain detection works for both protein and nucleic acids ([#61](https://github.com/plinder-org/plinder/issues/61)) - - **Custom CIF support**: new `Entry.from_custom_cif_file` for structure-prediction outputs (Boltz, AlphaFold3, Chai-1) that ship CIFs without `_chem_comp_bond` ([#117](https://github.com/plinder-org/plinder/issues/117)). Bond orders come from `ligand_smiles_dict` via positional atom-order match (the convention these tools follow); element/count mismatches raise with the offending position, `force_substructure_match=True` opts into substructure matching when atom order isn't preserved. User SMILES win over CCD for both `smiles` and `resolved_stereo_matches_template` — closes a silent gap where biotite's `LIG` placeholder would pass any 3D conformer. Input CIFs are never mutated; optional `save_fixed_cif` persists the enriched copy. - - **Stricter CIF ingest**: H/D filtered consistently (biotite's `filter_heavy`); multi-model CIFs warn and use model 1; multi-instance custom comp_ids must share heavy-atom naming (since `_chem_comp_bond` is comp_id-keyed); silent `connect_via_residue_names` and half-sanitized substructure fallbacks replaced with `ValueError` so corrupt inputs fail loudly. - - **Stereochemistry**: CCD ideal 3D coordinates used as stereo ground truth; new `resolved_stereo_matches_template` flag validates resolved structure chirality against CCD template (handles partial resolution via MCS trimming) - - **Interactions**: water bridge and metal bridge detection via peppr; halogen bond sidechain flag now computed (was hardcoded) - - **Binding affinity**: fixed BindingDB matching — target sequence now validated against PDB SEQRES with 100% core identity, terminal tags/truncations tolerated ([#94](https://github.com/plinder-org/plinder/issues/94)); updated code to get the latest BindingDB release - - **Optional eval**: `pip install plinder[eval]` adds PoseBusters ligand validation; OpenStructure-backed metrics use the Conda-only `openstructure` package; PoseBusters no longer runs during ingest - - **PlinderSystem API**: new `receptor_structure` and `ligand_structures` (Biotite AtomArray) plus `ligand_mols` (RDKit Mol) properties; OpenStructure is confined to the evaluation implementation - - **Chain type support**: `Chain.from_cif_data` now assigns proper one-letter codes and chem_types for nucleotides (`RNA Linking`, `DNA Linking`); new `Residue.is_modified` property covers both protein PTMs and modified nucleotide bases - - **Save utils**: receptor/ligand chain naming generalized (`PDB_RECEPTOR_CHAINS`); system saving works for protein, NA, and mixed complexes - - **System definition**: unified `min_polymer_size=12` replaces separate `min_polymer_size`/`max_non_small_mol_ligand_length` — polymers ≥ 12 residues are receptor, shorter are ligands (threshold matches minimum MMseqs2/Foldseek search length); molecules with BIRD annotation are ligands irrespective of size; ligand chains no longer appear in both receptor and ligand parts of system IDs. - - **System grouping**: pocket-based grouping (≥ 3 shared receptor residues on the same chain instance) for adjacent binding sites (e.g. orthosteric + allosteric, cofactor + substrate in same active site); artifacts attach only via 4 Å proximity - - **Dead code removal**: removed unused OST-based functions, PDB string roundtrips, duplicate SMILES derivation paths, v1 template matching (consolidated to Rascal MCES `get_matched_template`) - - **License**: changed from GPL-2.0 to Apache-2.0 (GPL was only required by PLIP, now removed) - -- 2024-06/v2: - - New systems added based on the 2024-06 RCSB sync - - Updated system definition to be more stable and depend only on ligand distance rather than PLIP - - Added annotations for crystal contacts - - Improved ligand handling and saving to fix some bond order issues - - Improved covalency detection and annotation to reference each bond explicitly - - Added linked apo/pred structures to v2/links and v2/linked_structures - - Added binding affinity annotations from [BindingDB](https://bindingdb.org) (see known bugs!) - - Added statistics requirement and other changes in the split to enrich test set diversity - -- 2024-04/v1: Version described in the preprint, with updated redundancy removal by protein pocket and ligand similarity. -- 2024-04/v0: Version used to re-train DiffDock in the paper, with redundancy removal based on \\_\ +```bash +pip install plinder +``` -## 🏅 Preprint benchmark sets +Files are downloaded as you use them and cached locally. Start with a small +query; downloading the full release is optional. -The historical `2024-04/v1` preprint release provides train, validation, and test splits that are -curated to minimize the information leakage based on protein-ligand interaction -similarity. -In addition, we have prioritized the systems that has a linked experimental `apo` -structure or matched molecular series to support realistic inference scenarios for hit -discovery and optimization. -Finally, a particular care is taken for test set that is further prioritized to contain -high quality structures to provide unambiguous ground-truths for performance -benchmarking. +```python +from plinder.core import query_table -![test_stratification](https://github.com/user-attachments/assets/5bb96534-f939-42b5-bf85-5ac3a71aa324) +ligands = query_table( + "annotation", + columns=["system_id", "ligand_id", "ligand_unique_ccd_code", "entry_resolution"], + filters=[("entry_pdb_id", "==", "4agi"), ("ligand_is_proper", "==", True)], +) +print(ligands) +``` -Moreover, as we enticipate this resource to be used for benchmarking a wide range of methods, including those simultaneously predicting protein structure (aka. co-folding) or those generating novel ligand structures, we further stratified test (by novel ligand, pocket, protein or all) to cover a wide range of tasks. +For protein–protein complexes, query `"interface_annotations"`. Use +`PlinderSystem` or `PlinderInterface` to access sequences, atom arrays, +ligand molecules, and reconstructed mmCIF files. -# 👨💻 Getting Started +## What can I do with PLINDER? -The *PLINDER* dataset is provided in two ways: +| Task | Runnable guide | +| --- | --- | +| Select ligands, interfaces, chains, and experimental annotations | [Query the whole annotated PDB](https://plinder-org.github.io/plinder/examples/2_query_filter_index.html) | +| Access protein–ligand systems and linked apo structures as CIF, SDF, FASTA, SMILES, and arrays | [Protein–ligand systems](https://plinder-org.github.io/plinder/examples/3_access_system_files.html) | +| Work with coordinate masks and ligand atom correspondences | [Atom arrays and mappings](https://plinder-org.github.io/plinder/examples/4_align_mask_crop.html) | +| Access protein–protein interfaces and their apo chains | [Protein–protein interfaces](https://plinder-org.github.io/plinder/examples/protein_interfaces.html) | +| Query ligand, pocket, chain, whole-interface, and half-interface scores | [Similarity scores](https://plinder-org.github.io/plinder/examples/similarity_and_representatives.html) | +| Select representatives and examine clusters | [Representative covers and components](https://plinder-org.github.io/plinder/examples/representatives_and_leakage.html) | +| Search with your own protein or complex sequences and structures | [Custom searches](https://plinder-org.github.io/plinder/examples/custom_scoring.html) | +| Evaluate ligand poses and protein-interface predictions | [Evaluation](https://plinder-org.github.io/plinder/examples/evaluation.html) | +| Download the full release locally | [Downloads](https://plinder-org.github.io/plinder/examples/1_download.html) | -- You can either use the files from the dataset directly using your preferred tooling - by downloading the data from the [Cameo file server](https://cameo3d.org/plinder/PLINDER-2026-09/), -- or you can utilize the dedicated `plinder` Python package for interfacing the data. +The [dataset reference](https://plinder-org.github.io/plinder/dataset.html) +describes release files, table relationships, and columns. The +[Python API reference](https://plinder-org.github.io/plinder/api/index.html) +lists individual functions and classes. +## Search and evaluate your models -## Downloading the dataset +`plinder.core.scores.search()` accepts protein FASTA files, structure files or +folders, and tables pairing receptors with ligand SDFs. It returns chain hits +and the applicable pocket, ligand, or interface scores. CIF/PDB searches use +MMseqs2 and Foldseek; FASTA queries use MMseqs2. -After installing the package, download the index tables and cluster assignments -for a release with: - -```console -$ plinder_download --release 2026-09 -``` +`plinder.eval.evaluate_predictions()` compares predictions with release +references and returns ligand/interface metrics and PoseBusters checks. +Search and evaluation use the same structure-table convention. -The command offers the larger ligand, alignment, score, export, and search -database groups separately, and dataset APIs also download required files -lazily. No Google Cloud credentials or SDK are needed: releases download -from `https://cameo3d.org/plinder/PLINDER-2026-09/`; `PLINDER_MIRROR_URL` -can select another HTTP file-server root containing `PLINDER-/` -directories. The legacy `2024-06/v2` layout requires an earlier `plinder` version. -For details on the release paths, see [Documentation](https://plinder-org.github.io/plinder/tutorial/dataset.html). +Custom searches require the corresponding MMseqs2/Foldseek executables. +For evaluation, install `plinder[eval]` and OpenStructure 2.12.0 or newer; +see the [installation instructions](https://plinder-org.github.io/plinder/evaluation.html#installation). -## Installing the Python package +## Releases and local files -`plinder` is available on *PyPI*. +The current dataset release is `2026-09`. Package versions are separate from +dataset releases. See [CHANGELOG.md](CHANGELOG.md) for the release history. -``` -pip install plinder -``` +Public files are available [here](https://cameo3d.org/plinder/PLINDER-2026-09/). +Set `PLINDER_MOUNT` before importing PLINDER to choose a cache location. +Use `plinder_download` to prepare a local copy and `PLINDER_OFFLINE=true` +to use cached files without network access. -For PoseBusters ligand validation: +## Community and examples -``` -pip install plinder[eval] -``` +PLINDER is a community effort launched by the University of Basel, +SIB Swiss Institute of Bioinformatics, Proxima (formerly VantAI), NVIDIA, +and MIT CSAIL. -OpenStructure is not published on PyPI. Evaluation metrics backed by -OpenStructure (including lDDT and RMSD) use its command-line actions and require -OpenStructure 2.12.0 or newer; the repository's `environment.yml` installs it -from Bioconda. See the [evaluation guide](docs/evaluation.md) for evaluating -folders of ligand or protein-interface predictions. +Join the [P(L)INDER Discord](https://discord.gg/KgUdMn7TuS), or explore the +[Moving Beyond Memorization workshop](https://github.com/plinder-org/moving_beyond_memorisation) +for the original 2024 tutorials on dataset selection, similarity, and evaluation. +The guides linked above use the current API and release. ## License -Data curated by PLINDER are made available under the Apache License 2.0. -All data curated by BindingDB staff are provided under the Creative Commons Attribution 4.0 License. Data imported from ChEMBL are provided under their Creative Commons Attribution-Share Alike 4.0 Unported License. - -# 📝 Documentation +PLINDER code and curated data are available under the Apache License 2.0. +BindingDB-curated data use CC BY 4.0; data imported from ChEMBL use +CC BY-SA 4.0. -A more detailed description is available on the -[documentation website](https://plinder-org.github.io/plinder/). - -# 📃 Citation +## Citation Durairaj, Janani, Yusuf Adeshina, Zhonglin Cao, Xuejin Zhang, Vladas Oleinikovas, Thomas Duignan, Zachary McClure, Xavier Robin, Gabriel Studer, Daniel Kovtun, Emanuele Rossi, Guoqing Zhou, Srimukh Prasad Veccham, Clemens Isert, Yuxing Peng, Prabindh Sundareson, Mehmet Akdel, Gabriele Corso, Hannes Stärk, Gerardo Tauriello, Zachary Wayne Carpenter, Michael M. Bronstein, Emine Kucukbenli, Torsten Schwede, Luca Naef. 2024. “PLINDER: The Protein-Ligand Interactions Dataset and Evaluation Resource.” [bioRxiv](https://doi.org/10.1101/2024.07.17.603955) [ICML'24 ML4LMS](https://openreview.net/forum?id=7UvbaTrNbP) Please see the [citation file](CITATION.cff) for details. - -![plinder_banner](https://github.com/user-attachments/assets/43d129f2-3bb6-4903-81fa-182c351c64b6) diff --git a/docs/dataset.md b/docs/dataset.md index d38441e4..50dfc768 100644 --- a/docs/dataset.md +++ b/docs/dataset.md @@ -83,6 +83,55 @@ group. APIs fetch missing optional files when they need them unless offline mode is enabled. Source PDB mmCIFs are separate: they are fetched per PDB entry during reconstruction and are not part of the bulk download. +### Selective score downloads + +Releases with a `score_store//manifest.json` support downloading +only the score shards needed by a query. The score-query APIs and columns +are unchanged. For example, filter on `query_ligand_id`, `query_system`, +`query_half_interface_id`, or `query_entry` using `==` or `in` to select +query shards. Chemical ligand scores use numeric ligand-ID ranges. +Row order is not guaranteed; sort the result explicitly when order matters. + +Target-only equality and membership queries use an exact target-to-shard +index. Scores remain directed: a target lookup does not substitute the +reverse pair's score. Queries with only a score threshold, or many widely +distributed IDs, can still need most of the release. Column projection +reduces local reads, but whole selected shard files are downloaded. + +The client checks for this manifest before fetching score data. Until the +new exports are published, it reads the existing release layout, which +can require a large first download. A missing shard in a manifest-backed +release is an error, not a reason to silently download a legacy table. +Offline use requires the manifest and every shard needed by the query. + +Release maintainers can repackage finalized scores without recomputing +them or changing the source files: + +```bash +python -m plinder.data.pipeline.score_store WORKING_RELEASE STAGING_RELEASE \ + --threads 2 --memory-limit 2GB +``` + +Run this release-wide operation on a compute node with adequate temporary +disk space. It writes only `STAGING_RELEASE/score_store/`, refuses to +overwrite an existing artifact, and installs each artifact atomically. +The memory limit controls the engine's working-memory budget, not total +process RAM; allow additional headroom and disk space for external sorts. +Each query bucket is sorted in a fresh engine session. Target-routing +indices are deduplicated per query bucket and finalized in eight groups, +without a release-wide in-memory sort or deduplication. +Exports retain all rows (including duplicate rows), original columns and +types, and directionality. Each bucket is sorted and checked using row +counts and order-independent row fingerprints. The manifest records the +total row count, schema, fingerprint, and all data/index file paths. + +For deployment, place the completed `score_store/` beside the other finalized +artifacts in a separate working release tree, then run +`plinder_prepare_release` as usual. The public-release allowlist includes +only completed, manifest-listed score-store files. Keep the existing score +artifacts for older clients. Publishing the new files is a separate step; +running the exporter never uploads anything. + (annotation-table-target)= (annotation-tables-index)= @@ -96,10 +145,14 @@ chain, validation, cluster, and representative data so clients only load it when needed. `query_table()` can request columns across related tables in a single query and selects the required relationships from those columns. -- `annotation`: ligand annotations and reconstructable system IDs; +- `annotation`: ligand annotations and reconstructable system IDs, including + `ligand_validation_*` for that ligand's own residues (all members of a + multi-chain covalent ligand) and `ligand_pocket_validation_*` for its own + neighboring receptor residues; - `system_validation`: ligand and pocket validation summaries for each system; -- `entry_chains`: one polymer chain in a PDB entry; +- `entry_chains`: one polymer chain in a PDB entry, including + `chain_unresolved_fraction` (unresolved residues / SEQRES length, from 0 to 1); - `entry_biounit_chains`: one chain instance in a biological assembly; - `entry_metadata`: experimental and entry-validation metadata; - `entry_sources`: the source mmCIF revision used for the release; @@ -125,6 +178,28 @@ single query and selects the required relationships from those columns. - `interface_half_representatives`: representative interface sides; - `interface_clusters`: protein-interface cover labels. +Per-ligand validation is available directly through `query_table()` without +loading or joining system summaries: + +```python +from plinder.core import query_table + +ligands = query_table( + "annotation", + columns=[ + "ligand_id", + "ligand_validation_average_rscc", + "ligand_validation_average_rsr", + "ligand_pocket_validation_average_rscc", + ], + filters=[("ligand_validation_average_rscc", ">=", 0.8)], +) +``` + +Use these per-ligand values to compare ligand copies. Optional system-level +summaries are available as `system_ligand_validation_*` and +`system_pocket_validation_*` columns. + The checked-in column reference below is generated from these release tables. BindingDB's PDB/ligand cross-references can include similar, rather than exact, diff --git a/docs/examples/2_query_filter_index.ipynb b/docs/examples/2_query_filter_index.ipynb index 4316f9ff..3239f571 100644 --- a/docs/examples/2_query_filter_index.ipynb +++ b/docs/examples/2_query_filter_index.ipynb @@ -19,7 +19,7 @@ "source": [ "import pandas as pd\n", "\n", - "from plinder.core import PlinderRelease, RELEASE_TABLES, query_table" + "from plinder.core import RELEASE_TABLES, PlinderRelease, query_table" ] }, { @@ -41,13 +41,36 @@ "source": [ "release = PlinderRelease()\n", "pd.DataFrame(\n", - " [\n", - " {\"table\": name, **release.table(name)}\n", - " for name in RELEASE_TABLES\n", - " ]\n", + " [{\"table\": name, **release.table(name)} for name in RELEASE_TABLES]\n", ").set_index(\"table\")" ] }, + { + "cell_type": "markdown", + "id": "column-reference", + "metadata": {}, + "source": [ + "## Find columns and descriptions\n", + "\n", + "`get_column_descriptions()` lists a table's names, types, and descriptions without downloading the table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "describe-columns", + "metadata": {}, + "outputs": [], + "source": [ + "from plinder.data.docs import get_column_descriptions\n", + "\n", + "descriptions = get_column_descriptions(\"annotation\")\n", + "descriptions.loc[\n", + " descriptions[\"Name\"].str.contains(\"unresolved|covalent|cofactor|validation\"),\n", + " [\"Name\", \"Type\", \"Description\"],\n", + "].head(12)" + ] + }, { "cell_type": "markdown", "id": "query-ligands", @@ -176,6 +199,142 @@ ")\n", "chains" ] + }, + { + "cell_type": "markdown", + "id": "quality-and-chemistry", + "metadata": {}, + "source": [ + "## Select experimental quality and ligand chemistry\n", + "\n", + "To rank ligand copies, query `ligand_validation_average_rscc` and `ligand_validation_average_rsr` from `annotation`. The `ligand_pocket_validation_*` columns describe each ligand's pocket. The optional `system_pass_validation_criteria` flag below screens the whole system." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "quality-query", + "metadata": {}, + "outputs": [], + "source": [ + "quality = query_table(\n", + " columns=[\n", + " \"system_id\",\n", + " \"ligand_id\",\n", + " \"ligand_is_cofactor\",\n", + " \"ligand_is_covalent\",\n", + " \"ligand_num_unresolved_heavy_atoms\",\n", + " \"ligand_num_heavy_atoms\",\n", + " \"system_pass_validation_criteria\",\n", + " \"entry_resolution\",\n", + " ],\n", + " filters=[\n", + " (\"entry_pdb_id\", \"in\", [\"4agi\", \"7eek\"]),\n", + " (\"ligand_is_proper\", \"==\", True),\n", + " ],\n", + ")\n", + "quality" + ] + }, + { + "cell_type": "markdown", + "id": "binding-affinity", + "metadata": {}, + "source": [ + "## Inspect binding affinities and source measurements\n", + "\n", + "The scalar affinity is a median pKi or pKd for compatible, uncensored measurements with a matched target sequence. Individual BindingDB records retain endpoint, qualifier, target sequence, and publication information." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "affinity-query", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " columns=[\n", + " \"ligand_id\",\n", + " \"ligand_binding_affinity\",\n", + " \"ligand_binding_affinity_endpoint\",\n", + " \"ligand_binding_affinity_measurement_count\",\n", + " ],\n", + " filters=[\n", + " (\"entry_pdb_id\", \"in\", [\"4agi\", \"7eek\"]),\n", + " (\"ligand_is_proper\", \"==\", True),\n", + " ],\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "affinity-records", + "metadata": {}, + "outputs": [], + "source": [ + "measurements = query_table(\n", + " \"bindingdb_measurements\",\n", + " columns=[\n", + " \"pdb_id\",\n", + " \"ligand_het_id\",\n", + " \"endpoint\",\n", + " \"raw_value\",\n", + " \"relation\",\n", + " \"pchembl\",\n", + " \"target_sequence\",\n", + " \"article_doi\",\n", + " ],\n", + " filters=[(\"pdb_id\", \"in\", [\"4agi\", \"7eek\"])],\n", + ")\n", + "measurements.head()" + ] + }, + { + "cell_type": "markdown", + "id": "chain-and-source-metadata", + "metadata": {}, + "source": [ + "## Inspect polymer types and source revisions\n", + "\n", + "Chain records distinguish protein, DNA, RNA, and hybrid polymers. `chain_sequence` contains the deposited sequence; `chain_num_unresolved_residues` describes coordinate completeness. Source revision numbers identify the mmCIF used for the release." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "chain-metadata", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " \"entry_chains\",\n", + " columns=[\n", + " \"entry_pdb_id\",\n", + " \"chain_asym_id\",\n", + " \"chain_entity_id\",\n", + " \"chain_receptor_type\",\n", + " \"chain_length\",\n", + " \"chain_num_unresolved_residues\",\n", + " \"chain_uniprot_ids\",\n", + " ],\n", + " filters=[(\"entry_pdb_id\", \"in\", [\"4agi\", \"7eek\"])],\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "source-metadata", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " \"entry_sources\",\n", + " filters=[(\"entry_pdb_id\", \"in\", [\"4agi\", \"7eek\"])],\n", + ")" + ] } ], "metadata": { diff --git a/docs/examples/3_access_system_files.ipynb b/docs/examples/3_access_system_files.ipynb index b9237ec3..45af5448 100644 --- a/docs/examples/3_access_system_files.ipynb +++ b/docs/examples/3_access_system_files.ipynb @@ -175,6 +175,30 @@ " )\n", " print(apo_cif, selected_cif)" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Superpose an apo chain onto the holo receptor\n", + "\n", + "`superpose_linked_apo()` writes the apo chain in the selected holo receptor's coordinate frame. Supply `reference_chain` for a multichain receptor." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if not link_info.empty:\n", + " fitted_apo = linked_system.superpose_linked_apo(\n", + " linked_structure_id=link_info.iloc[0][\"linked_structure_id\"],\n", + " reference_chain=\"1.C\",\n", + " overwrite=True,\n", + " )\n", + " print(fitted_apo)" + ] } ], "metadata": { diff --git a/docs/examples/custom_scoring.ipynb b/docs/examples/custom_scoring.ipynb index ffde8f4b..e34eedeb 100644 --- a/docs/examples/custom_scoring.ipynb +++ b/docs/examples/custom_scoring.ipynb @@ -6,20 +6,18 @@ "source": [ "# Search PLINDER with your structures and sequences\n", "\n", - "`search()` finds similar proteins, ligand pockets and protein interfaces in PLINDER. Supply an input table for ligand-containing structures, or a FASTA file or structure folder for ligand-free queries." + "`search()` finds similar proteins and potential binding sites. Pass a FASTA, a PDB/mmCIF file, a structure folder, or a table pairing receptors with ligand SDFs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Input requirements\n", + "## Inputs\n", "\n", - "Search and evaluation share a table with `input_id`, `structure_path`, and optional `ligand_path` columns. Evaluation adds `reference_id`. See the {ref}`input table reference ` for details.\n", + "Use a structure file directly, including a complete mmCIF with ligand chemistry. For receptors with separate ligand SDFs, use a table with `input_id`, `structure_path`, and `ligand_path`; the SDFs must share the receptor's coordinate frame. Search and evaluation use the same {ref}`input table format `.\n", "\n", - "Use an assembled PDB/mmCIF receptor with one or more ligand SDFs, or a complete mmCIF containing the ligand coordinates and chemistry. The SDFs must use the receptor's coordinate frame.\n", - "\n", - "For ligand-free queries, pass a FASTA file, a structure file, or a folder of PDB/mmCIF structures directly." + "PDB files supply protein coordinates; supply their ligand poses as SDFs. FASTA inputs support protein, pocket, and interface-site discovery." ] }, { @@ -33,7 +31,7 @@ "\n", "import pandas as pd\n", "\n", - "from plinder.core import PlinderSystem, PlinderRelease\n", + "from plinder.core import PlinderRelease, PlinderSystem, query_table\n", "from plinder.core.scores import (\n", " CustomProteinSearchConfig,\n", " search,\n", @@ -49,20 +47,26 @@ " release=PlinderRelease(data_dir=release_path),\n", " ).receptor_cif\n", ")\n", - "work_dir = Path(\n", - " os.environ.get(\"PLINDER_CUSTOM_WORK_DIR\", \"custom_scoring_example\")\n", - ")" + "work_dir = Path(os.environ.get(\"PLINDER_CUSTOM_WORK_DIR\", \"custom_scoring_example\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Run a small receptor-only example\n", + "## Find proteins and binding sites\n", + "\n", + "This small example uses a PLINDER receptor unless `PLINDER_CUSTOM_CIF` is set. Omit `plinder_entry_ids` to search the full annotated PDB release.\n", "\n", - "The example rebuilds a PLINDER receptor mmCIF when `PLINDER_CUSTOM_CIF` is not set. `plinder_entry_ids` limits the comparison to receptor and interface proteins from the selected entries; omit it to search all PLINDER proteins.\n", + "Every search returns protein-chain hits, including monomer targets by default. Leave `mode=\"auto\"` to run what your input supports, or use `mode=\"all\"` to select all feature families. To focus the search:\n", "\n", - "`mode=\"pockets\"` searches protein and pocket features. Use `mode=\"interfaces\"` for protein interfaces, `mode=\"ligands\"` for ligand comparisons, or `mode=\"both\"` for both. The default `mode=\"auto\"` selects applicable features." + "| Mode | What it finds |\n", + "| --- | --- |\n", + "| `pockets` | Potential ligand-binding sites on your proteins; no query ligand needed |\n", + "| `ligands` | Similar ligands, pockets, and interactions; requires query ligands |\n", + "| `interfaces` | Potential protein-binding sites on your chains, plus complete-interface matches when your input contains a complex |\n", + "\n", + "Hits are based on sequence/structure similarity, indicating potential pockets/interfaces." ] }, { @@ -76,7 +80,7 @@ " input_cif,\n", " output_dir=work_dir,\n", " release=PlinderRelease(data_dir=release_path),\n", - " mode=\"pockets\",\n", + " mode=\"all\",\n", " backends=(\"mmseqs\",),\n", " plinder_entry_ids=[\"4agi\"],\n", " search_config=CustomProteinSearchConfig(max_seqs=25),\n", @@ -85,6 +89,44 @@ ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read direct chain hits\n", + "\n", + "`chain_similarity_scores` contains custom-to-PLINDER chain hits, including identity, sequence similarity, and coverage. Monomer targets are included by default; pocket and interface outputs retain their respective target sets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "chain_hits = pd.read_parquet(result.chain_similarity_scores)\n", + "chain_hits[\n", + " [\n", + " \"query_chain_asym_id\",\n", + " \"target_entry\",\n", + " \"target_chain_asym_id\",\n", + " \"source\",\n", + " \"qcov\",\n", + " \"tcov\",\n", + " \"fident\",\n", + " \"seqsim\",\n", + " \"lddt\",\n", + " ]\n", + "].head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read pocket scores" + ] + }, { "cell_type": "code", "execution_count": null, @@ -141,15 +183,17 @@ "metadata": {}, "outputs": [], "source": [ - "residue_pairs[[\n", - " \"plinder_system_id\",\n", - " \"plinder_chain_instance\",\n", - " \"plinder_residue_number\",\n", - " \"custom_structure_id\",\n", - " \"custom_chain_asym_id\",\n", - " \"custom_residue_number\",\n", - " \"residue_identical\",\n", - "]].head()" + "residue_pairs[\n", + " [\n", + " \"plinder_system_id\",\n", + " \"plinder_chain_instance\",\n", + " \"plinder_residue_number\",\n", + " \"custom_structure_id\",\n", + " \"custom_chain_asym_id\",\n", + " \"custom_residue_number\",\n", + " \"residue_identical\",\n", + " ]\n", + "].head()" ] }, { @@ -168,13 +212,19 @@ "outputs": [], "source": [ "def search_ligand_model(receptor_path, ligand_path):\n", - " inputs = pd.DataFrame([{\n", - " \"input_id\": \"model_1\",\n", - " \"structure_path\": str(receptor_path),\n", - " \"ligand_path\": str(ligand_path),\n", - " }])\n", + " inputs = pd.DataFrame(\n", + " [\n", + " {\n", + " \"input_id\": \"model_1\",\n", + " \"structure_path\": str(receptor_path),\n", + " \"ligand_path\": str(ligand_path),\n", + " }\n", + " ]\n", + " )\n", " return search(\n", - " inputs, output_dir=\"ligand_search\", mode=\"ligands\",\n", + " inputs,\n", + " output_dir=\"ligand_search\",\n", + " mode=\"ligands\",\n", " )\n", "\n", "\n", @@ -186,9 +236,121 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Compare protein sequences\n", + "## Find pockets and interface sites from FASTA\n", + "\n", + "This example searches two sequences: a kinase and one chain from a known protein interface. Original FASTA names appear in `sequence_id`. No ligand or binding partner is required." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "example_system = PlinderSystem(\n", + " system_id=\"4agi__1__1.C__1.W\",\n", + " release=PlinderRelease(data_dir=release_path),\n", + ")\n", + "sequence = next(iter(example_system.sequences.values()))\n", + "interface_sequence = query_table(\n", + " \"entry_chains\",\n", + " columns=[\"chain_sequence\"],\n", + " filters=[(\"entry_pdb_id\", \"==\", \"7cm8\"), (\"chain_asym_id\", \"==\", \"A\")],\n", + " release=PlinderRelease(data_dir=release_path),\n", + ").iloc[0][\"chain_sequence\"]\n", + "fasta = work_dir / \"queries.fasta\"\n", + "fasta.write_text(\n", + " f\">my_kinase\\n{sequence}\\n>my_interface_protein\\n{interface_sequence}\\n\"\n", + ")\n", + "sequence_result = search(\n", + " fasta,\n", + " output_dir=work_dir / \"sequence_search\",\n", + " release=PlinderRelease(data_dir=release_path),\n", + " plinder_entry_ids=[\"4agi\", \"7cm8\"],\n", + " backends=(\"mmseqs\",),\n", + " search_config=CustomProteinSearchConfig(evalue=0.01, max_seqs=25),\n", + " store_aligned_pocket_residues=True,\n", + " threads=2,\n", + ")\n", + "pd.read_parquet(sequence_result.chain_similarity_scores)[\n", + " [\n", + " \"sequence_id\",\n", + " \"target_entry\",\n", + " \"target_chain_asym_id\",\n", + " \"fident\",\n", + " \"qcov\",\n", + " \"tcov\",\n", + " ]\n", + "].head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`interface_side_scores` identifies the known interface side and maps its residues onto your sequence. Coverage and identity are percentages of the entire known site, so unaligned residues count as zero." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pd.read_parquet(sequence_result.interface_side_scores)[\n", + " [\n", + " \"sequence_id\",\n", + " \"plinder_interface_id\",\n", + " \"plinder_chain\",\n", + " \"plinder_partner_chain\",\n", + " \"interface_side_qcov\",\n", + " \"interface_side_fident\",\n", + " ]\n", + "].head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pd.read_parquet(sequence_result.best_sequence_links).head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Set search filters\n", + "\n", + "The shared E-value defaults to 0.01. `mmseqs_evalue` and `foldseek_evalue` override it for the corresponding backend. Identity and coverage filters use fractions between zero and one." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "search_config = CustomProteinSearchConfig(\n", + " evalue=0.01,\n", + " mmseqs_evalue=0.001,\n", + " foldseek_evalue=0.01,\n", + " max_seqs=1000,\n", + " coverage=0.8,\n", + " min_seq_id=0.3,\n", + ")\n", + "search_config" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Search a monomer or a protein complex\n", "\n", - "Protein sequences can be searched without coordinates. This MMseqs route writes protein scores, ligand-pocket links, the best link per input sequence, and optionally the aligned pocket residues." + "Use the same call for one chain or a complex. `interface_side_scores` contains site matches on individual chains; `interface_scores` additionally contains complete two-chain comparisons when your input has an interface." ] }, { @@ -197,20 +359,26 @@ "metadata": {}, "outputs": [], "source": [ - "def search_sequences(fasta_path):\n", + "def search_interfaces(structures):\n", " return search(\n", - " Path(fasta_path),\n", - " output_dir=Path(fasta_path).with_suffix(\"\") / \"plinder_search\",\n", - " threads=8,\n", - " store_aligned_pocket_residues=True,\n", - " )" + " structures,\n", + " output_dir=\"interface_search\",\n", + " mode=\"interfaces\",\n", + " search_config=search_config,\n", + " )\n", + "\n", + "\n", + "# monomer_result = search_interfaces(\"monomer.cif\")\n", + "# pd.read_parquet(monomer_result.interface_side_scores).head()\n", + "# complex_result = search_interfaces(\"complex.cif\")\n", + "# pd.read_parquet(complex_result.interface_scores).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "For several ligand-free coordinate files, use `search(\"models/\", output_dir=\"search_results\")`. Search uses the first coordinate model as the supplied assembly." + "For a folder of structures, pass `search(\"models/\", output_dir=\"search_results\")`. Each file is searched independently, using its first coordinate model as the supplied assembly." ] } ], diff --git a/docs/examples/index.md b/docs/examples/index.md index 9555b3e2..889d120f 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -1,59 +1,34 @@ # Guides -The notebooks are arranged by task so that release discovery, structure work, -and similarity analysis can be learned independently. - -## Data access - -Start here when working directly with the release tables and files. +| Task | Runnable guide | +| --- | --- | +| Select ligands, interfaces, chains, and experimental annotations | [Query the whole annotated PDB](2_query_filter_index.ipynb) | +| Access protein–ligand systems and linked apo structures as CIF, SDF, FASTA, SMILES, and arrays | [Protein–ligand systems](3_access_system_files.ipynb) | +| Work with coordinate masks and ligand atom correspondences | [Atom arrays and mappings](4_align_mask_crop.ipynb) | +| Access protein–protein interfaces and their apo chains | [Protein–protein interfaces](protein_interfaces.ipynb) | +| Query ligand, pocket, chain, whole-interface, and half-interface scores | [Similarity scores](similarity_and_representatives.ipynb) | +| Select representatives and examine clusters | [Representative covers and components](representatives_and_leakage.ipynb) | +| Search with your own protein or complex sequences and structures | [Custom searches](custom_scoring.ipynb) | +| Evaluate ligand poses and protein-interface predictions | [Evaluation](evaluation.ipynb) | +| Download the full release locally | [Downloads](1_download.ipynb) | ```{toctree} +:hidden: :maxdepth: 1 2_query_filter_index -``` - -## Structures - -Reconstruct ligand systems and linked apo chains, then work with atom arrays, -masks, sequences, and ligand atom mappings. - -```{toctree} -:maxdepth: 1 - 3_access_system_files -protein_interfaces 4_align_mask_crop -``` - -## Similarity and search - -Inspect similarity scores and representative covers, or compare -custom structures and protein sequences with PLINDER. - -```{toctree} -:maxdepth: 1 - +protein_interfaces similarity_and_representatives +representatives_and_leakage custom_scoring -``` - -## Evaluation - -Compare predicted ligand poses and protein interfaces with known references. - -```{toctree} -:maxdepth: 1 - evaluation +1_download ``` -## Offline access - -Download a local copy when you need to work without internet access. - -```{toctree} -:maxdepth: 1 +## Workshop material -1_download -``` +The [Moving Beyond Memorization workshop](https://github.com/plinder-org/moving_beyond_memorisation) +contains the original 2024 dataset-selection and evaluation exercises. The +guides above use the current release tables and API. diff --git a/docs/examples/protein_interfaces.ipynb b/docs/examples/protein_interfaces.ipynb index efaf6c37..0f8fd81d 100644 --- a/docs/examples/protein_interfaces.ipynb +++ b/docs/examples/protein_interfaces.ipynb @@ -18,7 +18,7 @@ "import biotite.structure as struc\n", "import pandas as pd\n", "\n", - "from plinder.core import PlinderInterface" + "from plinder.core import PlinderInterface, query_table" ] }, { @@ -111,6 +111,41 @@ "source": [ "`interface_cif` is a `pathlib.Path`; `atom_array`, the values of `chain_structures`, and `interface_structure` are Biotite `AtomArray` objects. The written mmCIF is self-contained and contains only the selected interface chains." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Find apo chains for each interface side\n", + "\n", + "`interface_apo_structures` contains ranked candidates for each side. Here, apo means that the candidate chain has no contacting protein partner. Sequence identity, coverage, and available Foldseek lDDT describe each match." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "apo_links = query_table(\n", + " \"interface_apo_structures\",\n", + " columns=[\n", + " \"reference_system_id\",\n", + " \"reference_side\",\n", + " \"reference_chain_instance\",\n", + " \"source_entry_id\",\n", + " \"source_chain_instance\",\n", + " \"source_num_contacting_proteins\",\n", + " \"rank\",\n", + " \"mmseqs_fident\",\n", + " \"mmseqs_query_coverage\",\n", + " \"mmseqs_target_coverage\",\n", + " \"foldseek_lddt\",\n", + " ],\n", + " filters=[(\"reference_system_id\", \"==\", interface.system_id)],\n", + ")\n", + "apo_links.head(10)" + ] } ], "metadata": { diff --git a/docs/examples/representatives_and_leakage.ipynb b/docs/examples/representatives_and_leakage.ipynb new file mode 100644 index 00000000..d9a4a601 --- /dev/null +++ b/docs/examples/representatives_and_leakage.ipynb @@ -0,0 +1,267 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "representatives-and-components", + "metadata": {}, + "source": [ + "# Representative covers and components\n", + "\n", + "Choose diverse examples with representative covers, and inspect related groups with component IDs. Covers select representatives; components connect items through the full similarity graph.\n", + "\n", + "Component IDs and assignment thresholds are defined by the release." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cover-imports", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import pandas as pd\n", + "\n", + "from plinder.core import PlinderRelease, query_table\n", + "from plinder.core.scores import query_ligand_similarity\n", + "\n", + "example_data = os.environ.get(\"PLINDER_SCORE_EXAMPLE_DATA_DIR\")\n", + "release = PlinderRelease(data_dir=Path(example_data)) if example_data else PlinderRelease()\n", + "ligand_id = \"4agi__1__1.R\"\n", + "cover = \"pocket_qcov__70__ligand__directed_set_cover\"\n", + "component = \"pocket_qcov__70__ligand__reciprocal_component\"" + ] + }, + { + "cell_type": "markdown", + "id": "ligand-cover-columns", + "metadata": {}, + "source": [ + "## Find a ligand's representative group\n", + "\n", + "Cover columns join automatically onto ligand annotations. `__is_centroid` identifies published representatives." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-ligand-cover", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " columns=[\"system_id\", \"ligand_id\", cover, f\"{cover}__is_centroid\", component],\n", + " filters=[(\"ligand_id\", \"==\", ligand_id)],\n", + " release=release,\n", + ").T" + ] + }, + { + "cell_type": "markdown", + "id": "detailed-cover-assignments", + "metadata": {}, + "source": [ + "## Inspect the actual assignment threshold\n", + "\n", + "Directional covers can use the fallback threshold for members that lack a match at the requested threshold. The detailed cover table records the representative, similarity, and threshold used for each assignment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-cover-details", + "metadata": {}, + "outputs": [], + "source": [ + "details = pd.read_parquet(\n", + " release.fetch(\"ligand_sampling\")\n", + " / \"directed_set_cover\"\n", + " / \"metric=pocket_qcov\"\n", + " / \"threshold=70.parquet\",\n", + " columns=[\n", + " \"ligand_id\",\n", + " \"centroid_ligand_id\",\n", + " \"similarity_to_centroid\",\n", + " \"assignment_threshold\",\n", + " \"representative_selection_threshold\",\n", + " ],\n", + " filters=[(\"ligand_id\", \"==\", ligand_id)],\n", + ")\n", + "details" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-representative-annotations", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " columns=[\"ligand_id\", \"system_id\", \"ligand_unique_ccd_code\", \"entry_resolution\"],\n", + " filters=[(\"ligand_id\", \"in\", details[\"centroid_ligand_id\"].unique().tolist())],\n", + " release=release,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "interface-cover-columns", + "metadata": {}, + "source": [ + "## Whole-interface and half-interface groups\n", + "\n", + "Whole-interface columns use `interface_qcov`; each side has its own `interface_side_qcov` columns. Reciprocal components use edges meeting the threshold in both directions. Members can connect through other members." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-interface-components", + "metadata": {}, + "outputs": [], + "source": [ + "query_table(\n", + " \"interface_annotations\",\n", + " columns=[\n", + " \"system_id\",\n", + " \"interface_qcov__70__directed_set_cover\",\n", + " \"interface_qcov__70__reciprocal_component\",\n", + " \"interface_side_qcov__70__chain_1_directed_set_cover\",\n", + " \"interface_side_qcov__70__chain_2_directed_set_cover\",\n", + " \"interface_side_qcov__70__chain_1_reciprocal_component\",\n", + " \"interface_side_qcov__70__chain_2_reciprocal_component\",\n", + " ],\n", + " filters=[(\"entry_pdb_id\", \"==\", \"7cm8\")],\n", + " release=release,\n", + ").head()" + ] + }, + { + "cell_type": "markdown", + "id": "protein-cluster-columns", + "metadata": {}, + "source": [ + "## Protein sequence and structure clusters\n", + "\n", + "Protein clusters identify their representatives by PDB entry and label asym ID. The status column records how each chain was assigned." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-protein-clusters", + "metadata": {}, + "outputs": [], + "source": [ + "for table in (\"protein_sequence_clusters\", \"protein_structure_clusters\"):\n", + " display(\n", + " query_table(\n", + " table,\n", + " filters=[(\"entry_pdb_id\", \"in\", [\"4agi\", \"7eek\"])],\n", + " release=release,\n", + " )\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "component-screening", + "metadata": {}, + "source": [ + "## Screen a proposed holdout\n", + "\n", + "Here, `7eek` is a small illustrative holdout. We remove candidate training systems containing a ligand in any of its pocket components. Use the full candidate set and the metrics relevant to your benchmark for a real split." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "screen-heldout-components", + "metadata": {}, + "outputs": [], + "source": [ + "candidates = query_table(\n", + " columns=[\"entry_pdb_id\", \"system_id\", \"ligand_id\", component],\n", + " filters=[\n", + " (\"entry_pdb_id\", \"in\", [\"4agi\", \"4agc\", \"7eek\"]),\n", + " (\"ligand_is_proper\", \"==\", True),\n", + " ],\n", + " release=release,\n", + ").dropna(subset=[component])\n", + "heldout = candidates.loc[candidates[\"entry_pdb_id\"].eq(\"7eek\")]\n", + "heldout_components = set(heldout[component])\n", + "blocked_systems = set(\n", + " candidates.loc[candidates[component].isin(heldout_components), \"system_id\"]\n", + ")\n", + "training = candidates.loc[\n", + " ~candidates[\"entry_pdb_id\"].eq(\"7eek\")\n", + " & ~candidates[\"system_id\"].isin(blocked_systems)\n", + "]\n", + "pd.Series(\n", + " {\n", + " \"heldout_systems\": heldout[\"system_id\"].nunique(),\n", + " \"remaining_training_systems\": training[\"system_id\"].nunique(),\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "directional-cross-set-scores", + "metadata": {}, + "source": [ + "## Check directional similarities across the sets\n", + "\n", + "Inspect both query directions as a separate screen. Reciprocal components capture one metric's reciprocal graph; a benchmark can also need chemical, sequence, interface, or one-way similarity criteria." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "read-cross-set-scores", + "metadata": {}, + "outputs": [], + "source": [ + "cross_scores = pd.concat(\n", + " [\n", + " query_ligand_similarity(\n", + " columns=[\"query_ligand_id\", \"target_ligand_id\", \"pocket_qcov\", \"pli_qcov\"],\n", + " filters=[\n", + " (\"query_ligand_id\", \"in\", queries[\"ligand_id\"].tolist()),\n", + " (\"target_ligand_id\", \"in\", targets[\"ligand_id\"].tolist()),\n", + " ],\n", + " release=release,\n", + " )\n", + " for queries, targets in [(training, heldout), (heldout, training)]\n", + " ],\n", + " ignore_index=True,\n", + ")\n", + "cross_scores.sort_values(\"pocket_qcov\", ascending=False).head()" + ] + }, + { + "cell_type": "markdown", + "id": "workshop-reference", + "metadata": {}, + "source": [ + "For the original dataset-selection and similarity exercises, see the [Moving Beyond Memorization workshop](https://github.com/plinder-org/moving_beyond_memorisation). This guide uses the current release tables and API." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/examples/similarity_and_representatives.ipynb b/docs/examples/similarity_and_representatives.ipynb index b52decf4..0631cc97 100644 --- a/docs/examples/similarity_and_representatives.ipynb +++ b/docs/examples/similarity_and_representatives.ipynb @@ -4,9 +4,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Similarity scores and representative covers\n", + "# Query similarity scores\n", "\n", - "Compare ligands and protein interfaces, explore similarity scores, and find representative clusters. {ref}`Similarity scores ` defines the metrics and files." + "Compare ligand chemistry, binding pockets, protein chains, whole interfaces, and individual interface sides. The {ref}`similarity reference ` defines the metrics and units. For diverse selections and component IDs, continue with [Representative covers and components](representatives_and_leakage.ipynb).\n", + "\n", + "Filter by query IDs, score names, and thresholds to narrow score queries and limit downloads." ] }, { @@ -15,15 +17,33 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import os\n", + "from pathlib import Path\n", "\n", "from plinder.core import PlinderRelease, query_table\n", "from plinder.core.scores import (\n", - " reconstruct_interface_similarity_scores,\n", - " reconstruct_similarity_scores,\n", + " map_chain_alignment,\n", + " query_chain_overlap,\n", + " query_half_interface_similarity,\n", + " query_interface_similarity,\n", + " query_ligand_chemical_similarity,\n", + " query_ligand_similarity,\n", + " query_protein_similarity,\n", ")\n", "\n", - "release = PlinderRelease()" + "example_data = os.environ.get(\"PLINDER_SCORE_EXAMPLE_DATA_DIR\")\n", + "release = PlinderRelease(data_dir=Path(example_data)) if example_data else PlinderRelease()\n", + "ligand_id = \"4agi__1__1.R\"\n", + "interface_id = \"7cm8__1__1.A--2.A\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Ligand, pocket, and interaction scores\n", + "\n", + "Ligand-pair rows include pocket coverage, interaction coverage, shape, color, and SuCOS. Request both ligand IDs to inspect one pair, or omit the target filter for its stored neighborhood." ] }, { @@ -32,18 +52,33 @@ "metadata": {}, "outputs": [], "source": [ - "ligand_system_id = \"4agi__1__1.C__1.R\"\n", - "ligand_id = \"4agi__1__1.R\"\n", - "interface_id = \"7cm8__1__1.A--2.A\"" + "ligand_scores = query_ligand_similarity(\n", + " columns=[\n", + " \"query_ligand_id\",\n", + " \"target_ligand_id\",\n", + " \"pocket_qcov\",\n", + " \"pocket_fident_qcov\",\n", + " \"pli_qcov\",\n", + " \"shape\",\n", + " \"color\",\n", + " \"sucos_shape\",\n", + " ],\n", + " filters=[\n", + " (\"query_ligand_id\", \"==\", ligand_id),\n", + " (\"target_ligand_id\", \"==\", ligand_id),\n", + " ],\n", + " release=release,\n", + ")\n", + "ligand_scores" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Compare ligand similarities\n", + "## Chemical similarity\n", "\n", - "`reconstruct_similarity_scores()` reads the files needed for the requested query and target IDs. This example uses a one-pair self-comparison to keep execution small." + "Chemical scores use integer `ligand_smiles_id` values; occurrence-level ligand scores use strings such as `4agi__1__1.R`. ECFP4 Tanimoto describes ligand chemistry independently of the receptor." ] }, { @@ -52,20 +87,29 @@ "metadata": {}, "outputs": [], "source": [ - "ligand_scores = reconstruct_similarity_scores(\n", - " query_system_ids=[ligand_system_id],\n", - " target_system_ids=[ligand_system_id],\n", - " query_ligand_ids=[ligand_id],\n", - " target_ligand_ids=[ligand_id],\n", + "smiles_id = int(\n", + " query_table(\n", + " columns=[\"ligand_smiles_id\"],\n", + " filters=[(\"ligand_id\", \"==\", ligand_id)],\n", + " release=release,\n", + " ).iloc[0, 0]\n", ")\n", - "ligand_scores" + "query_ligand_chemical_similarity(\n", + " filters=[\n", + " (\"query_ligand_id\", \"==\", smiles_id),\n", + " (\"target_ligand_id\", \"==\", smiles_id),\n", + " ],\n", + " release=release,\n", + ").head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "For larger neighborhoods, read selected columns and rows from the complete ligand score table." + "## Protein-chain and monomer scores\n", + "\n", + "Chain scores include query/target coverage, sequence identity, sequence similarity, and Foldseek lDDT as integer percentages. `include_monomers=True` also reads the additional pairs for chains outside the pocket/interface search universe. MMseqs rows have no lDDT." ] }, { @@ -74,33 +118,37 @@ "metadata": {}, "outputs": [], "source": [ - "def read_ligand_neighborhood(system_id, ligand_id):\n", - " return pd.read_parquet(\n", - " release.fetch(\"ligand_similarity_scores\"),\n", - " columns=[\n", - " \"query_system\",\n", - " \"query_ligand_id\",\n", - " \"target_system\",\n", - " \"target_ligand_id\",\n", - " \"pocket_qcov\",\n", - " \"pocket_fident_qcov\",\n", - " \"pli_qcov\",\n", - " \"sucos_shape\",\n", - " ],\n", - " filters=[\n", - " (\"query_system\", \"==\", system_id),\n", - " (\"query_ligand_id\", \"==\", ligand_id),\n", - " ],\n", - " )" + "query_protein_similarity(\n", + " columns=[\n", + " \"query_entry\",\n", + " \"query_chain_mapped\",\n", + " \"target_entry\",\n", + " \"target_chain_mapped\",\n", + " \"source\",\n", + " \"qcov\",\n", + " \"tcov\",\n", + " \"fident\",\n", + " \"seqsim\",\n", + " \"lddt\",\n", + " ],\n", + " filters=[\n", + " (\"query_entry\", \"==\", \"4agi\"),\n", + " (\"query_chain_mapped\", \"==\", \"C\"),\n", + " (\"target_entry\", \"==\", \"4agi\"),\n", + " (\"target_chain_mapped\", \"==\", \"C\"),\n", + " ],\n", + " include_monomers=True,\n", + " release=release,\n", + ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Compare protein-interface similarities\n", + "## Whole-interface scores\n", "\n", - "The interface API likewise accepts selected query and target IDs." + "Whole-interface scores describe both sides of a protein-chain pair. Coverage is directional; swapping query and target can change the result." ] }, { @@ -109,20 +157,22 @@ "metadata": {}, "outputs": [], "source": [ - "interface_scores = reconstruct_interface_similarity_scores(\n", - " query_interface_ids=[interface_id],\n", - " target_interface_ids=[interface_id],\n", - ")\n", - "interface_scores" + "query_interface_similarity(\n", + " filters=[\n", + " (\"query_system\", \"==\", interface_id),\n", + " (\"target_system\", \"==\", interface_id),\n", + " ],\n", + " release=release,\n", + ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Representative covers\n", + "## Half-interface scores\n", "\n", - "Request cover columns through `query_table()` just like annotation columns. {ref}`Representative covers ` describes their construction and long-form files." + "Half-interface IDs end in `::side=1` or `::side=2`. These scores compare one contacting protein surface at a time." ] }, { @@ -131,17 +181,23 @@ "metadata": {}, "outputs": [], "source": [ - "pocket_cover = \"pocket_qcov__70__ligand__directed_set_cover\"\n", - "ligand_cover = query_table(\n", - " columns=[\n", - " \"system_id\",\n", - " \"ligand_id\",\n", - " pocket_cover,\n", - " f\"{pocket_cover}__is_centroid\",\n", + "half_id = f\"{interface_id}::side=1\"\n", + "query_half_interface_similarity(\n", + " filters=[\n", + " (\"query_half_interface_id\", \"==\", half_id),\n", + " (\"target_half_interface_id\", \"==\", half_id),\n", " ],\n", - " filters=[(\"ligand_id\", \"==\", ligand_id)],\n", - ")\n", - "ligand_cover.T" + " release=release,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Count overlapping pocket or interface residues\n", + "\n", + "`query_chain_overlap()` reports overlap counts, total residue counts, and fractions for a pair of label-asym protein chains. Choose `kind=\"pocket\"` or `kind=\"interface\"`. Residues are pooled across that chain's pockets or interfaces." ] }, { @@ -150,15 +206,59 @@ "metadata": {}, "outputs": [], "source": [ - "interface_cover = query_table(\n", - " \"interface_annotations\",\n", - " columns=[\n", - " \"system_id\",\n", - " \"interface_qcov__70__directed_set_cover\",\n", - " ],\n", - " filters=[(\"system_id\", \"==\", interface_id)],\n", + "query_chain_overlap(\n", + " \"4agi\",\n", + " \"C\",\n", + " \"4agi\",\n", + " \"C\",\n", + " kind=\"pocket\",\n", + " source=\"mmseqs\",\n", + " release=release,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query_chain_overlap(\n", + " \"7cm8\",\n", + " \"A\",\n", + " \"7cm8\",\n", + " \"A\",\n", + " kind=\"interface\",\n", + " source=\"mmseqs\",\n", + " release=release,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Map an alignment to sequence or coordinates\n", + "\n", + "`map_chain_alignment()` expands stored CIGARs. MMseqs positions are one-based SEQRES positions; Foldseek positions are zero-based indices over resolved coordinate residues. These calls fetch the optional CIGAR shard for the query entry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "positions = map_chain_alignment(\n", + " \"4agi\",\n", + " \"C\",\n", + " \"4agi\",\n", + " \"C\",\n", + " source=\"mmseqs\",\n", + " include_gaps=False,\n", + " release=release,\n", ")\n", - "interface_cover.T" + "positions.head()" ] }, { @@ -167,7 +267,7 @@ "source": [ "## Matched molecular pairs\n", "\n", - "Use `ligand_mmp_pairs` to look up transformations involving the selected ligand chemistry." + "Look up chemical transformations and their shared cores with `ligand_mmp_pairs`." ] }, { @@ -176,19 +276,25 @@ "metadata": {}, "outputs": [], "source": [ - "smiles_id = query_table(\n", - " columns=[\"ligand_smiles_id\"],\n", - " filters=[(\"ligand_id\", \"==\", ligand_id)],\n", - ").iloc[0, 0]\n", "mmp_neighbors = query_table(\n", " \"ligand_mmp_pairs\",\n", - " filters=[[\n", - " (\"ligand_smiles_id_1\", \"==\", smiles_id),\n", - " (\"ligand_smiles_id_2\", \"==\", smiles_id),\n", - " ]],\n", + " filters=[\n", + " [\n", + " (\"ligand_smiles_id_1\", \"==\", smiles_id),\n", + " (\"ligand_smiles_id_2\", \"==\", smiles_id),\n", + " ]\n", + " ],\n", + " release=release,\n", ")\n", "mmp_neighbors.head()" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For selected comparisons calculated from residue mappings, the API also provides `reconstruct_similarity_scores()` and `reconstruct_interface_similarity_scores()`. The query functions above read the published score tables directly." + ] } ], "metadata": { diff --git a/pyproject.toml b/pyproject.toml index bfef683e..487fffc4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,7 +2,7 @@ name = "plinder" dynamic = ["version"] dependencies = [ - "biotite >= 1.2", + "biotite >= 1.2, < 2", "numpy", "threadpoolctl>=3.6", "pandas", @@ -13,7 +13,7 @@ dependencies = [ "rdkit>=2025.09.1", "pyarrow", "omegaconf", - "duckdb>=1.5", + "duckdb>=1.5,<2", "cloudpathlib>=0.25,<0.26", ] description = "PLINDER: Protein & Ligand INteraction Dataset and Evaluation Resource" diff --git a/scripts/prepare_docs_score_release.py b/scripts/prepare_docs_score_release.py new file mode 100644 index 00000000..1e9c9f72 --- /dev/null +++ b/scripts/prepare_docs_score_release.py @@ -0,0 +1,174 @@ +"""Extract real, bounded release rows used by the two score/cluster guides.""" + +import argparse +import hashlib +import json +from pathlib import Path + +import duckdb +import pyarrow.parquet as pq + +parser = argparse.ArgumentParser(description=__doc__) +parser.add_argument("source", type=Path) +parser.add_argument("output", type=Path) +args = parser.parse_args() +con = duckdb.connect() +con.execute("SET threads=2") +con.execute("SET memory_limit='4GB'") +records = [] + + +def read(relative, predicate, parameters): + path = args.source / relative + glob = str(path / "**/*.parquet") if path.is_dir() else str(path) + if relative == "exports/protein_similarity_scores": + glob = sorted( + { + str(file) + for entry in entries + for file in path.glob(f"alignment_type=*/shard={entry[1:3]}.parquet") + } + ) + return con.execute( + "SELECT * FROM read_parquet(?, union_by_name=true, hive_partitioning=true) WHERE " + + predicate, + [glob, *parameters], + ).to_arrow_table() + + +def save(relative, table, selection): + output = args.output / relative + output.parent.mkdir(parents=True, exist_ok=True) + pq.write_table(table, output, compression="zstd") + records.append( + { + "path": relative, + "rows": table.num_rows, + "selection": selection, + "sha256": hashlib.sha256(output.read_bytes()).hexdigest(), + } + ) + print(relative, table.num_rows, output.stat().st_size, flush=True) + + +entries = {"4agi", "4agc", "7eek", "7cm8"} +cover_path = ( + "ligand_sampling/directed_set_cover/metric=pocket_qcov/threshold=70.parquet" +) +cover = read(cover_path, "ligand_id = ?", ["4agi__1__1.R"]) +assert cover.num_rows +centroids = cover.column("centroid_ligand_id").to_pylist() +entries.update(value.split("__")[0] for value in centroids) +save(cover_path, cover, "4agi__1__1.R assignment at pocket_qcov=70") +annotation = read( + "index/annotation_table.parquet", "entry_pdb_id IN ?", [sorted(entries)] +) +save( + "index/annotation_table.parquet", + annotation, + "example entries and cover representative entries", +) +ligands = annotation.column("ligand_id").to_pylist() +systems = annotation.column("system_id").to_pylist() +smiles = sorted( + set(v for v in annotation.column("ligand_smiles_id").to_pylist() if v is not None) +) +interfaces = read( + "index/interface_annotation_table.parquet", "entry_pdb_id IN ?", [sorted(entries)] +) +save("index/interface_annotation_table.parquet", interfaces, "example entries") +systems.extend(interfaces.column("system_id").to_pylist()) +for name, column, values in [ + ("entry_metadata", "entry_pdb_id", sorted(entries)), + ("entry_sources", "entry_pdb_id", sorted(entries)), + ("entry_chains", "entry_pdb_id", sorted(entries)), + ("entry_biounit_chains", "entry_pdb_id", sorted(entries)), + ("system_validation", "system_id", systems), + ("ligand_clusters", "ligand_id", ligands), + ("ligand_affinity", "ligand_id", ligands), + ("ligand_pocket_membership", "ligand_id", ligands), + ("interface_clusters", "system_id", systems), + ("alignment_chain_lookup", "entry_pdb_id", sorted(entries)), + ("ligand_pocket_residues", "entry_pdb_id", sorted(entries)), +]: + relative = f"index/{name}.parquet" + save( + relative, + read(relative, f'"{column}" IN ?', [values]), + f"{column} in example subset", + ) +for name in ("sequence", "structure"): + relative = f"protein_clusters/{name}.parquet" + save( + relative, + read(relative, "entry_pdb_id IN ?", [sorted(entries)]), + "example entries", + ) +relative = "index/ligand_mmp_pairs.parquet" +save( + relative, + read( + relative, "ligand_smiles_id_1 IN ? OR ligand_smiles_id_2 IN ?", [smiles, smiles] + ), + "either member in example SMILES IDs", +) +for name, column, values in [ + ("ligand_similarity_scores", "ligand_id", ligands), + ("interface_similarity_scores", "system", systems), + ( + "interface_half_similarity_scores", + "half_interface_id", + [f"{s}::side={side}" for s in systems for side in (1, 2)], + ), +]: + relative = f"exports/{name}.parquet" + save( + relative, + read( + relative, + f'"query_{column}" IN ? AND "target_{column}" IN ?', + [values, values], + ), + "both endpoints in example subset", + ) +save( + "ligand_scores/retained.parquet", + read( + "ligand_scores", + "query_ligand_id IN ? AND target_ligand_id IN ?", + [smiles, smiles], + ), + "both endpoints in example SMILES IDs", +) +for name in ("protein_similarity_scores", "monomer_similarity_scores"): + save( + f"exports/{name}/examples.parquet", + read( + f"exports/{name}", + "query_entry IN ? AND target_entry IN ?", + [sorted(entries), sorted(entries)], + ), + "both endpoints in example entries", + ) +for entry in ("4agi", "7cm8"): + relative = ( + "alignment_cigars/search_db=holo/alignment_type=mmseqs/" + f"shard={entry[1:3]}.parquet" + ) + save( + relative, + read(relative, "query_entry = ? AND target_entry = ?", [entry, entry]), + f"{entry} self alignments", + ) +metadata = { + "release": "2026-09", + "description": "Real release rows, not synthetic scores. Bounded integration-test input for the score and representative notebooks, not a standalone release or benchmark.", + "entries": sorted(entries), + "files": records, +} +(args.output / "manifest.json").write_text(json.dumps(metadata, indent=2) + "\n") +print( + "TOTAL BYTES", + sum((args.output / r["path"]).stat().st_size for r in records), + flush=True, +) diff --git a/scripts/update_readme_counts.py b/scripts/update_readme_counts.py new file mode 100644 index 00000000..b090e2fc --- /dev/null +++ b/scripts/update_readme_counts.py @@ -0,0 +1,91 @@ +"""Regenerate the marked README counts table from a local release.""" + +import argparse +from pathlib import Path + +import duckdb + +START = "" +END = "" + + +def release_counts(release_dir: Path) -> str: + """Count ligand records and receptor-polymer composition per assembly.""" + with duckdb.connect(config={"threads": 2, "memory_limit": "2GB"}) as con: + for name in ( + "annotation_table", + "interface_annotation_table", + "entry_chains", + "entry_biounit_chains", + ): + con.read_parquet( + str(release_dir / "index" / f"{name}.parquet") + ).create_view(name) + systems, ligands = con.sql(""" + SELECT count(DISTINCT system_id), + count(*) FILTER (WHERE ligand_is_proper) + FROM annotation_table + """).fetchone() + interfaces = con.sql( + "SELECT count(*) FROM interface_annotation_table" + ).fetchone()[0] + protein, dna, rna, hybrid, complexes = con.sql(""" + WITH assemblies AS ( + SELECT b.entry_pdb_id, b.biounit_id, + count(*) FILTER (WHERE c.chain_receptor_type = 'protein') AS proteins, + count(*) FILTER (WHERE c.chain_receptor_type = 'dna') AS dna, + count(*) FILTER (WHERE c.chain_receptor_type = 'rna') AS rna, + count(*) FILTER (WHERE c.chain_receptor_type = 'dna+rna') AS hybrid + FROM entry_biounit_chains b + JOIN entry_chains c USING (entry_pdb_id, chain_asym_id) + WHERE b.chain_role = 'receptor' + GROUP BY 1, 2 + ) + SELECT + count(*) FILTER (WHERE proteins = 1 AND dna + rna + hybrid = 0), + count(*) FILTER (WHERE proteins = 0 AND dna = 1 AND rna + hybrid = 0), + count(*) FILTER (WHERE proteins = 0 AND rna = 1 AND dna + hybrid = 0), + count(*) FILTER (WHERE proteins = 0 AND hybrid = 1 AND dna + rna = 0), + count(*) FILTER (WHERE proteins > 0 AND dna + rna + hybrid > 0) + FROM assemblies + """).fetchone() + rows = [ + ("Ligand systems", systems), + ("Proper ligands", ligands), + ("Protein–protein interfaces", interfaces), + ("Protein-monomer assemblies", protein), + ("Nucleic-acid-monomer assemblies", dna + rna + hybrid), + ("Protein–nucleic-acid assemblies", complexes), + ] + return "\n".join( + [ + START, + "| Release contents | Count |", + "| --- | ---: |", + *(f"| {label} | {count:,} |" for label, count in rows), + "", + f"Nucleic-acid monomers: {dna:,} DNA, {rna:,} RNA, and {hybrid:,} DNA/RNA hybrids.", + END, + ] + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("release_dir", type=Path) + parser.add_argument( + "--readme", type=Path, help="Update this README; otherwise print the table." + ) + args = parser.parse_args() + block = release_counts(args.release_dir) + if args.readme: + text = args.readme.read_text() + start = text.index(START) + end = text.index(END, start) + len(END) + args.readme.write_text(text[:start] + block + text[end:]) + else: + print(block) + + +if __name__ == "__main__": + main() diff --git a/src/plinder/core/index/query.py b/src/plinder/core/index/query.py index af779daa..54eb4fe7 100644 --- a/src/plinder/core/index/query.py +++ b/src/plinder/core/index/query.py @@ -299,6 +299,9 @@ def query_table( ``chain_sequence`` to include sequences automatically. When starting from ``entry_chains``, choose the cluster table with ``joins=[...]``; both cluster tables use the same representative column names. + + Per-ligand ``ligand_validation_*`` and ``ligand_pocket_validation_*`` + fields are read directly from ``annotation``; no system table is required. """ if table_name not in RELEASE_TABLES: choices = ", ".join(sorted(RELEASE_TABLES)) diff --git a/src/plinder/core/index/utils.py b/src/plinder/core/index/utils.py index f5a1aed0..a4921f4a 100644 --- a/src/plinder/core/index/utils.py +++ b/src/plinder/core/index/utils.py @@ -35,6 +35,7 @@ "ligand_similarity_scores", "interface_similarity_scores", "interface_half_similarity_scores", + "score_store", ), True, ), @@ -104,10 +105,15 @@ def download_plinder_cmd(args: list[str] | None = None) -> None: f"bulk download cannot resolve parameterized artifact " f"{artifact_name}" ) - cpl.get_plinder_path( - rel=artifact_path, - force_progress=large_download, - ) + try: + cpl.get_plinder_path( + rel=artifact_path, + force_progress=large_download, + ) + except FileNotFoundError: + if artifact_name != "score_store": + raise + LOG.info("this release has no partitioned score exports yet") t1 = time() total = t1 - t0 diff --git a/src/plinder/core/release.py b/src/plinder/core/release.py index 648385cf..a63e653d 100644 --- a/src/plinder/core/release.py +++ b/src/plinder/core/release.py @@ -40,6 +40,8 @@ def _release_file( RELEASE_PATHS = { + "score_store": "score_store", + "score_store_manifest": "score_store/{artifact}/manifest.json", "annotation_table": "index/annotation_table.parquet", "system_validation": "index/system_validation.parquet", "entry_chains": "index/entry_chains.parquet", diff --git a/src/plinder/core/scores/custom.py b/src/plinder/core/scores/custom.py index 7492e8ee..c9cf304e 100644 --- a/src/plinder/core/scores/custom.py +++ b/src/plinder/core/scores/custom.py @@ -159,11 +159,12 @@ class CustomScoringResult: aligned_pocket_residues: Path | None ligand_scores: Path | None interface_scores: Path | None + interface_side_scores: Path | None = None @dataclass(frozen=True) class CustomSequenceScoringResult: - """Files produced by protein-sequence scoring against PLINDER pockets.""" + """Files produced by protein-sequence search and binding-site discovery.""" query_inputs: CustomQueryInputs query_databases: CustomQueryDatabases @@ -174,6 +175,7 @@ class CustomSequenceScoringResult: aligned_pocket_residues: Path | None sequence_links: Path best_sequence_links: Path + interface_side_scores: Path | None = None @dataclass(frozen=True) @@ -2804,8 +2806,9 @@ def score_custom_sequence_file( threads: int = 1, store_aligned_pocket_residues: bool = False, include_monomers: bool = False, + include_interfaces: bool = True, ) -> CustomSequenceScoringResult: - """Search protein sequences and score PLINDER ligand-pocket identity. + """Search protein sequences and map known pockets and interface sites. ``plinder_entry_ids`` builds a small MMseqs target from the scoreable receptor and interface chains in the selected release entries instead of @@ -2911,6 +2914,16 @@ def score_custom_sequence_file( output_path=work_dir / "sequence_links.parquet", best_output_path=work_dir / "best_sequence_links.parquet", ) + interface_side_path = ( + work_dir / "interface_side_scores.parquet" if include_interfaces else None + ) + if interface_side_path is not None: + calculate_custom_interface_side_scores( + protein_score_alignments, + assets=assets, + output_path=interface_side_path, + chain_manifest=query_inputs.chain_manifest, + ) return CustomSequenceScoringResult( query_inputs=query_inputs, query_databases=query_databases, @@ -2921,6 +2934,7 @@ def score_custom_sequence_file( aligned_pocket_residues=aligned_pocket_residue_path, sequence_links=sequence_links, best_sequence_links=best_sequence_links, + interface_side_scores=interface_side_path, ) @@ -3014,6 +3028,47 @@ def calculate_custom_similarity_scores( return result +def calculate_custom_interface_side_scores( + protein_score_alignments: Mapping[str, Path], + *, + assets: CustomScoringAssets, + output_path: Path, + chain_manifest: Path | None = None, +) -> pd.DataFrame: + """Discover known interface sites on custom structures or FASTA sequences.""" + from plinder.core.scores.interface import calculate_interface_side_scores + + entries = _load_release_entry_views( + assets, pdb_ids=_query_entry_ids(protein_score_alignments) + ) + frames = [pd.read_parquet(path) for path in protein_score_alignments.values()] + result = calculate_interface_side_scores( + pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(), + interfaces={ + interface.id: interface + for entry in entries.values() + for interface in entry.interfaces.values() + }, + ) + if chain_manifest is not None: + identifiers = pd.read_parquet( + chain_manifest, columns=["structure_id", "sequence_id"] + ).drop_duplicates() + result = result.merge( + identifiers, + left_on="custom_structure_id", + right_on="structure_id", + how="left", + validate="many_to_one", + ).drop(columns="structure_id") + if result["sequence_id"].isna().any(): + raise ValueError("interface-side hits reference unknown FASTA sequences") + output_path = Path(output_path) + output_path.parent.mkdir(parents=True, exist_ok=True) + result.to_parquet(output_path, index=False, compression="zstd") + return result + + def calculate_custom_interface_similarity_scores( score_alignments: Mapping[str, Path], *, @@ -3106,9 +3161,12 @@ def score_custom_cif_files( scores use each PLINDER ligand pocket as the directed query and each custom protein chain as a ligand-free target, so ``include_ligands=False`` still yields protein metrics and ``pocket_fident``. Passing ``None`` for - ligand or interface inclusion annotates that feature and emits its score - table only when the custom structures contain a proper ligand or protein - interface, respectively. ``plinder_entry_ids`` builds a small MMseqs target + ligand inclusion annotates that feature and emits its score table only + when the custom structures contain a proper ligand. Interface-site scores + map known sites onto each custom chain, even without a binding partner; + complete-interface scores require an interface in the custom structure. + Set ``include_interfaces=False`` to skip both interface outputs. + ``plinder_entry_ids`` builds a small MMseqs target from scoreable receptor and interface chains in selected release entries rather than fetching the complete search database; this bounded mode currently requires ``backends=("mmseqs",)``. @@ -3245,8 +3303,21 @@ def score_custom_cif_files( score_interfaces = include_interfaces is True or ( include_interfaces is None and has_interfaces ) + interface_side_path = ( + work_dir / "interface_side_scores.parquet" + if include_interfaces is not False + else None + ) + if interface_side_path is not None: + calculate_custom_interface_side_scores( + protein_score_alignments, + assets=assets, + output_path=interface_side_path, + ) interface_score_path = ( - work_dir / "interface_scores.parquet" if score_interfaces else None + work_dir / "interface_scores.parquet" + if score_interfaces and has_interfaces + else None ) if interface_score_path is not None: calculate_custom_interface_similarity_scores( @@ -3267,4 +3338,5 @@ def score_custom_cif_files( aligned_pocket_residues=aligned_pocket_residue_path, ligand_scores=ligand_score_path, interface_scores=interface_score_path, + interface_side_scores=interface_side_path, ) diff --git a/src/plinder/core/scores/interface.py b/src/plinder/core/scores/interface.py index b0174ace..efb42ec3 100644 --- a/src/plinder/core/scores/interface.py +++ b/src/plinder/core/scores/interface.py @@ -4,12 +4,128 @@ from __future__ import annotations +from collections import defaultdict +from collections.abc import Mapping + import pandas as pd from plinder.core.release import PlinderRelease -from plinder.core.scores.query import Filters, read_score_table +from plinder.core.scores.entries import InterfaceView +from plinder.core.scores.mapping import unpack_residue_identities +from plinder.core.scores.query import Filters, query_score_table from plinder.core.utils.dec import timeit +INTERFACE_SIDE_COLUMNS = [ + "custom_structure_id", + "custom_chain", + "plinder_interface_id", + "plinder_chain", + "plinder_partner_chain", + "source", + "interface_side_qcov", + "interface_side_fident", + "plinder_residue_numbers", + "custom_residue_numbers", +] + + +def calculate_interface_side_scores( + alignments: pd.DataFrame, + *, + interfaces: Mapping[str, InterfaceView], +) -> pd.DataFrame: + """Map known interface sides onto custom chains, without requiring a partner. + + Alignments run from the release chain to the custom chain. Coverage and + identity are integer percentages of *all* residues in the known interface + side, so gaps and unaligned residues contribute zero. Each backend retains + its best single alignment per interface side and custom chain; alignments + are never combined to inflate coverage. + """ + sides: defaultdict[tuple[str, str], list[tuple[str, str, str, frozenset[int]]]] = ( + defaultdict(list) + ) + for interface in interfaces.values(): + for chain, partner, residues in ( + ( + interface.chain_1, + interface.chain_2, + interface.chain_1_residue_number_to_index, + ), + ( + interface.chain_2, + interface.chain_1, + interface.chain_2_residue_number_to_index, + ), + ): + if residues: + sides[(interface.pdb_id, chain.split(".", 1)[-1])].append( + (interface.id, chain, partner, frozenset(residues)) + ) + best: dict[ + tuple[str, str, str, str, str], tuple[tuple[int, int], dict[str, object]] + ] = {} + for row in alignments.itertuples(index=False): + matching = sides.get((str(row.query_entry), str(row.query_chain_mapped)), []) + if not matching: + continue + release_numbers = list(row.query_selected_residue_numbers) + custom_numbers = list(row.target_selected_residue_numbers) + identities = unpack_residue_identities( + row.selected_residue_identity_bits, len(release_numbers) + ) + pairs = { + int(release_number): (int(custom_number), identical) + for release_number, custom_number, identical in zip( + release_numbers, custom_numbers, identities, strict=True + ) + if int(custom_number) > 0 + } + for interface_id, chain, partner, side_residues in matching: + numbers = sorted(side_residues.intersection(pairs)) + if not numbers: + continue + identical_count = sum(pairs[number][1] for number in numbers) + key = ( + str(row.target_entry), + str(row.target_chain_mapped), + interface_id, + chain, + str(row.source), + ) + rank = (len(numbers), identical_count) + record: dict[str, object] = { + "custom_structure_id": key[0], + "custom_chain": key[1], + "plinder_interface_id": interface_id, + "plinder_chain": chain, + "plinder_partner_chain": partner, + "source": key[4], + "interface_side_qcov": int( + 100 * len(numbers) / len(side_residues) + 0.5 + ), + "interface_side_fident": int( + 100 * identical_count / len(side_residues) + 0.5 + ), + "plinder_residue_numbers": numbers, + "custom_residue_numbers": [pairs[number][0] for number in numbers], + } + if key not in best or rank > best[key][0]: + best[key] = (rank, record) + result = pd.DataFrame( + [record for _, record in best.values()], columns=INTERFACE_SIDE_COLUMNS + ) + return result.sort_values( + [ + "interface_side_qcov", + "interface_side_fident", + *INTERFACE_SIDE_COLUMNS[:4], + "source", + ], + ascending=[False, False, True, True, True, True, True], + ignore_index=True, + ) + @timeit def query_interface_similarity( @@ -19,8 +135,12 @@ def query_interface_similarity( release: PlinderRelease | None = None, ) -> pd.DataFrame: """Query complete directed protein-interface similarities.""" - dataset = (release or PlinderRelease()).fetch("interface_similarity_scores") - return read_score_table(dataset, columns=columns, filters=filters) + return query_score_table( + "interface_similarity_scores", + release=release or PlinderRelease(), + columns=columns, + filters=filters, + ) @timeit @@ -31,8 +151,12 @@ def query_half_interface_similarity( release: PlinderRelease | None = None, ) -> pd.DataFrame: """Query directed coverage between individual interface sides.""" - dataset = (release or PlinderRelease()).fetch("interface_half_similarity_scores") - return read_score_table(dataset, columns=columns, filters=filters) + return query_score_table( + "interface_half_similarity_scores", + release=release or PlinderRelease(), + columns=columns, + filters=filters, + ) __all__ = ["query_interface_similarity", "query_half_interface_similarity"] diff --git a/src/plinder/core/scores/ligand.py b/src/plinder/core/scores/ligand.py index 75decfdb..4a9cc8c5 100644 --- a/src/plinder/core/scores/ligand.py +++ b/src/plinder/core/scores/ligand.py @@ -8,7 +8,7 @@ from plinder.core.index.query import query_table from plinder.core.release import PlinderRelease -from plinder.core.scores.query import Filter, Filters, read_score_table +from plinder.core.scores.query import Filter, Filters, query_score_table from plinder.core.utils.dec import timeit @@ -44,9 +44,9 @@ def query_ligand_similarity( df : pd.DataFrame The ligand similarity results. """ - dataset = (release or PlinderRelease()).fetch("ligand_similarity_scores") - return read_score_table( - dataset, + return query_score_table( + "ligand_similarity_scores", + release=release or PlinderRelease(), columns=columns, filters=filters, ) @@ -60,8 +60,12 @@ def query_ligand_chemical_similarity( release: PlinderRelease | None = None, ) -> pd.DataFrame: """Query ECFP4 Tanimoto similarities between canonical ligand SMILES.""" - dataset = (release or PlinderRelease()).fetch("ligand_scores") - return read_score_table(dataset, columns=columns, filters=filters) + return query_score_table( + "ligand_scores", + release=release or PlinderRelease(), + columns=columns, + filters=filters, + ) @timeit @@ -136,7 +140,6 @@ def cross_ligand_chemical_similarity( """ query_ids = _ligand_ids(query_ligands) target_ids = _ligand_ids(target_ligands) - dataset = (release or PlinderRelease()).fetch("ligand_scores") if metric is None: metric = "tanimoto_similarity_ecfp4_1024" filters: list[list[Filter]] = [ @@ -150,8 +153,9 @@ def cross_ligand_chemical_similarity( ], ] columns = ["query_ligand_id", "target_ligand_id", metric] - similarities = read_score_table( - dataset, + similarities = query_score_table( + "ligand_scores", + release=release or PlinderRelease(), columns=columns, filters=filters, ) diff --git a/src/plinder/core/scores/protein.py b/src/plinder/core/scores/protein.py index e1079654..91fce926 100644 --- a/src/plinder/core/scores/protein.py +++ b/src/plinder/core/scores/protein.py @@ -14,7 +14,7 @@ decode_cigar_residue_positions, unpack_residue_identities, ) -from plinder.core.scores.query import Filters, read_score_table +from plinder.core.scores.query import Filters, query_score_table from plinder.core.utils.dec import timeit OVERLAP_COLUMNS = [ @@ -47,11 +47,16 @@ def query_protein_similarity( ligand pockets and protein interfaces. """ selected_release = release or PlinderRelease() - paths = [selected_release.fetch("protein_similarity_scores")] + artifacts = ["protein_similarity_scores"] if include_monomers: - paths.append(selected_release.fetch("monomer_similarity_scores")) + artifacts.append("monomer_similarity_scores") return pd.concat( - [read_score_table(path, columns=columns, filters=filters) for path in paths], + [ + query_score_table( + artifact, release=selected_release, columns=columns, filters=filters + ) + for artifact in artifacts + ], ignore_index=True, ) diff --git a/src/plinder/core/scores/query.py b/src/plinder/core/scores/query.py index bcae68f7..630dbcee 100644 --- a/src/plinder/core/scores/query.py +++ b/src/plinder/core/scores/query.py @@ -9,6 +9,9 @@ import pandas as pd +from plinder.core.release import PlinderRelease +from plinder.core.scores.store import read_partitioned_scores + Filter: TypeAlias = tuple[str, str, Any] Filters: TypeAlias = list[Filter] | list[list[Filter]] | None @@ -25,4 +28,22 @@ def read_score_table( return pd.read_parquet(dataset, columns=columns, filters=filters) +def query_score_table( + artifact: str, + *, + release: PlinderRelease, + columns: list[str] | None = None, + filters: Filters = None, +) -> pd.DataFrame: + """Route filtered reads before downloading, with legacy-release fallback.""" + if not filters or any(not branch for branch in filters): + raise ValueError("at least one score filter is required") + result = read_partitioned_scores( + release, artifact, columns=columns, filters=filters + ) + if result is not None: + return result + return read_score_table(release.fetch(artifact), columns=columns, filters=filters) + + __all__ = ["Filter", "Filters", "read_score_table"] diff --git a/src/plinder/core/scores/search.py b/src/plinder/core/scores/search.py index 9e845e18..3ad7ed7e 100644 --- a/src/plinder/core/scores/search.py +++ b/src/plinder/core/scores/search.py @@ -30,7 +30,7 @@ def search( *, output_dir: str | Path, release: PlinderRelease | None = None, - mode: Literal["auto", "pockets", "ligands", "interfaces", "both"] = "auto", + mode: Literal["auto", "all", "pockets", "ligands", "interfaces"] = "auto", backends: Iterable[str] | None = None, threads: int = 1, search_config: CustomProteinSearchConfig | None = None, @@ -44,12 +44,20 @@ def search( A receptor and its SDFs share an input ID; paths are relative to the table. Complete mmCIFs use an empty ligand_path. ``reference_id`` is ignored here. - For ligand-free queries, pass a FASTA, one PDB/mmCIF, or a folder of those - structures. FASTA searches use MMseqs; structure searches use Foldseek and - MMseqs by default. ``mode='pockets'`` compares protein/pocket features, - ``'ligands'`` includes ligand similarities, and ``'interfaces'`` includes - protein interfaces. ``'both'`` includes both; ``'auto'`` uses applicable - features (ligand comparisons require a table). + Pass a FASTA, one PDB/mmCIF, a structure folder, or a table pairing receptors + with ligand SDFs. FASTA searches use MMseqs2; structures use Foldseek and + MMseqs2 by default. Every search returns protein-chain hits. + + ``'pockets'`` maps PLINDER ligand pockets onto similar regions of your + proteins, ignoring query ligands. ``'ligands'`` also compares query ligands + and their interactions. ``'interfaces'`` maps PLINDER interface sites onto + similar regions of each input chain, and also compares complete interfaces + when the input contains interacting chains. + ``'auto'`` (the default) runs the comparisons supported by the input; + ``'all'`` selects all feature families, skipping unavailable comparisons. + FASTA supports + pocket and interface-site discovery, but not ligand or complete-interface + comparisons. PDB receptors contribute amino-acid residues, including modified amino acids; other PDB components are omitted. Supply ligand poses through ``ligand_path``. @@ -61,14 +69,19 @@ def search( With ``include_monomers=True``, chain hits also cover proteins outside ligand pockets and protein interfaces. Pocket and interface scores still use their own targets. + ``interface_side_scores`` locates potential interface sites on input chains; + its coverage and identity percentages use the entire known interface side + as the denominator. ``interface_scores`` compares two-chain interfaces and + is absent for monomer inputs. Site matches do not establish that a protein + binds the partner observed in the matched PLINDER interface. ``search_config`` controls search filters. Its shared ``evalue`` defaults to 0.01; ``foldseek_evalue``, ``mmseqs_evalue`` and ``steam_evalue`` override it independently for their respective backends. Input IDs become custom structure IDs in these outputs. Generated ligand chain IDs map to the original SDF paths in ``inputs/.ligands.tsv``. """ - if mode not in {"auto", "pockets", "ligands", "interfaces", "both"}: - raise ValueError("mode must be auto, pockets, ligands, interfaces or both") + if mode not in {"auto", "pockets", "ligands", "interfaces", "all"}: + raise ValueError("mode must be auto, pockets, ligands, interfaces or all") output_dir = Path(output_dir).resolve() common: dict[str, Any] = dict( work_dir=output_dir, @@ -83,21 +96,20 @@ def search( if path is not None and path.name.lower().endswith( (".fasta", ".fa", ".faa", ".fasta.gz", ".fa.gz", ".faa.gz") ): - if mode not in {"auto", "pockets"}: + if mode == "ligands": raise ValueError( - "FASTA inputs support pocket search; use structures for ligand/interface comparisons" + "FASTA inputs support pocket and interface-site discovery; use structures for ligand comparisons" ) return score_custom_sequence_file( path, backends=tuple(backends) if backends is not None else ("mmseqs",), + include_interfaces=mode in {"auto", "interfaces", "all"}, **common, ) is_table = path is None or path.suffix.lower() in {".csv", ".tsv", ".parquet"} if is_table: structures = read_structure_table(inputs) else: - if mode in {"ligands", "both"}: - raise ValueError("Use an input table for ligand searches") assert path is not None rows = [] for file in find_structure_files(path): @@ -108,7 +120,7 @@ def search( if len({row["input_id"] for row in rows}) != len(rows): raise ValueError("Structure filenames must have unique basenames") structures = read_structure_table(pd.DataFrame(rows)) - include_ligands = is_table and mode in {"auto", "ligands", "both"} + include_ligands = mode in {"auto", "ligands", "all"} sources = [] for input_id, structure in structures.items(): original = Path(structure.coordinates) @@ -130,7 +142,7 @@ def search( backends=tuple(backends) if backends is not None else (("mmseqs",) if plinder_entry_ids is not None else CIF_SEARCH_BACKENDS), - include_ligands=None if mode == "auto" and include_ligands else include_ligands, - include_interfaces=None if mode == "auto" else mode in {"interfaces", "both"}, + include_ligands=None if mode in {"auto", "all"} else include_ligands, + include_interfaces=mode in {"auto", "interfaces", "all"}, **common, ) diff --git a/src/plinder/core/scores/store.py b/src/plinder/core/scores/store.py new file mode 100644 index 00000000..89478082 --- /dev/null +++ b/src/plinder/core/scores/store.py @@ -0,0 +1,164 @@ +# Copyright (c) 2024, Plinder Development Team +# Distributed under the terms of the Apache License 2.0 +"""Manifest-directed downloads for partitioned score exports. + +Only positive equality/membership predicates are used for routing. All original +predicates are still applied by Arrow, including every branch of an OR filter. +""" + +from __future__ import annotations + +import base64 +import hashlib +import json +import re +from pathlib import Path +from typing import Any + +import pandas as pd +import pyarrow as pa +import pyarrow.dataset as ds +import pyarrow.parquet as pq + +from plinder.core.release import PlinderRelease +from plinder.core.utils import cpl + +# Query and target aliases that share the same entry (or numeric-ID) bucket. +SCORE_KEYS: dict[str, tuple[tuple[str, ...], tuple[str, ...]]] = { + "ligand_similarity_scores": ( + ("query_system", "query_ligand_id"), + ("target_system", "target_ligand_id"), + ), + "interface_similarity_scores": (("query_system",), ("target_system",)), + "interface_half_similarity_scores": ( + ("query_half_interface_id",), + ("target_half_interface_id",), + ), + "protein_similarity_scores": (("query_entry",), ("target_entry",)), + "monomer_similarity_scores": (("query_entry",), ("target_entry",)), + "ligand_scores": (("query_ligand_id",), ("target_ligand_id",)), +} + + +def score_bucket(value: Any, *, numeric: bool = False) -> str: + """Stable entry-prefix buckets; numeric ligand IDs use ranges of 1024.""" + if numeric: + return f"n{int(value) // 1024}" + text = str(value) + if re.match(r"^[a-zA-Z0-9]{4}(?:__|$)", text): + return "p" + text[1:3].lower() + return "h" + hashlib.sha256(text.split("__", 1)[0].encode()).hexdigest()[:2] + + +def _file(root: Path, relative: str, release: PlinderRelease) -> Path: + path = Path(relative) + if ( + path.is_absolute() + or "\\" in relative + or any(part in {"", ".", ".."} for part in relative.split("/")) + ): + raise ValueError(f"unsafe score-store path: {relative!r}") + local = root / path + local.resolve().relative_to(root.resolve()) + return local + + +def _files(root: Path, relatives: list[str], release: PlinderRelease) -> list[Path]: + paths = [_file(root, relative, release) for relative in relatives] + if release.data_dir is None: + cpl.download_paths(paths=paths) + for path in paths: + if not path.is_file(): + raise FileNotFoundError(f"missing score-store file: {path}") + return paths + + +def _to_pandas(table: pa.Table, partition_dictionaries: dict[str, str]) -> pd.DataFrame: + """Preserve source Hive categories, including those absent from the rows.""" + result = table.to_pandas() + for name, encoded in partition_dictionaries.items(): + if name not in result: + continue + dictionary = pa.ipc.open_stream(base64.b64decode(encoded)).read_all() + result[name] = pd.Categorical( + result[name], + categories=dictionary.column(0).to_pandas().rename(None), + ordered=table.schema.field(name).type.ordered, + ) + return result + + +def read_partitioned_scores( + release: PlinderRelease, + artifact: str, + *, + columns: list[str] | None, + filters: Any, +) -> pd.DataFrame | None: + """Return None only when this release has no partitioned export yet.""" + try: + manifest_path = release.fetch("score_store_manifest", artifact=artifact) + except FileNotFoundError: + return None + manifest = json.loads(manifest_path.read_text()) + if manifest.get("version") != 1 or manifest.get("artifact") != artifact: + raise ValueError(f"unsupported score-store manifest: {manifest_path}") + schema = pa.ipc.read_schema(pa.BufferReader(base64.b64decode(manifest["schema"]))) + partition_dictionaries = manifest.get("partition_dictionaries", {}) + # Bind filters/columns against the schema even if no shard matches. + expression = pq.filters_to_expression(filters) + empty = ds.dataset(pa.Table.from_batches([], schema=schema)) + empty_result = _to_pandas( + empty.to_table(columns=columns, filter=expression), partition_dictionaries + ) + query_keys, target_keys = SCORE_KEYS[artifact] + numeric = artifact == "ligand_scores" + data: dict[str, list[str]] = manifest["data"] + targets: dict[str, list[str]] = manifest["targets"] + branches = [filters] if isinstance(filters[0], tuple) else filters + selected: set[str] = set() + index_cache: dict[str, pd.DataFrame] = {} + for branch in branches: + buckets = set(data) + for column, operator, value in branch: + if column not in query_keys or operator not in {"=", "==", "in"}: + continue + values = value if operator == "in" else [value] + buckets &= {score_bucket(v, numeric=numeric) for v in values} + for column, operator, value in branch: + if not buckets or len(buckets) == 1: + break # The original Arrow filter handles target predicates. + if column not in target_keys or operator not in {"=", "==", "in"}: + continue + values = value if operator == "in" else [value] + strings = {str(int(v)) if numeric else str(v) for v in values} + matching: set[str] = set() + for bucket in {score_bucket(v, numeric=numeric) for v in values}: + if bucket not in index_cache: + paths = _files( + manifest_path.parent, targets.get(bucket, []), release + ) + index_cache[bucket] = ( + pd.read_parquet(paths) + if paths + else pd.DataFrame(columns=["column", "value", "shard"]) + ) + index = index_cache[bucket] + matching.update( + index.loc[ + (index["column"] == column) & index["value"].isin(strings), + "shard", + ] + ) + buckets &= matching + selected.update(buckets) + relatives = [path for bucket in sorted(selected) for path in data[bucket]] + paths = _files(manifest_path.parent, relatives, release) + if not paths: + return empty_result + return _to_pandas( + ds.dataset(paths, format="parquet", schema=schema).to_table( + columns=columns, filter=expression + ), + partition_dictionaries, + ) diff --git a/src/plinder/data/annotations/aggregate_annotations.py b/src/plinder/data/annotations/aggregate_annotations.py index 5ff69eda..37590608 100644 --- a/src/plinder/data/annotations/aggregate_annotations.py +++ b/src/plinder/data/annotations/aggregate_annotations.py @@ -2798,6 +2798,7 @@ def chains_to_df(self) -> pd.DataFrame: "chain_modified_residues", "chain_length", "chain_num_unresolved_residues", + "chain_unresolved_fraction", "chain_is_holo", "chain_is_ligand_like", "chain_uniprot_ids", @@ -2825,6 +2826,12 @@ def chains_to_df(self) -> pd.DataFrame: "chain_modified_residues": list(chain.modified_residues), "chain_length": chain.length, "chain_num_unresolved_residues": chain.num_unresolved_residues, + "chain_unresolved_fraction": ( + chain.num_unresolved_residues / chain.length + if chain.length > 0 + and 0 <= chain.num_unresolved_residues <= chain.length + else None + ), "chain_is_holo": chain.holo, "chain_is_ligand_like": chain_id in self.ligand_like_chains, "chain_uniprot_ids": sorted(chain.mappings.get("UniProt", {})), @@ -3029,7 +3036,11 @@ def set_validation( instance_chain.split(".", maxsplit=1)[-1] for instance_chain in system.protein_chains_asym_id } - | {ligand.asym_id for ligand in system.ligands} + | { + member.split(".", maxsplit=1)[-1] + for ligand in system.ligands + for member in ligand._members + } for system_id, system in self.systems.items() } failed_chains = set() diff --git a/src/plinder/data/annotations/get_ligand_validation.py b/src/plinder/data/annotations/get_ligand_validation.py index 2adf7ee8..1e7a00ba 100644 --- a/src/plinder/data/annotations/get_ligand_validation.py +++ b/src/plinder/data/annotations/get_ligand_validation.py @@ -4,6 +4,7 @@ import typing as ty from collections.abc import Collection, Sequence +from copy import copy from functools import cached_property import numpy as np @@ -13,6 +14,7 @@ from plinder.data.annotations.utils import DocBaseModel LOG = setup_logger(__name__) +HYDROGEN_ELEMENTS = frozenset({"H", "D", "T"}) if ty.TYPE_CHECKING: from PDBValidation.Validation import PDBValidation @@ -31,6 +33,24 @@ def nanmean_return_nan(arr: list[float]) -> float: return float(np.nanmean(arr)) +def _heavy_atom_count_is_consistent( + residue: ty.Any, observed_count: int, reference_count: int +) -> bool: + """Keep dependency consistency rules, with isotope-aware heavy counts. + + Normalize only a shallow counting view, never the original residue or + the shared compound library. The subclass's polymer/leaving-atom rules + remain responsible for deciding consistency. + """ + counting_view = copy(residue) + counting_view.countAtomsHeavyPDBX = lambda: observed_count + counting_view.countAtomsHeavyConop = lambda: reference_count + counting_view.countAtomsHeavyOST = lambda: sum( + str(atom.element).upper() not in HYDROGEN_ELEMENTS for atom in residue.ost.atoms + ) + return bool(counting_view.isAtomCountConsistent()) + + def _select_altcode( altcodes: Collection[str], preferred_altcode: str | None = None, @@ -93,7 +113,7 @@ def from_residue( residue_with_alts = Residue.CreateFromMmCIFPosition( chain, resnum, str(entity), doc ) - except (ResidueNotFound, ModelledSubgroupNotFound) as e: + except (ResidueNotFound, ModelledSubgroupNotFound, AssertionError) as e: LOG.warning( f"from_residue: Error creating residue {chain} {resnum} {entity}: {e}" ) @@ -103,8 +123,22 @@ def from_residue( try: alt = _select_altcode(alts, preferred_altcode) residue = residue_with_alts.getAlt(alt) - heavy_atom_count = residue.countAtomsHeavyPDBX() - heavy_atom_count_conop = residue.countAtomsHeavyConop() + heavy = ( + ~residue.pdbx_residue["type_symbol"] + .astype(str) + .str.upper() + .isin(HYDROGEN_ELEMENTS) + .to_numpy() + ) + heavy_atom_count = int(heavy.sum()) + heavy_atom_count_conop = ( + sum( + str(atom.element).upper() not in HYDROGEN_ELEMENTS + for atom in residue.conop.atom_specs + ) + if residue.conop is not None + else None + ) num_unresolved_heavy_atoms = None if heavy_atom_count_conop is not None: difference = heavy_atom_count_conop - heavy_atom_count @@ -120,8 +154,12 @@ def from_residue( rsr=residue.getRSR(), rsrz=residue.getRSRZ(), rscc=residue.getRSCC(), - average_occupancy=np.nanmean(residue.getOccupancies()), - average_b_factor=np.nanmean(residue.getBFactors()), + average_occupancy=nanmean_return_nan( + np.asarray(residue.getOccupancies())[heavy].tolist() + ), + average_b_factor=nanmean_return_nan( + np.asarray(residue.getBFactors())[heavy].tolist() + ), unknown_residue=residue.getResName() in PDBValidation.unknown_resnames, atom_count=residue.countAtomsPDBX(), heavy_atom_count=heavy_atom_count, @@ -132,7 +170,9 @@ def from_residue( for s in ["geometry", "density", "chirality", "clashes"] }, is_atom_count_consistent=( - residue.isAtomCountConsistent() + _heavy_atom_count_is_consistent( + residue, heavy_atom_count, heavy_atom_count_conop + ) if heavy_atom_count_conop is not None else None ), diff --git a/src/plinder/data/annotations/ligand_utils.py b/src/plinder/data/annotations/ligand_utils.py index dbf33cca..2e73f570 100644 --- a/src/plinder/data/annotations/ligand_utils.py +++ b/src/plinder/data/annotations/ligand_utils.py @@ -2455,6 +2455,16 @@ def format(self, chains: dict[str, Chain]) -> dict[str, ty.Any]: # spans several — so the source-reconstruction SDF covers the whole # molecule, not just the primary chain. data["ligand_instance_chains"] = sorted(self._members) + from plinder.data.annotations.ligand_validation import ( + format_selection_validation, + ) + + data.update(format_selection_validation(self._members, chains, prefix="ligand")) + data.update( + format_selection_validation( + self.neighboring_residues, chains, prefix="ligand_pocket" + ) + ) data["ligand_water_residues"] = sorted( f"{chain_id}_{residue_number}" for chain_id, residue_numbers in self.waters.items() diff --git a/src/plinder/data/annotations/ligand_validation.py b/src/plinder/data/annotations/ligand_validation.py new file mode 100644 index 00000000..24d596c7 --- /dev/null +++ b/src/plinder/data/annotations/ligand_validation.py @@ -0,0 +1,61 @@ +# Copyright (c) 2024, Plinder Development Team +# Distributed under the terms of the Apache License 2.0 +"""Validation of a ligand and its own pocket, independent of system grouping.""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from typing import TYPE_CHECKING + +from plinder.data.annotations.get_ligand_validation import ( + ResidueListValidation, + ResidueValidation, + ResidueValidationThresholds, +) + +if TYPE_CHECKING: + from plinder.data.annotations.protein_utils import Chain + + +def validation_columns(prefix: str) -> list[str]: + """Return all summary fields, including fields unavailable for an entry.""" + fields = [ + key + for key in ResidueListValidation.model_fields + if key not in {"thresholds", "percent_outliers"} + ] + fields.extend( + f"percent_outliers_{kind}" + for kind in ("geometry", "density", "chirality", "clashes") + ) + return [f"{prefix}_validation_{key}" for key in fields] + + +def format_selection_validation( + selection: Mapping[str, Sequence[int]], + chains: Mapping[str, Chain], + *, + prefix: str, + thresholds: ResidueValidationThresholds | None = None, +) -> dict[str, float | int | None]: + """Aggregate selected residues, never other ligands sharing a system. + + A multi-chain covalent ligand includes all its members. Duplicate residue + numbers within a chain are removed; distinct assembly instances remain + distinct. Missing residue validation remains in the coverage denominator. + """ + residues: list[ResidueValidation | None] = [] + selected_thresholds = thresholds + for instance_chain, numbers in sorted(selection.items()): + chain = chains[instance_chain.split(".", maxsplit=1)[-1]] + if selected_thresholds is None and chain.validation is not None: + selected_thresholds = chain.validation.thresholds + residues.extend( + chain.residues[number].validation for number in sorted(set(numbers)) + ) + summary = ResidueListValidation.from_residues( + residues, selected_thresholds or ResidueValidationThresholds() + ) + if summary is None: + return dict.fromkeys(validation_columns(prefix)) + return {f"{prefix}_{key}": value for key, value in summary.format().items()} diff --git a/src/plinder/data/column_descriptions/tables/entry_chains.tsv b/src/plinder/data/column_descriptions/tables/entry_chains.tsv index 88eeb816..e9aff7b6 100644 --- a/src/plinder/data/column_descriptions/tables/entry_chains.tsv +++ b/src/plinder/data/column_descriptions/tables/entry_chains.tsv @@ -10,6 +10,7 @@ chain_sequence_noncanonical string Full polymer sequence from _entity_poly.pdbx_ chain_modified_residues list Non-canonical SEQRES monomers as {auth_seq}:{comp_id}:{asym}:{label_seq}>{parent} (details), the residue address of ligand_covalent_linkages (comp_id is entity_poly_seq.mon_id); unresolved positions have auth_seq ?; parent and details from _pdbx_struct_mod_residue, else the CCD parent chain_length int64 SEQRES length chain_num_unresolved_residues int64 Number of unresolved residues (SEQRES length - len(residues)) +chain_unresolved_fraction double Fraction of SEQRES residues without coordinates: chain_num_unresolved_residues / chain_length (0 to 1) chain_is_holo bool Whether the chain participates in a holo receptor or is excluded from the apo partition chain_is_ligand_like bool Whether this chain is ligand-like rather than an eligible receptor chain chain_uniprot_ids list UniProt accessions mapped to the chain diff --git a/src/plinder/data/docs.py b/src/plinder/data/docs.py index 8994b374..4f29b0a6 100644 --- a/src/plinder/data/docs.py +++ b/src/plinder/data/docs.py @@ -96,6 +96,10 @@ "Full polymer sequence in one-letter code, keyed by the source " "asymmetric-chain ID" ), + "chain_unresolved_fraction": ( + "Fraction of SEQRES residues without coordinates: " + "chain_num_unresolved_residues / chain_length (0 to 1)" + ), "chain_sequence_noncanonical": ( "Full polymer sequence from _entity_poly.pdbx_seq_one_letter_code, with " "modified residues as (CCD) tokens" @@ -548,6 +552,8 @@ def _model_description_lookup() -> dict[str, str]: for prefix in [ "system_pocket", "system_ligand", + "ligand", + "ligand_pocket", "ligand_interacting_ligand_chains", "ligand_neighboring_ligand_chains", "ligand_protein_chains", diff --git a/src/plinder/data/pipeline/collate.py b/src/plinder/data/pipeline/collate.py index 3fc7eab0..f22383ec 100644 --- a/src/plinder/data/pipeline/collate.py +++ b/src/plinder/data/pipeline/collate.py @@ -199,6 +199,7 @@ def _raw_annotation_columns_to_keep(columns: Iterable[str]) -> list[str]: ("chain_modified_residues", pa.list_(pa.string())), ("chain_length", pa.int64()), ("chain_num_unresolved_residues", pa.int64()), + ("chain_unresolved_fraction", pa.float64()), ("chain_is_holo", pa.bool_()), ("chain_is_ligand_like", pa.bool_()), ("chain_uniprot_ids", pa.list_(pa.string())), @@ -863,6 +864,22 @@ def _relation_columns( return [str(row[1]) for row in rows] +def _chain_unresolved_fraction_projection(columns: Sequence[str]) -> str: + """Derive missing completeness only; supplied values remain for validation.""" + fraction = ( + "CASE WHEN chain_length > 0 AND " + "chain_num_unresolved_residues BETWEEN 0 AND chain_length " + "THEN chain_num_unresolved_residues::DOUBLE / chain_length " + "ELSE NULL END" + ) + if "chain_unresolved_fraction" in columns: + return ( + "* REPLACE (coalesce(chain_unresolved_fraction, " + f"{fraction}) AS chain_unresolved_fraction)" + ) + return f"*, {fraction} AS chain_unresolved_fraction" + + def _require_columns( columns: Iterable[str], required: Iterable[str], label: str ) -> None: @@ -1500,6 +1517,17 @@ def _validate_final_tables( ) ), } + invalid_chain_metadata["incorrect_unresolved_fraction"] = int( + _fetch_scalar( + connection, + "SELECT count(*) FROM entry_chains WHERE " + "chain_unresolved_fraction IS DISTINCT FROM " + "CASE WHEN chain_length > 0 AND " + "chain_num_unresolved_residues BETWEEN 0 AND chain_length " + "THEN chain_num_unresolved_residues::DOUBLE / chain_length " + "ELSE NULL END", + ) + ) if any(invalid_chain_metadata.values()): raise ValueError( f"invalid entry-chain sequence metadata: {invalid_chain_metadata}" @@ -1992,6 +2020,7 @@ def repair_collation( } chain_annotations_changed = False + chain_completeness_backfilled = False try: for name, schema in SIDECAR_SCHEMAS.items(): source_key = { @@ -2085,8 +2114,32 @@ def repair_collation( "entry_sources", ): connection.read_parquet(str(final_paths[name])).create_view( - f"installed_{name}", replace=True + ( + "installed_entry_chains_raw" + if name == "entry_chains" + else f"installed_{name}" + ), + replace=True, ) + if name == "entry_chains": + raw_columns = _relation_columns( + connection, "installed_entry_chains_raw" + ) + chain_completeness_backfilled = ( + "chain_unresolved_fraction" not in raw_columns + or bool( + _fetch_scalar( + connection, + "SELECT EXISTS (SELECT 1 FROM " + "installed_entry_chains_raw WHERE " + "chain_unresolved_fraction IS NULL)", + ) + ) + ) + projection = _chain_unresolved_fraction_projection(raw_columns) + connection.sql( + f"SELECT {projection} FROM installed_entry_chains_raw" + ).create_view("installed_entry_chains", replace=True) connection.read_parquet(str(replacement_paths[name])).create_view( f"replacement_{name}", replace=True ) @@ -2112,24 +2165,32 @@ def repair_collation( connection, name, columns ) replacements = ["entry_metadata", "interfaces"] - if chain_annotations_changed: + if chain_annotations_changed or chain_completeness_backfilled: temporary_paths["entry_chains"] = _temporary_path( final_paths["entry_chains"] ) replacements.append("entry_chains") for name in replacements: - connection.read_parquet(str(final_paths[name])).create_view( - f"installed_{name}", replace=True - ) + # Keep the normalized chain view when publishing its backfill. + if name != "entry_chains": + connection.read_parquet(str(final_paths[name])).create_view( + f"installed_{name}", replace=True + ) connection.read_parquet(str(replacement_paths[name])).create_view( f"replacement_{name}", replace=True ) + query = ( + f"SELECT installed.* FROM installed_{name} AS installed" + if name == "entry_chains" and not chain_annotations_changed + else ( + f"SELECT * FROM (SELECT installed.* FROM installed_{name} " + "AS installed ANTI JOIN repaired_entries USING (entry_pdb_id) " + f"UNION ALL BY NAME SELECT * FROM replacement_{name})" + ) + ) _copy_query( connection, - f"SELECT * FROM (SELECT installed.* FROM installed_{name} " - "AS installed ANTI JOIN repaired_entries USING (entry_pdb_id) " - f"UNION ALL BY NAME SELECT * FROM replacement_{name}) " - f"ORDER BY {ENTRY_TABLES[name][1]}", + f"{query} ORDER BY {ENTRY_TABLES[name][1]}", temporary_paths[name], row_group_size=row_group_size, ) diff --git a/src/plinder/data/pipeline/public_release.py b/src/plinder/data/pipeline/public_release.py index efcb05f6..e7492249 100644 --- a/src/plinder/data/pipeline/public_release.py +++ b/src/plinder/data/pipeline/public_release.py @@ -5,12 +5,14 @@ from __future__ import annotations import argparse +import json import os import shutil import tempfile from pathlib import Path from plinder.core.release import RELEASE_PATHS, RELEASE_TABLES +from plinder.core.scores.store import SCORE_KEYS _FILE_ARTIFACTS = ( *(table["artifact"] for table in RELEASE_TABLES.values()), @@ -38,6 +40,32 @@ def _public_files(source: Path) -> list[Path]: files = {Path(RELEASE_PATHS[name]) for name in _FILE_ARTIFACTS} + # Optional until a release has been repackaged. Publish only manifest-listed + # files, never scratch files or an unfinished export staging directory. + for artifact in SCORE_KEYS: + root = Path("score_store") / artifact + manifest = source / root / "manifest.json" + if not manifest.is_file(): + continue + payload = json.loads(manifest.read_text()) + if payload.get("version") != 1 or payload.get("artifact") != artifact: + raise ValueError(f"invalid score-store manifest: {manifest}") + files.add(root / "manifest.json") + for group in ("data", "targets"): + for paths in payload[group].values(): + for relative in paths: + path = Path(relative) + if ( + path.is_absolute() + or "\\" in relative + or any(part in {"", ".", ".."} for part in relative.split("/")) + or path.suffix != ".parquet" + ): + raise ValueError(f"unsafe score-store path: {relative!r}") + (source / root / path).resolve().relative_to( + (source / root).resolve() + ) + files.add(root / path) for name in _PARQUET_DIRECTORIES: directory = source / RELEASE_PATHS.get(name, name) if not directory.is_dir(): diff --git a/src/plinder/data/pipeline/score_store.py b/src/plinder/data/pipeline/score_store.py new file mode 100644 index 00000000..a553bf95 --- /dev/null +++ b/src/plinder/data/pipeline/score_store.py @@ -0,0 +1,338 @@ +# Copyright (c) 2024, Plinder Development Team +# Distributed under the terms of the Apache License 2.0 +"""Repackage existing scores for selective downloads, without rescoring. + +Run after score finalization and before preparing a public release. Output is +atomic per artifact, immutable once installed, and always separate from inputs. +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import shutil +import tempfile +from collections.abc import Iterator +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +import duckdb +import pyarrow as pa +import pyarrow.parquet as pq + +from plinder.core.release import PlinderRelease +from plinder.core.scores.store import SCORE_KEYS + + +def _literal(value: str | Path) -> str: + return "'" + str(value).replace("'", "''") + "'" + + +def _identifier(value: str) -> str: + return '"' + value.replace('"', '""') + '"' + + +def _bucket(column: str, *, numeric: bool) -> str: + quoted = _identifier(column) + if numeric: + return f"'n' || CAST(CAST(floor({quoted} / 1024.0) AS BIGINT) AS VARCHAR)" + return ( + f"CASE WHEN regexp_matches({quoted}, '^[a-zA-Z0-9]{{4}}(__|$)') " + f"THEN 'p' || lower(substr({quoted}, 2, 2)) " + f"ELSE 'h' || substr(sha256(split_part({quoted}, '__', 1)), 1, 2) END" + ) + + +def _partition_files(root: Path) -> dict[str, list[str]]: + result: dict[str, list[str]] = {} + for path in sorted(root.rglob("*.parquet")): + bucket = path.parent.name.removeprefix("_bucket=") + result.setdefault(bucket, []).append(path.relative_to(root.parent).as_posix()) + return result + + +@contextmanager +def _connection( + threads: int, memory_limit: str, scratch: Path +) -> Iterator[duckdb.DuckDBPyConnection]: + """Give each bounded unit its own engine and release its state on exit.""" + with duckdb.connect() as con: + con.execute(f"SET threads = {threads}") + con.execute(f"SET memory_limit = {_literal(memory_limit)}") + con.execute(f"SET temp_directory = {_literal(scratch)}") + con.execute("SET preserve_insertion_order = false") + con.execute("SET partitioned_write_max_open_files = 8") + yield con + + +def _sort_bucket( + bucket_dir: Path, + sorted_file: Path, + columns: str, + order: str, + *, + threads: int, + memory_limit: str, + scratch: Path, +) -> tuple[int, int, int]: + """Sort/check one bucket without retaining previous buckets' sort state.""" + raw = f"read_parquet({_literal(bucket_dir / '*.parquet')}, hive_partitioning=false)" + sorted_file.parent.mkdir(parents=True, exist_ok=True) + with _connection(threads, memory_limit, scratch) as con: + con.execute( + f"COPY (SELECT {columns} FROM {raw} ORDER BY {order}) " + f"TO {_literal(sorted_file)} (FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 32768)" + ) + fingerprint = ( + f"count(*), sum(CAST(hash({columns}) AS HUGEINT)), bit_xor(hash({columns}))" + ) + before = con.execute(f"SELECT {fingerprint} FROM {raw}").fetchone() + after = con.execute( + f"SELECT {fingerprint} FROM read_parquet({_literal(sorted_file)})" + ).fetchone() + if before != after: + raise ValueError(f"score-store fingerprint mismatch: {bucket_dir.name}") + return int(after[0]), int(after[1]), int(after[2]) + + +def _target_index( + staging: Path, + data: dict[str, list[str]], + column: str, + *, + numeric: bool, + threads: int, + memory_limit: str, + scratch: Path, +) -> None: + """Deduplicate one query bucket at a time, then sort eight coarse groups. + + No release-wide DISTINCT, sort, or thousands-of-writers operation is held + in memory. The temporary coarse partition has just eight writers; final + partitions in separate coarse groups never overlap. + """ + seeds = staging / f"index-parts-{column}" + seeds.mkdir(parents=True, exist_ok=True) + target = staging / f"target-{column}" + target.mkdir(parents=True, exist_ok=True) + quoted = _identifier(column) + for number, (shard, relative_paths) in enumerate(sorted(data.items()), start=1): + inputs = "[" + ", ".join(_literal(staging / p) for p in relative_paths) + "]" + with _connection(threads, memory_limit, scratch) as con: + con.execute( + f"COPY (SELECT *, hash(target_bucket) % 8 AS _coarse FROM (" + f"SELECT DISTINCT {_literal(column)} AS column, " + f"CAST({quoted} AS VARCHAR) AS value, {_literal(shard)} AS shard, " + f"{_bucket(column, numeric=numeric)} AS target_bucket " + f"FROM read_parquet({inputs}, hive_partitioning=false))) " + f"TO {_literal(seeds / shard)} (FORMAT parquet, PARTITION_BY (_coarse), " + "COMPRESSION zstd, ROW_GROUP_SIZE 32768)" + ) + if number % 100 == 0 or number == len(data): + print(f"{column}: indexed {number}/{len(data)} query buckets", flush=True) + for coarse in range(8): + seed_files = sorted(seeds.glob(f"*/_coarse={coarse}/*.parquet")) + if not seed_files: + continue + inputs = "[" + ", ".join(_literal(p) for p in seed_files) + "]" + with _connection(threads, memory_limit, scratch) as con: + con.execute( + 'COPY (SELECT "column", value, shard, target_bucket FROM ' + f"read_parquet({inputs}, hive_partitioning=false) " + "ORDER BY target_bucket, value, shard) " + f"TO {_literal(target / f'group-{coarse}')} " + "(FORMAT parquet, PARTITION_BY (target_bucket), COMPRESSION zstd, ROW_GROUP_SIZE 32768)" + ) + for path in seed_files: + path.unlink() + print(f"{column}: finalized coarse group {coarse + 1}/8", flush=True) + if seeds.exists(): + shutil.rmtree(seeds) + + +def export_score_store( + source: Path, + destination: Path, + artifact: str, + *, + threads: int = 2, + memory_limit: str = "2GB", + scratch_dir: Path | None = None, +) -> dict[str, Any]: + """Create sorted query shards and exact target-to-shard routing indices. + + Original row multiplicity, columns, score direction, and Arrow types are + retained. Installation refuses overwrite, including a partially staged + destination, so an existing release cannot accidentally be modified. + """ + query_keys, target_keys = SCORE_KEYS[artifact] + release = PlinderRelease(source) + input_path = release.fetch(artifact).resolve() + output = destination.resolve() / "score_store" / artifact + if output.exists(): + raise FileExistsError(output) + if ( + input_path == output + or input_path in output.parents + or output in input_path.parents + ): + raise ValueError("score-store input and output must be separate") + if threads < 1: + raise ValueError("threads must be positive") + dataset = pq.ParquetDataset(input_path) + schema = dataset.schema + # Arrow schemas record dictionary types, not their values. Keep the Hive + # categories even when a routed query selects no rows (or no shards). + partition_dictionaries = {} + if dataset.partitioning is not None: + for field, dictionary in zip( + dataset.partitioning.schema, dataset.partitioning.dictionaries + ): + if dictionary is None or not pa.types.is_dictionary(field.type): + continue + table = pa.table({field.name: dictionary}) + sink = pa.BufferOutputStream() + with pa.ipc.new_stream(sink, table.schema) as writer: + writer.write_table(table) + partition_dictionaries[field.name] = base64.b64encode( + sink.getvalue().to_pybytes() + ).decode() + columns = ", ".join(_identifier(name) for name in schema.names) + numeric = artifact == "ligand_scores" + inputs = ( + str(input_path / "**/*.parquet") if input_path.is_dir() else str(input_path) + ) + output.parent.mkdir(parents=True, exist_ok=True) + staging = Path(tempfile.mkdtemp(prefix=f".{artifact}-", dir=output.parent)) + scratch = scratch_dir or staging / "scratch" + scratch.mkdir(parents=True, exist_ok=True) + try: + with _connection(threads, memory_limit, scratch) as con: + con.execute( + f"CREATE VIEW scores AS SELECT {columns} FROM " + f"read_parquet({_literal(inputs)}, hive_partitioning=true)" + ) + count = int(con.execute("SELECT count(*) FROM scores").fetchone()[0]) + print(f"{artifact}: exporting {count:,} rows", flush=True) + order = ", ".join(_identifier(c) for c in (*query_keys, *target_keys)) + nulls = " OR ".join( + f"{_identifier(c)} IS NULL" for c in (*query_keys, *target_keys) + ) + if con.execute( + f"SELECT EXISTS(SELECT 1 FROM scores WHERE {nulls})" + ).fetchone()[0]: + raise ValueError("score routing keys cannot be null") + for alias in query_keys[1:]: + mismatch = f"{_bucket(alias, numeric=numeric)} != {_bucket(query_keys[0], numeric=numeric)}" + if con.execute( + f"SELECT EXISTS(SELECT 1 FROM scores WHERE {mismatch})" + ).fetchone()[0]: + raise ValueError( + f"inconsistent query entry in routing alias: {alias}" + ) + if count: + # Partition first: sorting a whole multi-billion-row export + # would spill vastly more than sorting one bounded bucket. + con.execute( + f"COPY (SELECT {columns}, {_bucket(query_keys[0], numeric=numeric)} " + f"AS _bucket FROM scores) " + f"TO {_literal(staging / 'raw')} (FORMAT parquet, PARTITION_BY (_bucket), " + "COMPRESSION zstd, ROW_GROUP_SIZE 32768)" + ) + fingerprint_sum = 0 + fingerprint_xor = 0 + actual = 0 + bucket_dirs = sorted((staging / "raw").glob("_bucket=*")) + for bucket_number, bucket_dir in enumerate(bucket_dirs, start=1): + sorted_file = staging / "data" / bucket_dir.name / "data_0.parquet" + rows, hash_sum, hash_xor = _sort_bucket( + bucket_dir, + sorted_file, + columns, + order, + threads=threads, + memory_limit=memory_limit, + scratch=scratch, + ) + actual += rows + fingerprint_sum += hash_sum + fingerprint_xor ^= hash_xor + shutil.rmtree(bucket_dir) + if bucket_number % 100 == 0 or bucket_number == len(bucket_dirs): + print( + f"{artifact}: sorted {bucket_number}/{len(bucket_dirs)} buckets", + flush=True, + ) + if (staging / "raw").exists(): + (staging / "raw").rmdir() + if actual != count: + raise ValueError(f"score-store row-count mismatch: {actual} != {count}") + data = _partition_files(staging / "data") + for column in target_keys if count else (): + _target_index( + staging, + data, + column, + numeric=numeric, + threads=threads, + memory_limit=memory_limit, + scratch=scratch, + ) + targets: dict[str, list[str]] = {} + for column in target_keys: + for path in sorted((staging / f"target-{column}").rglob("*.parquet")): + bucket = path.parent.name.removeprefix("target_bucket=") + targets.setdefault(bucket, []).append( + path.relative_to(staging).as_posix() + ) + manifest: dict[str, Any] = { + "version": 1, + "artifact": artifact, + "schema": base64.b64encode(schema.serialize().to_pybytes()).decode(), + "partition_dictionaries": partition_dictionaries, + "rows": count, + "fingerprint": { + "hash_sum": str(fingerprint_sum), + "hash_xor": str(fingerprint_xor), + }, + "data": data, + "targets": targets, + "bytes": sum(p.stat().st_size for p in staging.rglob("*.parquet")), + } + (staging / "manifest.json").write_text( + json.dumps(manifest, sort_keys=True, indent=2) + "\n" + ) + if scratch == staging / "scratch": + shutil.rmtree(scratch) + staging.rename(output) + print(f"{artifact}: installed {len(data)} buckets at {output}", flush=True) + return manifest + finally: + if staging.exists(): + shutil.rmtree(staging) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source", type=Path) + parser.add_argument("destination", type=Path) + parser.add_argument("--artifact", action="append", choices=tuple(SCORE_KEYS)) + parser.add_argument("--threads", type=int, default=2) + parser.add_argument("--memory-limit", default="2GB") + parser.add_argument("--scratch-dir", type=Path) + args = parser.parse_args() + for artifact in args.artifact or SCORE_KEYS: + export_score_store( + args.source, + args.destination, + artifact, + threads=args.threads, + memory_limit=args.memory_limit, + scratch_dir=args.scratch_dir, + ) + + +if __name__ == "__main__": + main() diff --git a/src/plinder/data/pipeline/update_entries.py b/src/plinder/data/pipeline/update_entries.py index 6fd7730f..685e3554 100644 --- a/src/plinder/data/pipeline/update_entries.py +++ b/src/plinder/data/pipeline/update_entries.py @@ -76,6 +76,11 @@ def write_table(item: tuple[str, tuple[str, str]]) -> tuple[str, int]: else "old_rows.entry_pdb_id = removed_entries.entry_pdb_id" ) retained = f"SELECT old_rows.* FROM old_rows ANTI JOIN removed_entries ON {condition}" + if name == "entry_chains": + projection = collate._chain_unresolved_fraction_projection( + connection.table("old_rows").columns + ) + retained = f"SELECT {projection} FROM ({retained})" query = retained if incoming is not None: connection.read_parquet(str(incoming / "index" / filename)).create_view( diff --git a/tests/core/test_core_scores.py b/tests/core/test_core_scores.py index f960dc06..9fa96dea 100644 --- a/tests/core/test_core_scores.py +++ b/tests/core/test_core_scores.py @@ -5,6 +5,8 @@ from plinder.core import PlinderRelease, query_table, scores from plinder.core.scores import ligand as ligand_module +from plinder.core.scores.query import query_score_table +from plinder.data.pipeline.score_store import export_score_store def test_custom_scoring_public_api(): @@ -13,6 +15,97 @@ def test_custom_scoring_public_api(): assert scores.CustomProteinSearchConfig().max_seqs == 10_000 +@pytest.mark.parametrize( + "artifact,query,filters,options", + [ + ( + "ligand_similarity_scores", + scores.query_ligand_similarity, + [("query_ligand_id", "==", "query__1.A")], + {}, + ), + ( + "interface_similarity_scores", + scores.query_interface_similarity, + [("query_system", "==", "query_interface")], + {}, + ), + ( + "interface_half_similarity_scores", + scores.query_half_interface_similarity, + [("target_half_interface_id", "==", "target_interface::side=2")], + {}, + ), + ( + "protein_similarity_scores", + scores.query_protein_similarity, + [("query_entry", "==", "1abc")], + {}, + ), + ( + "protein_similarity_scores", + scores.query_protein_similarity, + [("target_entry", "==", "1abc")], + {"include_monomers": True}, + ), + ( + "ligand_scores", + scores.query_ligand_chemical_similarity, + [("query_ligand_id", "==", 29)], + {}, + ), + ], +) +def test_public_queries_use_partitioned_exports( + similarity_release, tmp_path, artifact, query, filters, options +): + output = tmp_path / "optimized" + export_score_store(similarity_release.data_dir, output, artifact) + if options.get("include_monomers"): + export_score_store( + similarity_release.data_dir, output, "monomer_similarity_scores" + ) + expected = query(release=similarity_release, filters=filters, **options) + actual = query(release=PlinderRelease(output), filters=filters, **options) + columns = list(expected.columns) + pd.testing.assert_frame_equal( + expected.sort_values(columns).reset_index(drop=True), + actual.sort_values(columns).reset_index(drop=True), + ) + + +@pytest.mark.parametrize("columns", [None, ["alignment_type"]]) +@pytest.mark.parametrize( + "filters", + [ + [("query_entry", "==", "1abc")], + [("target_entry", "==", "1abc")], + [("query_entry", "==", "missing")], + ], +) +def test_partitioned_scores_preserve_hive_categories( + similarity_release, tmp_path, columns, filters +): + output = tmp_path / "optimized" + artifact = "protein_similarity_scores" + source = similarity_release.fetch(artifact) + extra = pd.read_parquet(source).drop(columns="alignment_type") + extra["query_entry"] = "2def" + extra["target_entry"] = "3ghi" + extra["source"] = "mmseqs" + partition = source / "alignment_type=mmseqs" + partition.mkdir() + extra.to_parquet(partition / "part-0.parquet", index=False) + export_score_store(similarity_release.data_dir, output, artifact) + expected = query_score_table( + artifact, release=similarity_release, columns=columns, filters=filters + ) + actual = query_score_table( + artifact, release=PlinderRelease(output), columns=columns, filters=filters + ) + pd.testing.assert_frame_equal(expected, actual) + + @pytest.fixture def similarity_release(tmp_path): index = tmp_path / "index" diff --git a/tests/core/test_docs_score_release.py b/tests/core/test_docs_score_release.py new file mode 100644 index 00000000..8c67a4b9 --- /dev/null +++ b/tests/core/test_docs_score_release.py @@ -0,0 +1,29 @@ +"""Keep the bounded documentation input complete and reproducible.""" + +import hashlib +import json +from pathlib import Path + +import pyarrow.parquet as pq + + +def test_docs_score_release_manifest(): + root = Path(__file__).resolve().parents[1] / "test_data/docs_score_release" + manifest = json.loads((root / "manifest.json").read_text()) + assert manifest["release"] == "2026-09" + assert {"4agi", "4agc", "7eek", "7cm8"} <= set(manifest["entries"]) + paths = [row["path"] for row in manifest["files"]] + assert len(paths) == len(set(paths)) + assert "exports/ligand_similarity_scores.parquet" in paths + assert "exports/interface_similarity_scores.parquet" in paths + assert "exports/interface_half_similarity_scores.parquet" in paths + assert ( + "alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=cm.parquet" + in paths + ) + for row in manifest["files"]: + path = root / row["path"] + assert path.is_relative_to(root) + assert hashlib.sha256(path.read_bytes()).hexdigest() == row["sha256"] + assert pq.ParquetFile(path).metadata.num_rows == row["rows"] + assert sum((root / path).stat().st_size for path in paths) < 5_000_000 diff --git a/tests/core/test_index_utils.py b/tests/core/test_index_utils.py index 6de0e399..c2a76ce5 100644 --- a/tests/core/test_index_utils.py +++ b/tests/core/test_index_utils.py @@ -170,3 +170,13 @@ def track_path(**kwargs): } assert requested assert large_paths.isdisjoint(requested) + + +def test_bulk_download_supports_releases_without_score_store(mock_cpl, monkeypatch): + def track_path(**kwargs): + if kwargs["rel"] == "score_store": + raise FileNotFoundError("not repackaged yet") + return mock_path(**kwargs) + + monkeypatch.setattr(utils.cpl, "get_plinder_path", track_path) + utils.download_plinder_cmd(args=["-y"]) diff --git a/tests/core/test_interface_site_search.py b/tests/core/test_interface_site_search.py new file mode 100644 index 00000000..d6d49a3a --- /dev/null +++ b/tests/core/test_interface_site_search.py @@ -0,0 +1,149 @@ +"""Known interface sites can be located without an input binding partner.""" + +from types import SimpleNamespace + +import pandas as pd +import pytest + +from plinder.core.scores.custom import calculate_custom_interface_side_scores +from plinder.core.scores.entries import InterfaceView +from plinder.core.scores.interface import ( + INTERFACE_SIDE_COLUMNS, + calculate_interface_side_scores, +) +from plinder.core.scores.mapping import pack_residue_identities + + +def interface(): + return InterfaceView( + id="1abc__1__1.B--2.D", + pdb_id="1abc", + biounit_id="1", + chain_1="1.B", + chain_2="2.D", + chain_1_residue_number_to_index={10: 0, 20: 1, 30: 2, 40: 3}, + chain_2_residue_number_to_index={1: 0, 2: 1}, + num_contact_residue_pairs=4, + ) + + +def alignment(**changes): + return { + "query_entry": "1abc", + "target_entry": "monomer", + "query_chain_mapped": "B", + "target_chain_mapped": "A.with.dot", + "source": "mmseqs", + "query_selected_residue_numbers": [10, 20, 30, 99], + "target_selected_residue_numbers": [101, 102, -1, 109], + "selected_residue_identity_bits": pack_residue_identities( + [True, False, True, True] + ), + **changes, + } + + +def score(rows): + known = interface() + return calculate_interface_side_scores( + pd.DataFrame(rows), interfaces={known.id: known} + ) + + +def test_monomer_hits_have_directional_site_coverage_and_identity(): + result = score([alignment()]) + assert len(result) == 1 + hit = result.iloc[0] + assert hit.custom_structure_id == "monomer" + assert hit.custom_chain == "A.with.dot" + assert hit.plinder_interface_id == interface().id + assert hit.plinder_chain == "1.B" + assert hit.plinder_partner_chain == "2.D" + assert hit.interface_side_qcov == 50 + assert hit.interface_side_fident == 25 + assert hit.plinder_residue_numbers == [10, 20] + assert hit.custom_residue_numbers == [101, 102] + + +def test_alternative_alignments_are_not_unioned_and_backends_stay_separate(): + result = score( + [ + alignment(), + alignment( + query_selected_residue_numbers=[30, 40], + target_selected_residue_numbers=[103, 104], + selected_residue_identity_bits=pack_residue_identities([True, True]), + ), + alignment(source="foldseek"), + ] + ) + assert len(result) == 2 + assert result.interface_side_qcov.tolist() == [50, 50] + mmseqs = result.set_index("source").loc["mmseqs"] + assert mmseqs.interface_side_fident == 50 + assert mmseqs.plinder_residue_numbers == [30, 40] + + +def test_either_interface_side_can_match(): + result = score( + [ + alignment( + query_chain_mapped="D", + query_selected_residue_numbers=[1, 2], + target_selected_residue_numbers=[101, 102], + ) + ] + ) + assert result.iloc[0].plinder_chain == "2.D" + assert result.iloc[0].plinder_partner_chain == "1.B" + assert result.iloc[0].interface_side_qcov == 100 + + +@pytest.mark.parametrize( + "rows", + [ + [], + [alignment(query_chain_mapped="unrelated")], + [alignment(target_selected_residue_numbers=[-1, -1, -1, -1])], + ], +) +def test_no_site_match_keeps_output_columns(rows): + result = score(rows) + assert result.empty + assert list(result.columns) == INTERFACE_SIDE_COLUMNS + + +def test_mismatched_residue_maps_fail(): + with pytest.raises(ValueError): + score([alignment(target_selected_residue_numbers=[101])]) + + +def test_duplicate_residues_do_not_inflate_coverage(): + result = score([alignment(query_selected_residue_numbers=[10, 10, 20, 20])]) + assert result.iloc[0].interface_side_qcov == 50 + + +def test_sequence_output_preserves_original_fasta_identifier(tmp_path, monkeypatch): + from plinder.core.scores import custom + + known = interface() + monkeypatch.setattr( + custom, + "_load_release_entry_views", + lambda assets, *, pdb_ids: { + "1abc": SimpleNamespace(interfaces={known.id: known}) + }, + ) + path = tmp_path / "alignments.parquet" + pd.DataFrame([alignment()]).to_parquet(path) + manifest = tmp_path / "manifest.parquet" + pd.DataFrame( + {"structure_id": ["monomer"], "sequence_id": ["my_original_sequence"]} + ).to_parquet(manifest) + output = tmp_path / "sites.parquet" + calculate_custom_interface_side_scores( + {"mmseqs": path}, assets=None, output_path=output, chain_manifest=manifest + ) + result = pd.read_parquet(output) + assert result.sequence_id.tolist() == ["my_original_sequence"] + assert result.custom_residue_numbers.iloc[0].tolist() == [101, 102] diff --git a/tests/core/test_score_store.py b/tests/core/test_score_store.py new file mode 100644 index 00000000..172ecaad --- /dev/null +++ b/tests/core/test_score_store.py @@ -0,0 +1,444 @@ +# Copyright (c) 2024, Plinder Development Team +# Distributed under the terms of the Apache License 2.0 +"""Storage routing must never change directed score results.""" + +import json +from contextlib import contextmanager +from functools import partial +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from threading import Thread +from types import SimpleNamespace +from urllib.parse import unquote + +import pandas as pd +import pytest + +from plinder.core.release import PlinderRelease +from plinder.core.scores.query import query_score_table +from plinder.core.scores.store import score_bucket +from plinder.data.pipeline.score_store import export_score_store + + +@pytest.fixture +def stored_scores(tmp_path): + source = tmp_path / "source" + path = source / "exports/interface_similarity_scores.parquet" + path.parent.mkdir(parents=True) + frame = pd.DataFrame( + { + "query_system": ["1abc__1", "2def__1", "3ghi__1", "1abc__1"], + "target_system": ["2def__1", "1abc__1", "2def__1", "2def__1"], + "similarity": pd.Series([91, 32, 73, 91], dtype="int8"), + } + ) + frame.to_parquet(path, index=False) + destination = tmp_path / "optimized" + manifest = export_score_store(source, destination, "interface_similarity_scores") + return source, destination, frame, manifest + + +def _canonical(frame): + return frame.sort_values(list(frame.columns)).reset_index(drop=True) + + +@pytest.mark.parametrize( + "filters", + [ + [("query_system", "==", "1abc__1")], + [("target_system", "==", "2def__1")], + [("target_system", "in", {"2def__1", "1abc__1"})], + [("query_system", "==", "not-present")], + [("target_system", "==", "not-present")], + [("query_system", "!=", "1abc__1")], + [("similarity", ">", 50)], + [ + [("query_system", "==", "1abc__1"), ("similarity", ">", 50)], + [("target_system", "==", "1abc__1")], + ], + [ + [("query_system", "==", "1abc__1")], + [("similarity", "<", 50)], + ], + [("query_system", "==", "1abc__1"), ("query_system", "==", "2def__1")], + ], +) +@pytest.mark.parametrize("columns", [None, ["similarity"]]) +def test_sharded_queries_match_legacy(stored_scores, filters, columns): + source, destination, _, _ = stored_scores + expected = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(source), + filters=filters, + columns=columns, + ) + actual = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=filters, + columns=columns, + ) + pd.testing.assert_frame_equal(_canonical(actual), _canonical(expected)) + + +def test_only_routed_files_are_opened(stored_scores, monkeypatch): + import plinder.core.scores.store as store + + _, destination, _, manifest = stored_scores + opened = [] + original = store._file + + def record(root, relative, release): + opened.append(relative) + return original(root, relative, release) + + monkeypatch.setattr(store, "_file", record) + query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=[("query_system", "==", "1abc__1")], + ) + assert opened == manifest["data"][score_bucket("1abc__1")] + opened.clear() + result = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=[("target_system", "==", "1abc__1")], + ) + assert result["similarity"].tolist() == [32] # Not the reverse score, 91. + assert set(opened) == set( + manifest["targets"][score_bucket("1abc__1")] + + manifest["data"][score_bucket("2def__1")] + ) + + +def test_reject_unbounded_query_before_fetch(monkeypatch): + monkeypatch.setattr(PlinderRelease, "fetch", lambda *a, **kw: pytest.fail("fetch")) + for filters in [None, [], [[]]]: + with pytest.raises(ValueError, match="filter"): + query_score_table( + "interface_similarity_scores", release=PlinderRelease(), filters=filters + ) + + +def test_missing_shard_does_not_fall_back(stored_scores): + _, destination, _, manifest = stored_scores + root = destination / "score_store/interface_similarity_scores" + (root / manifest["data"][score_bucket("1abc__1")][0]).unlink() + with pytest.raises(FileNotFoundError, match="score-store"): + query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=[("query_system", "==", "1abc__1")], + ) + + +def test_bad_manifest_is_not_legacy(stored_scores): + _, destination, _, _ = stored_scores + path = destination / "score_store/interface_similarity_scores/manifest.json" + path.write_text(json.dumps({"version": 999})) + with pytest.raises(ValueError, match="manifest"): + query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=[("similarity", ">", 50)], + ) + + +def test_export_never_overwrites_existing_store(stored_scores): + source, destination, _, _ = stored_scores + with pytest.raises(FileExistsError): + export_score_store(source, destination, "interface_similarity_scores") + + +def test_export_isolates_engines_and_builds_exact_target_index(tmp_path, monkeypatch): + import plinder.data.pipeline.score_store as exporter + + source = tmp_path / "source" + path = source / "exports/interface_similarity_scores.parquet" + path.parent.mkdir(parents=True) + frame = pd.DataFrame( + [(f"1a{q}b__1", f"2c{t}d__1", 80) for q in "0123" for t in "0123456789ab"], + columns=["query_system", "target_system", "similarity"], + ) + frame = pd.concat([frame, frame.iloc[:1]], ignore_index=True) + frame.to_parquet(path, index=False) + original = exporter._connection + connections = [] + active = 0 + + @contextmanager + def isolated(*args): + nonlocal active + assert active == 0 + active += 1 + try: + with original(*args) as con: + connections.append(con) + yield con + finally: + active -= 1 + + monkeypatch.setattr(exporter, "_connection", isolated) + destination = tmp_path / "out" + manifest = export_score_store(source, destination, "interface_similarity_scores") + root = destination / "score_store/interface_similarity_scores" + index = pd.concat( + [ + pd.read_parquet(root / p) + for paths in manifest["targets"].values() + for p in paths + ] + ) + expected = frame.assign( + column="target_system", + value=frame["target_system"], + shard=frame["query_system"].map(score_bucket), + )[["column", "value", "shard"]].drop_duplicates() + pd.testing.assert_frame_equal(_canonical(index), _canonical(expected)) + assert manifest["rows"] == len(frame) # Index deduplication must not drop scores. + assert not list(root.glob("index-parts-*")) + # One initial partitioning engine, one sort and one index engine per query + # bucket, then between one and eight coarse target-index engines. + assert ( + 1 + 2 * len(manifest["data"]) + < len(connections) + <= 9 + 2 * len(manifest["data"]) + ) + for con in connections: + with pytest.raises(Exception, match="closed"): + con.execute("SELECT 1") + + +def test_export_failure_removes_staging_and_preserves_source(tmp_path, monkeypatch): + import plinder.data.pipeline.score_store as exporter + + source = tmp_path / "source" + path = source / "exports/interface_similarity_scores.parquet" + path.parent.mkdir(parents=True) + pd.DataFrame( + {"query_system": ["1abc__1"], "target_system": ["2def__1"], "similarity": [80]} + ).to_parquet(path, index=False) + before = path.read_bytes() + + def fail(*args, **kwargs): + raise RuntimeError("injected sort failure") + + monkeypatch.setattr(exporter, "_sort_bucket", fail) + destination = tmp_path / "out" + with pytest.raises(RuntimeError, match="injected sort failure"): + export_score_store(source, destination, "interface_similarity_scores") + assert path.read_bytes() == before + assert not list((destination / "score_store").iterdir()) + + +def test_numeric_buckets_and_duplicate_rows(tmp_path): + source = tmp_path / "source" + (source / "ligand_scores").mkdir(parents=True) + frame = pd.DataFrame( + { + "query_ligand_id": [29, 1025, 29], + "target_ligand_id": [1025, 29, 1025], + "score": [91, 32, 91], + } + ) + frame.to_parquet(source / "ligand_scores/part.parquet", index=False) + export_score_store(source, tmp_path / "out", "ligand_scores") + result = query_score_table( + "ligand_scores", + release=PlinderRelease(tmp_path / "out"), + filters=[("query_ligand_id", "==", 29)], + ) + assert result["score"].tolist() == [91, 91] + reverse = query_score_table( + "ligand_scores", + release=PlinderRelease(tmp_path / "out"), + filters=[("target_ligand_id", "==", 29.0)], + ) + assert reverse["score"].tolist() == [32] + + +def test_query_aliases_route_to_the_same_entry_bucket(tmp_path): + source = tmp_path / "source" + path = source / "exports/ligand_similarity_scores.parquet" + path.parent.mkdir(parents=True) + pd.DataFrame( + { + "query_system": ["query__1__A__B", "1abc__1__A__B"], + "query_ligand_id": ["query__1__B", "1abc__1__B"], + "target_system": ["2def__1__C__D"] * 2, + "target_ligand_id": ["2def__1__D"] * 2, + "score": [91, 32], + } + ).to_parquet(path, index=False) + output = tmp_path / "output" + export_score_store(source, output, "ligand_similarity_scores") + for column, value in [ + ("query_ligand_id", "query__1__B"), + ("target_ligand_id", "2def__1__D"), + ]: + filters = [(column, "==", value)] + result = query_score_table( + "ligand_similarity_scores", release=PlinderRelease(output), filters=filters + ) + pd.testing.assert_frame_equal( + _canonical(result), _canonical(pd.read_parquet(path, filters=filters)) + ) + + +@pytest.fixture +def score_mirror(stored_scores, tmp_path, monkeypatch): + from plinder.core import release as release_module + from plinder.core.utils import cpl + + _, origin, _, manifest = stored_scores + requests = [] + + class Handler(SimpleHTTPRequestHandler): + def log_message(self, *args): + pass + + def do_GET(self): + requests.append(unquote(self.path)) + super().do_GET() + + server = ThreadingHTTPServer( + ("127.0.0.1", 0), partial(Handler, directory=str(origin)) + ) + thread = Thread(target=server.serve_forever, daemon=True) + thread.start() + cfg = SimpleNamespace( + data=SimpleNamespace( + plinder_release="2026-09", + plinder_release_number="", + plinder_dir=str(tmp_path / "cache"), + plinder_remote=f"http://127.0.0.1:{server.server_port}", + force_update=False, + ) + ) + monkeypatch.setattr(cpl, "get_config", lambda: cfg) + monkeypatch.setattr(release_module, "get_config", lambda: cfg) + monkeypatch.delenv("PLINDER_OFFLINE", raising=False) + monkeypatch.delenv("PLINDER_OFFLINE_MODE", raising=False) + yield cfg, requests, manifest + server.shutdown() + server.server_close() + thread.join() + + +def test_http_downloads_only_manifest_and_needed_shard(score_mirror, monkeypatch): + cfg, requests, manifest = score_mirror + filters = [("query_system", "==", "1abc__1")] + expected_requests = ["/score_store/interface_similarity_scores/manifest.json"] + [ + "/score_store/interface_similarity_scores/" + p + for p in manifest["data"][score_bucket("1abc__1")] + ] + result = query_score_table( + "interface_similarity_scores", release=PlinderRelease(), filters=filters + ) + assert result["similarity"].tolist() == [91, 91] + assert requests == expected_requests + requests.clear() + query_score_table( + "interface_similarity_scores", release=PlinderRelease(), filters=filters + ) + assert requests == [] # Metadata checks don't re-download cached files. + monkeypatch.setenv("PLINDER_OFFLINE", "true") + result = query_score_table( + "interface_similarity_scores", release=PlinderRelease(), filters=filters + ) + assert len(result) == 2 + assert requests == [] + # A different bucket is absent, not a valid empty result in a partial cache. + with pytest.raises(FileNotFoundError, match="score-store"): + query_score_table( + "interface_similarity_scores", + release=PlinderRelease(), + filters=[("query_system", "==", "2def__1")], + ) + assert requests == [] + + +def test_http_target_only_lookup_preserves_direction(score_mirror): + _, requests, manifest = score_mirror + result = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(), + filters=[("target_system", "==", "1abc__1")], + ) + assert result["similarity"].tolist() == [32] + expected = ( + ["manifest.json"] + + manifest["targets"][score_bucket("1abc__1")] + + manifest["data"][score_bucket("2def__1")] + ) + assert requests == [ + "/score_store/interface_similarity_scores/" + p for p in expected + ] + + +def test_http_unknown_query_downloads_no_score_file(score_mirror): + _, requests, _ = score_mirror + result = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(), + filters=[("query_system", "==", "missing")], + ) + assert result.empty + assert requests == ["/score_store/interface_similarity_scores/manifest.json"] + + +def test_export_preserves_hive_columns_and_uint8(tmp_path): + source = tmp_path / "source" + path = ( + source / "exports/protein_similarity_scores/alignment_type=mmseqs/part.parquet" + ) + path.parent.mkdir(parents=True) + pd.DataFrame( + { + "query_entry": ["1abc"], + "target_entry": ["2def"], + "qcov": pd.Series([80], dtype="uint8"), + } + ).to_parquet(path, index=False) + output = tmp_path / "output" + export_score_store(source, output, "protein_similarity_scores") + filters = [("query_entry", "==", "1abc")] + expected = pd.read_parquet(path.parents[1], filters=filters) + result = query_score_table( + "protein_similarity_scores", release=PlinderRelease(output), filters=filters + ) + pd.testing.assert_frame_equal(expected, result) + + +def test_empty_export_retains_schema(tmp_path): + source = tmp_path / "source" + path = source / "exports/interface_similarity_scores.parquet" + path.parent.mkdir(parents=True) + pd.DataFrame( + { + "query_system": pd.Series(dtype="str"), + "target_system": pd.Series(dtype="str"), + "similarity": pd.Series(dtype="int8"), + } + ).to_parquet(path, index=False) + export_score_store(source, tmp_path / "output", "interface_similarity_scores") + result = query_score_table( + "interface_similarity_scores", + release=PlinderRelease(tmp_path / "output"), + filters=[("similarity", ">", 0)], + ) + assert result.empty + assert str(result["similarity"].dtype) == "int8" + + +def test_unsafe_manifest_path_is_rejected_before_download(stored_scores): + _, destination, _, manifest = stored_scores + path = destination / "score_store/interface_similarity_scores/manifest.json" + manifest["data"][score_bucket("1abc__1")] = ["../../../outside.parquet"] + path.write_text(json.dumps(manifest)) + with pytest.raises(ValueError, match="unsafe"): + query_score_table( + "interface_similarity_scores", + release=PlinderRelease(destination), + filters=[("query_system", "==", "1abc__1")], + ) diff --git a/tests/core/test_search.py b/tests/core/test_search.py new file mode 100644 index 00000000..7922b1f2 --- /dev/null +++ b/tests/core/test_search.py @@ -0,0 +1,83 @@ +"""Public search modes select the requested comparisons.""" + +import importlib + +import pandas as pd +import pytest + + +@pytest.mark.parametrize( + "mode,ligands,interfaces", + [ + ("auto", None, True), + ("pockets", False, False), + ("ligands", True, False), + ("interfaces", False, True), + ("all", None, True), + ], +) +@pytest.mark.parametrize("include_monomers", [True, False]) +def test_search_mode_dispatch( + tmp_path, monkeypatch, mode, ligands, interfaces, include_monomers +): + module = importlib.import_module("plinder.core.scores.search") + source = tmp_path / "query.cif" + source.write_text("data_query\n") + table = pd.DataFrame({"input_id": ["query"], "structure_path": [str(source)]}) + monkeypatch.setattr(module, "score_custom_cif_files", lambda sources, **kw: kw) + + result = module.search( + table, + output_dir=tmp_path / "output", + mode=mode, + include_monomers=include_monomers, + ) + + assert result["include_ligands"] is ligands + assert result["include_interfaces"] is interfaces + assert result["include_monomers"] is include_monomers + + +@pytest.mark.parametrize("mode", ["auto", "pockets", "interfaces", "all"]) +def test_fasta_discovers_sites(tmp_path, monkeypatch, mode): + module = importlib.import_module("plinder.core.scores.search") + monkeypatch.setattr(module, "score_custom_sequence_file", lambda source, **kw: kw) + result = module.search( + tmp_path / "query.fa", output_dir=tmp_path / "output", mode=mode + ) + assert result["include_interfaces"] is (mode != "pockets") + assert result["backends"] == ("mmseqs",) + assert result["include_monomers"] is True + + +@pytest.mark.parametrize("mode", ["all", "ligands"]) +def test_structure_file_does_not_require_a_table(tmp_path, monkeypatch, mode): + module = importlib.import_module("plinder.core.scores.search") + monkeypatch.setattr(module, "score_custom_cif_files", lambda sources, **kw: kw) + source = tmp_path / "query.cif" + source.write_text("data_query\n") + result = module.search(source, output_dir=tmp_path / "output", mode=mode) + assert result["include_ligands"] is (True if mode == "ligands" else None) + assert result["include_interfaces"] is (mode != "ligands") + + +def test_fasta_cannot_compare_ligands(tmp_path): + module = importlib.import_module("plinder.core.scores.search") + with pytest.raises(ValueError, match="FASTA inputs"): + module.search( + tmp_path / "query.fa", output_dir=tmp_path / "output", mode="ligands" + ) + + +@pytest.mark.parametrize("mode", ["invalid", "both"]) +def test_invalid_mode_lists_all(tmp_path, mode): + module = importlib.import_module("plinder.core.scores.search") + with pytest.raises(ValueError, match="interfaces or all"): + module.search(tmp_path / "query.cif", output_dir=tmp_path / "output", mode=mode) + + +def test_default_runs_applicable_features(tmp_path, monkeypatch): + module = importlib.import_module("plinder.core.scores.search") + monkeypatch.setattr(module, "score_custom_sequence_file", lambda source, **kw: kw) + result = module.search(tmp_path / "query.fa", output_dir=tmp_path / "output") + assert result["include_interfaces"] is True diff --git a/tests/core/test_structure_inputs.py b/tests/core/test_structure_inputs.py index c4d0d819..56691d17 100644 --- a/tests/core/test_structure_inputs.py +++ b/tests/core/test_structure_inputs.py @@ -266,16 +266,14 @@ def test_search_dispatches_fasta_folder_and_table(test_dir, tmp_path, monkeypatc original = test_dir / "reconstructed_systems/1avd__1__1.A__1.C/receptor.cif" (folder / "one.cif").write_bytes(original.read_bytes()) module.search(folder, output_dir=tmp_path / "folder_search") - assert calls[-1][1]["include_ligands"] is False + assert calls[-1][1]["include_ligands"] is None table = pd.DataFrame({"input_id": ["custom_id"], "structure_path": [str(original)]}) module.search(table, output_dir=tmp_path / "table_search", mode="ligands") assert calls[-1][0][0].name == "custom_id.cif" assert calls[-1][1]["include_ligands"] is True - with pytest.raises(ValueError, match="input table"): - module.search(folder, output_dir=tmp_path / "bad", mode="ligands") with pytest.raises(ValueError, match="FASTA inputs"): module.search( - tmp_path / "query.fa", output_dir=tmp_path / "bad", mode="interfaces" + tmp_path / "query.fa", output_dir=tmp_path / "bad", mode="ligands" ) diff --git a/tests/core/test_table_query.py b/tests/core/test_table_query.py index 5e10dacb..712800f9 100644 --- a/tests/core/test_table_query.py +++ b/tests/core/test_table_query.py @@ -328,6 +328,40 @@ def test_requested_entry_metadata_is_joined_automatically( assert pd.isna(result["entry_resolution"].iloc[2]) +def test_per_ligand_validation_needs_no_system_table(tmp_path: Path) -> None: + release = PlinderRelease(tmp_path) + _write_table( + release, + "annotation_table", + { + "ligand_id": ["9hb7__1__1.F", "9hb7__1__1.E"], + "ligand_ccd_code": ["JCR", "FE"], + "ligand_validation_average_rscc": [0.852, 0.984], + "ligand_pocket_validation_average_rscc": [0.91, None], + }, + ) + result = query_table( + columns=[ + "ligand_id", + "ligand_validation_average_rscc", + "ligand_pocket_validation_average_rscc", + ], + filters=[ + ("ligand_ccd_code", "==", "JCR"), + ("ligand_validation_average_rscc", ">=", 0.8), + ], + release=release, + ) + assert result.to_dict("records") == [ + { + "ligand_id": "9hb7__1__1.F", + "ligand_validation_average_rscc": 0.852, + "ligand_pocket_validation_average_rscc": 0.91, + } + ] + assert not release.path("system_validation").exists() + + def test_requested_system_validation_is_joined_automatically( local_release: PlinderRelease, ) -> None: diff --git a/tests/data/pipeline/test_collate.py b/tests/data/pipeline/test_collate.py index 1599a320..f4647b41 100644 --- a/tests/data/pipeline/test_collate.py +++ b/tests/data/pipeline/test_collate.py @@ -140,6 +140,7 @@ def _write_entry( "chain_modified_residues": [[], []], "chain_length": [300, 200], "chain_num_unresolved_residues": [0, 0], + "chain_unresolved_fraction": [0.0, 0.0], "chain_is_holo": [True, True], "chain_is_ligand_like": [False, False], "chain_uniprot_ids": [["P12345"], ["Q12345"]], @@ -464,8 +465,23 @@ def test_collation_keeps_unknown_ligand_interest_boolean(tmp_path): assert flag.null_count == table.num_rows == 2 -def test_weekly_removal_only_needs_no_raw_entries(entry_update_case, tmp_path): +@pytest.mark.parametrize("fraction_column", ["present", "absent", "null"]) +def test_weekly_removal_only_needs_no_raw_entries( + entry_update_case, tmp_path, fraction_column +): base, args, calls = entry_update_case + chain_path = base / "index/entry_chains.parquet" + chains = pd.read_parquet(chain_path) + chains.loc[chains.chain_asym_id.eq("A"), "chain_num_unresolved_residues"] = 30 + chains["chain_unresolved_fraction"] = ( + chains.chain_num_unresolved_residues / chains.chain_length + ) + if fraction_column == "absent": + chains = chains.drop(columns="chain_unresolved_fraction") + elif fraction_column == "null": + chains["chain_unresolved_fraction"] = None + chains.to_parquet(chain_path, index=False) + before = _index_bytes(base) original_plan = json.loads((args["plan_dir"] / "plan.json").read_text()) plan_dir = tmp_path / "removal-plan" updates.plan_update( @@ -483,6 +499,7 @@ def test_weekly_removal_only_needs_no_raw_entries(entry_update_case, tmp_path): report = update_entries.apply_entry_update(**args) assert report["ingested_pdb_ids"] == [] assert calls == [] + assert _index_bytes(base) == before for filename, _ in update_entries.ENTRY_TABLES.values(): frame = pd.read_parquet(args["output_dir"] / "index" / filename) ids = ( @@ -491,6 +508,56 @@ def test_weekly_removal_only_needs_no_raw_entries(entry_update_case, tmp_path): else frame.system_id.str.split("__").str[0] ) assert set(ids) == {"1abc", "3ghi"} + updated_chains = pd.read_parquet(args["output_dir"] / "index/entry_chains.parquet") + assert updated_chains["chain_unresolved_fraction"].tolist() == [0.1, 0.0] * 2 + + +@pytest.mark.parametrize("fraction_column", ["absent", "null"]) +def test_weekly_update_derives_missing_retained_chain_fraction( + entry_update_case, fraction_column +): + base, args, _ = entry_update_case + path = base / "index/entry_chains.parquet" + chains = pd.read_parquet(path) + retained = chains.entry_pdb_id.eq("3ghi") & chains.chain_asym_id.eq("A") + chains.loc[retained, "chain_num_unresolved_residues"] = 60 + if fraction_column == "absent": + chains = chains.drop(columns="chain_unresolved_fraction") + else: + chains.loc[retained, "chain_unresolved_fraction"] = None + chains.to_parquet(path, index=False) + before = _index_bytes(base) + + report = update_entries.apply_entry_update(**args) + + assert report["status"] == "requires_downstream_repair" + assert _index_bytes(base) == before + updated = pd.read_parquet(args["output_dir"] / "index/entry_chains.parquet") + row = updated.loc[ + updated.entry_pdb_id.eq("3ghi") & updated.chain_asym_id.eq("A") + ].iloc[0] + assert row.chain_unresolved_fraction == 0.2 + incoming = updated.loc[updated.entry_pdb_id.isin(["1abc", "4jkl"])] + assert incoming.chain_unresolved_fraction.tolist() == [0.0] * len(incoming) + + +@pytest.mark.parametrize("supplied_fraction", [0.25, -0.1, 1.1]) +def test_weekly_update_rejects_incorrect_retained_chain_fraction( + entry_update_case, supplied_fraction +): + base, args, _ = entry_update_case + path = base / "index/entry_chains.parquet" + chains = pd.read_parquet(path) + retained = chains.entry_pdb_id.eq("3ghi") & chains.chain_asym_id.eq("A") + chains.loc[retained, "chain_unresolved_fraction"] = supplied_fraction + chains.to_parquet(path, index=False) + before = _index_bytes(base) + + with pytest.raises(ValueError, match="incorrect_unresolved_fraction"): + update_entries.apply_entry_update(**args) + + assert _index_bytes(base) == before + assert not (args["output_dir"] / "index/collation.json").exists() def test_weekly_update_failure_and_resume_preserve_base(entry_update_case, monkeypatch): @@ -803,6 +870,8 @@ def test_plan_shards_and_finalize_release_contract(tmp_path: Path) -> None: "entry_sources": 2, } assert report["interface_count"] == 2 + chains = pd.read_parquet(tmp_path / "index/entry_chains.parquet") + assert chains["chain_unresolved_fraction"].tolist() == [0.0] * 4 annotation = pd.read_parquet(tmp_path / "index/annotation_table.parquet") assert [column for column in annotation.columns if column.startswith("entry_")] == [ "entry_pdb_id" @@ -1197,15 +1266,28 @@ def test_targeted_repair_preserves_unaffected_release_only_columns( assert finalized_report["downstream_repair_complete"] is True -def test_targeted_repair_updates_chain_annotations(tmp_path: Path) -> None: +@pytest.mark.parametrize("fraction_column", ["present", "absent", "null"]) +def test_targeted_repair_updates_chain_annotations( + tmp_path: Path, fraction_column: str +) -> None: _write_release(tmp_path) run_collation(tmp_path, memory_limit="1GB") path = tmp_path / "index/entry_chains.parquet" installed = pd.read_parquet(path) installed["release_only"] = "keep" + installed.loc[installed.chain_asym_id.eq("A"), "chain_num_unresolved_residues"] = 30 + installed["chain_unresolved_fraction"] = ( + installed.chain_num_unresolved_residues / installed.chain_length + ) + if fraction_column == "absent": + installed = installed.drop(columns="chain_unresolved_fraction") + elif fraction_column == "null": + installed["chain_unresolved_fraction"] = None installed.to_parquet(path, index=False) replacement_path = tmp_path / "raw_entries/ab/1abc/entry_chains.parquet" replacement = pd.read_parquet(replacement_path) + replacement.loc[0, "chain_num_unresolved_residues"] = 30 + replacement.loc[0, "chain_unresolved_fraction"] = 0.1 replacement.loc[0, "chain_sequence_noncanonical"] = "(MSE)" + "A" * 299 replacement.at[0, "chain_modified_residues"] = ["1:MSE:A:1"] replacement.to_parquet(replacement_path, index=False) @@ -1241,10 +1323,65 @@ def test_targeted_repair_updates_chain_annotations(tmp_path: Path) -> None: selected = repaired.loc[repaired.entry_pdb_id.eq("1abc"), replacement.columns] pd.testing.assert_frame_equal(selected.reset_index(drop=True), replacement) untouched = repaired.loc[repaired.entry_pdb_id.eq("2def"), installed.columns] + if fraction_column == "null": + installed["chain_unresolved_fraction"] = ( + installed.chain_num_unresolved_residues / installed.chain_length + ) pd.testing.assert_frame_equal( untouched.reset_index(drop=True), installed.loc[installed.entry_pdb_id.eq("2def")].reset_index(drop=True), ) + assert repaired["chain_unresolved_fraction"].tolist() == [0.1, 0.0] * 2 + + +@pytest.mark.parametrize("fraction_column", ["absent", "null"]) +def test_metadata_only_repair_derives_missing_chain_fraction( + tmp_path: Path, fraction_column: str +) -> None: + _write_release(tmp_path) + run_collation(tmp_path, memory_limit="1GB") + chain_path = tmp_path / "index/entry_chains.parquet" + chains = pd.read_parquet(chain_path) + retained = chains.entry_pdb_id.eq("2def") & chains.chain_asym_id.eq("A") + chains.loc[retained, "chain_num_unresolved_residues"] = 60 + chains["release_only"] = "keep" + if fraction_column == "absent": + chains = chains.drop(columns="chain_unresolved_fraction") + else: + chains.loc[retained, "chain_unresolved_fraction"] = None + chains.to_parquet(chain_path, index=False) + metadata_path = tmp_path / "raw_entries/ab/1abc/entry_metadata.parquet" + metadata = pd.read_parquet(metadata_path) + metadata["entry_pH"] = 6.5 + metadata.to_parquet(metadata_path, index=False) + + report = repair_collation(tmp_path, ["1abc"], memory_limit="1GB") + + assert report["status"] == "requires_downstream_repair" + repaired = pd.read_parquet(chain_path) + assert repaired["chain_unresolved_fraction"].tolist() == [0.0, 0.0, 0.2, 0.0] + assert repaired["release_only"].tolist() == ["keep"] * len(chains) + repaired_metadata = pd.read_parquet(tmp_path / "index/entry_metadata.parquet") + assert repaired_metadata.set_index("entry_pdb_id").loc["1abc", "entry_pH"] == 6.5 + + +@pytest.mark.parametrize("supplied_fraction", [0.25, -0.1, 1.1]) +def test_targeted_repair_rejects_incorrect_retained_chain_fraction( + tmp_path: Path, supplied_fraction: float +) -> None: + _write_release(tmp_path) + run_collation(tmp_path, memory_limit="1GB") + path = tmp_path / "index/entry_chains.parquet" + chains = pd.read_parquet(path) + retained = chains.entry_pdb_id.eq("2def") & chains.chain_asym_id.eq("A") + chains.loc[retained, "chain_unresolved_fraction"] = supplied_fraction + chains.to_parquet(path, index=False) + before = _index_bytes(tmp_path) + + with pytest.raises(ValueError, match="incorrect_unresolved_fraction"): + repair_collation(tmp_path, ["1abc"], memory_limit="1GB") + + assert _index_bytes(tmp_path) == before @pytest.mark.parametrize( @@ -1485,6 +1622,17 @@ def test_final_validation_rejects_invalid_chain_sequence_metadata( run_collation(tmp_path, threads=1, memory_limit="1GB") +def test_final_validation_rejects_incorrect_unresolved_fraction(tmp_path: Path) -> None: + _write_release(tmp_path) + chain_path = tmp_path / "raw_entries/ab/1abc/entry_chains.parquet" + chains = pd.read_parquet(chain_path) + chains.loc[0, "chain_unresolved_fraction"] = 0.25 + chains.to_parquet(chain_path, index=False) + + with pytest.raises(ValueError, match="incorrect_unresolved_fraction"): + run_collation(tmp_path, threads=1, memory_limit="1GB") + + def test_final_validation_rejects_missing_chain_sequence(tmp_path: Path) -> None: _write_release(tmp_path) chain_path = tmp_path / "raw_entries/ab/1abc/entry_chains.parquet" diff --git a/tests/data/pipeline/test_end_to_end.py b/tests/data/pipeline/test_end_to_end.py index 21f9a5d8..82fb2ad7 100644 --- a/tests/data/pipeline/test_end_to_end.py +++ b/tests/data/pipeline/test_end_to_end.py @@ -1,11 +1,29 @@ # Copyright (c) 2024, Plinder Development Team # Distributed under the terms of the Apache License 2.0 +from shutil import copy2 + +import pyarrow.parquet as pq + from plinder.data.pipeline import config, io, pipeline, tasks -def test_end_to_end(mock_alternative_datasets, monkeypatch): +def test_end_to_end(mock_alternative_datasets, monkeypatch, test_dir): mock_alternative_datasets("4jvm") # different from 19hc + def local_archive(*, kind, data_dir, two_char_code): + assert two_char_code == "9h" + destination = data_dir / two_char_code + destination.mkdir(parents=True, exist_ok=True) + if kind == "cif": + source = test_dir / "xx/pdb_000019hc/pdb_000019hc_xyz-enrich.cif.gz" + destination = destination / source.parent.name + destination.mkdir(exist_ok=True) + copy2(source, destination / source.name) + else: + assert kind == "val" + + monkeypatch.setattr(io, "rsync_rcsb", local_archive) + # refresh_bundled_ccd re-syncs biotite's bundled CCD from wwPDB via # setup_ccd (a heavy online download); the shipped bt_info snapshot is # already usable, so stub it out for the offline end-to-end run. @@ -27,6 +45,13 @@ def test_end_to_end(mock_alternative_datasets, monkeypatch): }, "flow": { "run_specific_stages": stages, + "make_entries_cpu": 1, + "protein_clustering_cpu": 1, + "make_dbs_cpu": 1, + }, + "source": { + "pdb_nextgen_root": "ingest", + "validation_root": "reports", }, "scorer": { "sub_databases": "holo", @@ -35,3 +60,6 @@ def test_end_to_end(mock_alternative_datasets, monkeypatch): cfg = config.get_config(config=conf) pipe = pipeline.IngestPipeline(conf=cfg) pipe.run() + annotations = pq.read_table(pipe.plinder_dir / "index/annotation_table.parquet") + assert annotations.num_rows > 0 + assert set(annotations["entry_pdb_id"].to_pylist()) == {"19hc"} diff --git a/tests/data/pipeline/test_public_release.py b/tests/data/pipeline/test_public_release.py index e44080f7..b87dc4e2 100644 --- a/tests/data/pipeline/test_public_release.py +++ b/tests/data/pipeline/test_public_release.py @@ -1,3 +1,4 @@ +import json from pathlib import Path import pytest @@ -102,6 +103,52 @@ def test_public_release_replacement_drops_old_files(tmp_path: Path) -> None: ).stat().st_ino +def test_public_release_includes_only_completed_score_store_files( + tmp_path: Path, +) -> None: + working = tmp_path / "working" + _working_release(working) + root = working / "score_store/interface_similarity_scores" + root.mkdir(parents=True) + (root / "data.parquet").write_bytes(b"scores") + (root / "target.parquet").write_bytes(b"index") + (root / "unlisted.parquet").write_bytes(b"private work") + (root / "manifest.json").write_text( + json.dumps( + { + "version": 1, + "artifact": "interface_similarity_scores", + "data": {"ab": ["data.parquet"]}, + "targets": {"de": ["target.parquet"]}, + } + ) + ) + public = tmp_path / "public" + prepare_public_release(working, public) + assert (public / root.relative_to(working) / "manifest.json").is_file() + assert (public / root.relative_to(working) / "data.parquet").is_file() + assert not (public / root.relative_to(working) / "unlisted.parquet").exists() + + +def test_public_release_rejects_score_store_traversal(tmp_path: Path) -> None: + working = tmp_path / "working" + _working_release(working) + root = working / "score_store/interface_similarity_scores" + root.mkdir(parents=True) + (root / "manifest.json").write_text( + json.dumps( + { + "version": 1, + "artifact": "interface_similarity_scores", + "data": {"ab": ["../../private.parquet"]}, + "targets": {}, + } + ) + ) + with pytest.raises(ValueError, match="unsafe"): + prepare_public_release(working, tmp_path / "public") + + def test_missing_public_file_does_not_replace_release(tmp_path: Path) -> None: working = tmp_path / "working" public = tmp_path / "public" diff --git a/tests/data/test_chain_modifications.py b/tests/data/test_chain_modifications.py index a2f49d73..6f517d57 100644 --- a/tests/data/test_chain_modifications.py +++ b/tests/data/test_chain_modifications.py @@ -3,10 +3,13 @@ from __future__ import annotations import pyarrow as pa +import pyarrow.parquet as pq from plinder.data.annotations.aggregate_annotations import Entry from plinder.data.annotations.cif_utils import read_mmcif_file +from plinder.data.annotations.interface_utils import INTERFACE_ANNOTATION_SCHEMA from plinder.data.annotations.protein_utils import get_modified_residues +from plinder.data.get_system_annotations import GetPlinderAnnotation from plinder.data.pipeline.collate import ENTRY_CHAIN_SCHEMA CIF_6M92 = "xx/pdb_00006m92/pdb_00006m92_xyz-enrich.cif.gz" @@ -26,7 +29,9 @@ def test_modified_residues_fall_back_to_the_ccd_parent(test_dir): assert get_modified_residues(block) == {"C": ["33:SEP:C:18>SER"]} -def test_entry_chain_table_carries_modifications_and_noncanonical_sequence(test_dir): +def test_entry_chain_table_carries_modifications_and_noncanonical_sequence( + test_dir, tmp_path +): entry = Entry.from_cif_file(test_dir / CIF_6M92, include_interfaces=False) assert entry.chains["C"].modified_residues == ["33:SEP:C:18>SER (modified residue)"] @@ -46,3 +51,20 @@ def test_entry_chain_table_carries_modifications_and_noncanonical_sequence(test_ == entry.chain_to_seqres_noncanonical["C"] ) assert rows["A"]["chain_modified_residues"] == [] + for row in rows.values(): + assert row["chain_unresolved_fraction"] == ( + row["chain_num_unresolved_residues"] / row["chain_length"] + ) + + # Exercise the actual ingest writer, not just the in-memory export. + annotator = GetPlinderAnnotation(test_dir / CIF_6M92, "", save_folder=tmp_path) + annotator.entry = entry + annotator._write_shared_sidecars( + tmp_path, pa.Table.from_batches([], schema=INTERFACE_ANNOTATION_SCHEMA) + ) + stored = pq.read_table(tmp_path / "entry_chains.parquet") + assert stored["chain_unresolved_fraction"].type == pa.float64() + assert ( + stored["chain_unresolved_fraction"].to_pylist() + == frame["chain_unresolved_fraction"].tolist() + ) diff --git a/tests/data/test_ligand_validation.py b/tests/data/test_ligand_validation.py index f4838a53..e7814f7a 100644 --- a/tests/data/test_ligand_validation.py +++ b/tests/data/test_ligand_validation.py @@ -54,8 +54,17 @@ def test_unresolved_atoms_require_a_reference( "getRSR.return_value": 0.1, "getRSRZ.return_value": 0.2, "getRSCC.return_value": 0.9, - "getOccupancies.return_value": [1.0], - "getBFactors.return_value": [20.0], + "getOccupancies.return_value": [1.0] * 5, + "pdbx_residue": pd.DataFrame({"type_symbol": ["C"] * 5}), + "getBFactors.return_value": [20.0] * 5, + "conop": ( + SimpleNamespace( + atom_specs=[SimpleNamespace(element="C")] * reference_count + ) + if reference_count is not None + else None + ), + "ost": SimpleNamespace(atoms=[SimpleNamespace(element="C")] * 5), "getResName.return_value": "LIG", "countAtomsPDBX.return_value": 5, "countUnknownAtoms.return_value": 0, @@ -84,6 +93,94 @@ def test_unresolved_atoms_require_a_reference( residue.isAtomCountConsistent.assert_called_once_with() +@pytest.mark.parametrize( + ("isotope", "reference_isotope", "ost_isotope"), + [ + ("H", "H", "H"), + ("D", "H", "H"), + ("D", "D", "D"), + ("T", "H", "T"), + ("T", "T", "T"), + ("D", "D", "H"), + ("H", "D", "H"), + ], +) +@pytest.mark.parametrize("reference_heavy_atoms", [2, 3]) +def test_hydrogen_isotopes_do_not_change_validation_counts_or_averages( + monkeypatch, isotope, reference_isotope, ost_isotope, reference_heavy_atoms +): + from PDBValidation.Residue import Ligand + + # Use the dependency's real counter and consistency methods: they count + # D/T as heavy atoms unless PLINDER normalizes the counting view. + residue = Ligand.__new__(Ligand) + residue.pdbx_residue = pd.DataFrame( + { + "type_symbol": ["C", "O", isotope], + "occupancy": [1.0, 0.8, 0.0], + "B_iso_or_equiv": [10.0, 30.0, 999.0], + } + ) + residue.ost = SimpleNamespace( + atoms=[SimpleNamespace(element=e) for e in ["C", "O", ost_isotope]] + ) + residue.conop = SimpleNamespace( + atom_specs=[ + SimpleNamespace(element=e) + for e in ["C", "O"] + + (["N"] if reference_heavy_atoms == 3 else []) + + [reference_isotope] + ] + ) + residue.modelled_subgroup = { + "rsr": "0.1", + "rsrz": "0.2", + "rscc": "0.9", + "ligand_clashes_outlier": "yes", + } + residue.getAltCode = lambda: "." + residue.getInsertionCode = lambda: "." + residue.getResName = lambda: "LIG" + residue.hasClashingPartialOccupancyAtoms = lambda: False + original_counts = ( + residue.countAtomsHeavyPDBX(), + residue.countAtomsHeavyConop(), + residue.countAtomsHeavyOST(), + residue.isAtomCountConsistent(), + ) + monkeypatch.setattr( + "PDBValidation.Residue.Residue.CreateFromMmCIFPosition", + lambda *_args: SimpleNamespace( + listAlt=lambda: {"."}, getAlt=lambda _alt: residue + ), + ) + + validation = ResidueValidation.from_residue("A", 1, "1", object()) + + assert validation is not None + assert validation.atom_count == 3 # Total atoms still includes hydrogen isotopes. + assert validation.heavy_atom_count == 2 + assert validation.num_unresolved_heavy_atoms == reference_heavy_atoms - 2 + assert validation.is_atom_count_consistent is (reference_heavy_atoms == 2) + assert validation.average_occupancy == pytest.approx(0.9) + assert validation.average_b_factor == 20.0 + assert (validation.rsr, validation.rsrz, validation.rscc) == (0.1, 0.2, 0.9) + assert validation.is_outlier["clashes"] is True + assert ( + residue.countAtomsHeavyPDBX(), + residue.countAtomsHeavyConop(), + residue.countAtomsHeavyOST(), + residue.isAtomCountConsistent(), + ) == original_counts # Do not mutate the dependency's residue or library. + summary = ResidueListValidation.from_residues( + [validation], ResidueValidationThresholds(min_average_occupancy=0.8) + ) + assert summary is not None + assert summary.heavy_atom_count == 2 + assert summary.num_unresolved_heavy_atoms == reference_heavy_atoms - 2 + assert summary.percent_occupancy_over_threshold == 100.0 + + def _validation(unresolved: int | None) -> ResidueValidation: return ResidueValidation( altcode=".", diff --git a/tests/data/test_ligand_validation_export.py b/tests/data/test_ligand_validation_export.py new file mode 100644 index 00000000..0ded79b1 --- /dev/null +++ b/tests/data/test_ligand_validation_export.py @@ -0,0 +1,90 @@ +from types import SimpleNamespace + +import pytest + +from plinder.data.annotations.get_ligand_validation import ResidueValidation +from plinder.data.annotations.ligand_validation import ( + format_selection_validation, + validation_columns, +) + + +def _residue(rscc): + return SimpleNamespace( + validation=ResidueValidation( + altcode=".", + inscode=".", + rsr=0.1, + rsrz=0.2, + rscc=rscc, + average_occupancy=1.0, + average_b_factor=20.0, + unknown_residue=False, + atom_count=5, + unknown_atom_count=0, + heavy_atom_count=5, + num_unresolved_heavy_atoms=0, + is_outlier=dict.fromkeys( + ["geometry", "density", "chirality", "clashes"], False + ), + is_atom_count_consistent=True, + has_clashing_partial_occupancy_atoms=False, + alt_count=1, + ) + ) + + +def _chain(*residues): + return SimpleNamespace(residues=dict(enumerate(residues, 1)), validation=None) + + +def test_ligand_summary_excludes_other_system_ligands(): + chains = {"F": _chain(_residue(0.852)), "E": _chain(_residue(0.984))} + result = format_selection_validation({"1.F": [1]}, chains, prefix="ligand") + assert result["ligand_validation_average_rscc"] == pytest.approx(0.852) + assert result["ligand_validation_num_residues"] == 1 + assert set(result) == set(validation_columns("ligand")) + + +def test_multichain_summary_pools_residues_not_chain_means(): + chains = {"A": _chain(_residue(0.9), _residue(0.6)), "B": _chain(_residue(0.3))} + result = format_selection_validation( + {"1.A": [1, 2, 2], "1.B": [1]}, chains, prefix="ligand" + ) + assert result["ligand_validation_average_rscc"] == pytest.approx(0.6) + assert result["ligand_validation_num_residues"] == 3 + + +def test_missing_validation_is_null_and_partial_coverage_is_visible(): + chains = {"A": _chain(_residue(0.9), SimpleNamespace(validation=None))} + missing = format_selection_validation({"1.A": [2]}, chains, prefix="ligand") + assert all(value is None for value in missing.values()) + partial = format_selection_validation( + {"1.A": [1, 2]}, chains, prefix="ligand_pocket" + ) + assert partial["ligand_pocket_validation_num_processed_residues"] == 1 + assert partial["ligand_pocket_validation_num_residues"] == 2 + assert partial["ligand_pocket_validation_percent_processed_residues"] == 50 + assert partial["ligand_pocket_validation_num_unresolved_heavy_atoms"] is None + + +def test_collation_keeps_ligand_validation_independent_of_system(): + from plinder.data.pipeline.collate import _annotation_columns_to_publish + + columns = [ + "ligand_id", + *validation_columns("ligand"), + *validation_columns("ligand_pocket"), + "system_ligand_validation_average_rscc", + ] + assert _annotation_columns_to_publish(columns) == columns[:-1] + + +def test_ambiguous_validation_residue_remains_missing(monkeypatch): + def ambiguous(*args, **kwargs): + raise AssertionError("Duplicate alternative configurations") + + monkeypatch.setattr( + "PDBValidation.Residue.Residue.CreateFromMmCIFPosition", ambiguous + ) + assert ResidueValidation.from_residue("A", 1, "1", object()) is None diff --git a/tests/data/test_readme_counts.py b/tests/data/test_readme_counts.py new file mode 100644 index 00000000..b43bc149 --- /dev/null +++ b/tests/data/test_readme_counts.py @@ -0,0 +1,48 @@ +"""Check README counting units and assembly composition.""" + +from pathlib import Path +from runpy import run_path + +import duckdb + + +def test_release_counts(tmp_path: Path) -> None: + index = tmp_path / "index" + index.mkdir() + tables = { + "annotation_table": "SELECT * FROM (VALUES ('s1', true), ('s1', false), ('s2', true)) t(system_id, ligand_is_proper)", + "interface_annotation_table": "SELECT 'i1' AS system_id", + "entry_chains": """SELECT * FROM (VALUES + ('p', 'A', 'protein'), ('p', 'L', 'rna'), + ('d', 'A', 'dna'), ('r', 'A', 'rna'), ('h', 'A', 'dna+rna'), + ('c', 'A', 'protein'), ('c', 'B', 'dna'), + ('m', 'A', 'protein'), ('m', 'B', 'protein') + ) t(entry_pdb_id, chain_asym_id, chain_receptor_type)""", + "entry_biounit_chains": """SELECT * FROM (VALUES + ('p', '1', 'A', 'receptor'), ('p', '1', 'L', 'ligand'), + ('d', '1', 'A', 'receptor'), ('d', '2', 'A', 'receptor'), + ('r', '1', 'A', 'receptor'), ('h', '1', 'A', 'receptor'), + ('c', '1', 'A', 'receptor'), ('c', '1', 'B', 'receptor'), + ('m', '1', 'A', 'receptor'), ('m', '1', 'B', 'receptor') + ) t(entry_pdb_id, biounit_id, chain_asym_id, chain_role)""", + } + with duckdb.connect() as con: + for name, sql in tables.items(): + con.sql(sql).write_parquet(str(index / f"{name}.parquet")) + script = Path(__file__).resolve().parents[2] / "scripts" / "update_readme_counts.py" + counts = run_path(str(script))["release_counts"](tmp_path) + assert ( + counts + == """ +| Release contents | Count | +| --- | ---: | +| Ligand systems | 2 | +| Proper ligands | 2 | +| Protein–protein interfaces | 1 | +| Protein-monomer assemblies | 1 | +| Nucleic-acid-monomer assemblies | 4 | +| Protein–nucleic-acid assemblies | 1 | + +Nucleic-acid monomers: 2 DNA, 1 RNA, and 1 DNA/RNA hybrids. +""" + ) diff --git a/tests/test_data/docs_score_release/README.md b/tests/test_data/docs_score_release/README.md new file mode 100644 index 00000000..dd84e6bf --- /dev/null +++ b/tests/test_data/docs_score_release/README.md @@ -0,0 +1,28 @@ +# Documentation score examples + +This is a bounded slice of the September 2026 PLINDER release for executing +`similarity_and_representatives.ipynb` and `representatives_and_leakage.ipynb` +in CI and documentation deployment. It contains real published rows, not +synthetic scores. The manifest records the selection, row counts, and SHA-256 +of each generated file. + +Cluster IDs and representative assignments are copied from the full release. +They are not recalculated on this subset. Pair-score files retain both endpoints +within the example entries (and the ligand cover representative's entry). +The matched-molecular-pair table retains rows with either endpoint in the +example ligand SMILES IDs. + +Only these two notebooks use `PLINDER_SCORE_EXAMPLE_DATA_DIR`. Without it, +they query the configured full release. Other guides continue to use the +configured release and their usual structure/prediction downloads. This slice +is not a complete release and must not be used for benchmark construction. + +To regenerate from a complete, local September release, run on a compute node: + +```bash +python scripts/prepare_docs_score_release.py /path/to/PLINDER-2026-09 /tmp/docs-score-release +``` + +Review the manifest and copy the generated files into this directory. Then run +`tests/core/test_docs_score_release.py` and execute the two notebooks with +`PLINDER_SCORE_EXAMPLE_DATA_DIR` set to this directory's absolute path. diff --git a/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=ag.parquet b/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=ag.parquet new file mode 100644 index 00000000..8cc36c49 Binary files /dev/null and b/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=ag.parquet differ diff --git a/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=cm.parquet b/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=cm.parquet new file mode 100644 index 00000000..3b3d0e80 Binary files /dev/null and b/tests/test_data/docs_score_release/alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=cm.parquet differ diff --git a/tests/test_data/docs_score_release/exports/interface_half_similarity_scores.parquet b/tests/test_data/docs_score_release/exports/interface_half_similarity_scores.parquet new file mode 100644 index 00000000..42a058ab Binary files /dev/null and b/tests/test_data/docs_score_release/exports/interface_half_similarity_scores.parquet differ diff --git a/tests/test_data/docs_score_release/exports/interface_similarity_scores.parquet b/tests/test_data/docs_score_release/exports/interface_similarity_scores.parquet new file mode 100644 index 00000000..d9b28558 Binary files /dev/null and b/tests/test_data/docs_score_release/exports/interface_similarity_scores.parquet differ diff --git a/tests/test_data/docs_score_release/exports/ligand_similarity_scores.parquet b/tests/test_data/docs_score_release/exports/ligand_similarity_scores.parquet new file mode 100644 index 00000000..c856e9a0 Binary files /dev/null and b/tests/test_data/docs_score_release/exports/ligand_similarity_scores.parquet differ diff --git a/tests/test_data/docs_score_release/exports/monomer_similarity_scores/examples.parquet b/tests/test_data/docs_score_release/exports/monomer_similarity_scores/examples.parquet new file mode 100644 index 00000000..92d25fd6 Binary files /dev/null and b/tests/test_data/docs_score_release/exports/monomer_similarity_scores/examples.parquet differ diff --git a/tests/test_data/docs_score_release/exports/protein_similarity_scores/examples.parquet b/tests/test_data/docs_score_release/exports/protein_similarity_scores/examples.parquet new file mode 100644 index 00000000..147ee581 Binary files /dev/null and b/tests/test_data/docs_score_release/exports/protein_similarity_scores/examples.parquet differ diff --git a/tests/test_data/docs_score_release/index/alignment_chain_lookup.parquet b/tests/test_data/docs_score_release/index/alignment_chain_lookup.parquet new file mode 100644 index 00000000..3d78a956 Binary files /dev/null and b/tests/test_data/docs_score_release/index/alignment_chain_lookup.parquet differ diff --git a/tests/test_data/docs_score_release/index/annotation_table.parquet b/tests/test_data/docs_score_release/index/annotation_table.parquet new file mode 100644 index 00000000..90dd7be4 Binary files /dev/null and b/tests/test_data/docs_score_release/index/annotation_table.parquet differ diff --git a/tests/test_data/docs_score_release/index/entry_biounit_chains.parquet b/tests/test_data/docs_score_release/index/entry_biounit_chains.parquet new file mode 100644 index 00000000..5d76ad99 Binary files /dev/null and b/tests/test_data/docs_score_release/index/entry_biounit_chains.parquet differ diff --git a/tests/test_data/docs_score_release/index/entry_chains.parquet b/tests/test_data/docs_score_release/index/entry_chains.parquet new file mode 100644 index 00000000..c6e5cd2f Binary files /dev/null and b/tests/test_data/docs_score_release/index/entry_chains.parquet differ diff --git a/tests/test_data/docs_score_release/index/entry_metadata.parquet b/tests/test_data/docs_score_release/index/entry_metadata.parquet new file mode 100644 index 00000000..2f07f5b1 Binary files /dev/null and b/tests/test_data/docs_score_release/index/entry_metadata.parquet differ diff --git a/tests/test_data/docs_score_release/index/entry_sources.parquet b/tests/test_data/docs_score_release/index/entry_sources.parquet new file mode 100644 index 00000000..22954d47 Binary files /dev/null and b/tests/test_data/docs_score_release/index/entry_sources.parquet differ diff --git a/tests/test_data/docs_score_release/index/interface_annotation_table.parquet b/tests/test_data/docs_score_release/index/interface_annotation_table.parquet new file mode 100644 index 00000000..27e110bd Binary files /dev/null and b/tests/test_data/docs_score_release/index/interface_annotation_table.parquet differ diff --git a/tests/test_data/docs_score_release/index/interface_clusters.parquet b/tests/test_data/docs_score_release/index/interface_clusters.parquet new file mode 100644 index 00000000..4bd69c03 Binary files /dev/null and b/tests/test_data/docs_score_release/index/interface_clusters.parquet differ diff --git a/tests/test_data/docs_score_release/index/ligand_affinity.parquet b/tests/test_data/docs_score_release/index/ligand_affinity.parquet new file mode 100644 index 00000000..9dd1dc45 Binary files /dev/null and b/tests/test_data/docs_score_release/index/ligand_affinity.parquet differ diff --git a/tests/test_data/docs_score_release/index/ligand_clusters.parquet b/tests/test_data/docs_score_release/index/ligand_clusters.parquet new file mode 100644 index 00000000..cd3b0dfb Binary files /dev/null and b/tests/test_data/docs_score_release/index/ligand_clusters.parquet differ diff --git a/tests/test_data/docs_score_release/index/ligand_mmp_pairs.parquet b/tests/test_data/docs_score_release/index/ligand_mmp_pairs.parquet new file mode 100644 index 00000000..34899580 Binary files /dev/null and b/tests/test_data/docs_score_release/index/ligand_mmp_pairs.parquet differ diff --git a/tests/test_data/docs_score_release/index/ligand_pocket_membership.parquet b/tests/test_data/docs_score_release/index/ligand_pocket_membership.parquet new file mode 100644 index 00000000..0ec192f2 Binary files /dev/null and b/tests/test_data/docs_score_release/index/ligand_pocket_membership.parquet differ diff --git a/tests/test_data/docs_score_release/index/ligand_pocket_residues.parquet b/tests/test_data/docs_score_release/index/ligand_pocket_residues.parquet new file mode 100644 index 00000000..13eb479c Binary files /dev/null and b/tests/test_data/docs_score_release/index/ligand_pocket_residues.parquet differ diff --git a/tests/test_data/docs_score_release/index/system_validation.parquet b/tests/test_data/docs_score_release/index/system_validation.parquet new file mode 100644 index 00000000..699bdd57 Binary files /dev/null and b/tests/test_data/docs_score_release/index/system_validation.parquet differ diff --git a/tests/test_data/docs_score_release/ligand_sampling/directed_set_cover/metric=pocket_qcov/threshold=70.parquet b/tests/test_data/docs_score_release/ligand_sampling/directed_set_cover/metric=pocket_qcov/threshold=70.parquet new file mode 100644 index 00000000..986d5f79 Binary files /dev/null and b/tests/test_data/docs_score_release/ligand_sampling/directed_set_cover/metric=pocket_qcov/threshold=70.parquet differ diff --git a/tests/test_data/docs_score_release/ligand_scores/retained.parquet b/tests/test_data/docs_score_release/ligand_scores/retained.parquet new file mode 100644 index 00000000..7b38b9d0 Binary files /dev/null and b/tests/test_data/docs_score_release/ligand_scores/retained.parquet differ diff --git a/tests/test_data/docs_score_release/manifest.json b/tests/test_data/docs_score_release/manifest.json new file mode 100644 index 00000000..f644293d --- /dev/null +++ b/tests/test_data/docs_score_release/manifest.json @@ -0,0 +1,163 @@ +{ + "release": "2026-09", + "description": "Real release rows, not synthetic scores. Bounded integration-test input for the score and representative notebooks, not a standalone release or benchmark.", + "entries": [ + "4agc", + "4agi", + "5h47", + "7cm8", + "7eek" + ], + "files": [ + { + "path": "ligand_sampling/directed_set_cover/metric=pocket_qcov/threshold=70.parquet", + "rows": 1, + "selection": "4agi__1__1.R assignment at pocket_qcov=70", + "sha256": "f65d8844d3441cd1eed54b82309c53c94b3acc68e9fb4fd3bfca7b0e8064bc39" + }, + { + "path": "index/annotation_table.parquet", + "rows": 101, + "selection": "example entries and cover representative entries", + "sha256": "8d06c60b06fb8d9c81e8f841fa0110eeb5d72caff5e365096da793467c67ac23" + }, + { + "path": "index/interface_annotation_table.parquet", + "rows": 3, + "selection": "example entries", + "sha256": "4141ddc278ffb7d39dfa8ddb34bbf005cf0e619f25e3082c063dc2742cce27a4" + }, + { + "path": "index/entry_metadata.parquet", + "rows": 5, + "selection": "entry_pdb_id in example subset", + "sha256": "6ef624e18232705ec7085f8d59a717d4cd9dc2eadddc734369917b2c54669abd" + }, + { + "path": "index/entry_sources.parquet", + "rows": 5, + "selection": "entry_pdb_id in example subset", + "sha256": "84dbb311da61bc3a0e16108e9823fc12bfa26a9999f8f2e44027858170b6bc3b" + }, + { + "path": "index/entry_chains.parquet", + "rows": 20, + "selection": "entry_pdb_id in example subset", + "sha256": "fc7fe9e448cfecaf8807bdd9e137af4967b7823af1ea3f6449039613c0414bec" + }, + { + "path": "index/entry_biounit_chains.parquet", + "rows": 165, + "selection": "entry_pdb_id in example subset", + "sha256": "d7ffc0dae81e1a6c80eb12833fa2c2c96de977d7be5267fce724be22b480f117" + }, + { + "path": "index/system_validation.parquet", + "rows": 99, + "selection": "system_id in example subset", + "sha256": "3f9cc25f9a0dd47553ddbeab1bc8d3aa9fb8d195588e86179e78c2cd8ba72b70" + }, + { + "path": "index/ligand_clusters.parquet", + "rows": 101, + "selection": "ligand_id in example subset", + "sha256": "4365ab7f85d3830430a6ae45e0247f0deb1b5ba6a97e76faea1822b9d4b3fbdd" + }, + { + "path": "index/ligand_affinity.parquet", + "rows": 101, + "selection": "ligand_id in example subset", + "sha256": "7a015df9aa0bf746ad86ccb9f6c2963386c47a18e9d1a1f1dc032000399f8201" + }, + { + "path": "index/ligand_pocket_membership.parquet", + "rows": 101, + "selection": "ligand_id in example subset", + "sha256": "db171a2c1044137bcac0522553949371254dcd942ac4d84dbade623f3a7f6da7" + }, + { + "path": "index/interface_clusters.parquet", + "rows": 3, + "selection": "system_id in example subset", + "sha256": "f45ef6f0af64de21f288f15626a83a6a0857976d9866034bc2e6373111046d69" + }, + { + "path": "index/alignment_chain_lookup.parquet", + "rows": 20, + "selection": "entry_pdb_id in example subset", + "sha256": "276656e34a0517adc1ef21903ee1bf77943bb26f95774aad344a445bbac483d1" + }, + { + "path": "index/ligand_pocket_residues.parquet", + "rows": 1406, + "selection": "entry_pdb_id in example subset", + "sha256": "0031b1cfabdfc819cf0f049e1bdf6e80dfd5a4399c0bac19c840154201a6a969" + }, + { + "path": "protein_clusters/sequence.parquet", + "rows": 20, + "selection": "example entries", + "sha256": "7dead603438cb6cc32ba84d4d328fb53bbfe537ac6033a909728be06d6d46757" + }, + { + "path": "protein_clusters/structure.parquet", + "rows": 20, + "selection": "example entries", + "sha256": "c0fd8de996c98913399befedf97e531e9b97b68c0d31c7d4acab60c3704efa62" + }, + { + "path": "index/ligand_mmp_pairs.parquet", + "rows": 22458, + "selection": "either member in example SMILES IDs", + "sha256": "1a4148680987539b448a99ad6bd9a7d2a3b7102dc68f5c4043c9001c79118c08" + }, + { + "path": "exports/ligand_similarity_scores.parquet", + "rows": 1605, + "selection": "both endpoints in example subset", + "sha256": "5540db09c0095151a34284072825441c03011cb6f21255983f9404c4d9f08627" + }, + { + "path": "exports/interface_similarity_scores.parquet", + "rows": 2, + "selection": "both endpoints in example subset", + "sha256": "d0d72cfde80cdccc336bde74af12ff14771fc17cbf311876fed69e4bf1b83e59" + }, + { + "path": "exports/interface_half_similarity_scores.parquet", + "rows": 12, + "selection": "both endpoints in example subset", + "sha256": "67a3f879331575509637f04d1d60a86c37103f798582fc1d6960890f8c5738a1" + }, + { + "path": "ligand_scores/retained.parquet", + "rows": 6, + "selection": "both endpoints in example SMILES IDs", + "sha256": "f32be5c5047f2786292dd6a1c979093af950ffaa792230b13bb3637d32569661" + }, + { + "path": "exports/protein_similarity_scores/examples.parquet", + "rows": 524, + "selection": "both endpoints in example entries", + "sha256": "094690f94cb7411e63d90b17b77d5e0ecce10d63c1c653b414e2b5c40cbdadcd" + }, + { + "path": "exports/monomer_similarity_scores/examples.parquet", + "rows": 0, + "selection": "both endpoints in example entries", + "sha256": "fbd2f71db5bc00db097466d824025a6f3cfe06cfd5cb48871823d0990f379cd7" + }, + { + "path": "alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=ag.parquet", + "rows": 16, + "selection": "4agi self alignments", + "sha256": "c8f51a30afe4006551c7e9edaa6b0b75ce4da1dde6b8ae34c9bdd918b06675dc" + }, + { + "path": "alignment_cigars/search_db=holo/alignment_type=mmseqs/shard=cm.parquet", + "rows": 1, + "selection": "7cm8 self alignments", + "sha256": "2f438a9353d28b015ed4c9f6819711c8217d329bbfa7d855206aab6b215e2849" + } + ] +} diff --git a/tests/test_data/docs_score_release/protein_clusters/sequence.parquet b/tests/test_data/docs_score_release/protein_clusters/sequence.parquet new file mode 100644 index 00000000..fa0cfa0f Binary files /dev/null and b/tests/test_data/docs_score_release/protein_clusters/sequence.parquet differ diff --git a/tests/test_data/docs_score_release/protein_clusters/structure.parquet b/tests/test_data/docs_score_release/protein_clusters/structure.parquet new file mode 100644 index 00000000..b54ff468 Binary files /dev/null and b/tests/test_data/docs_score_release/protein_clusters/structure.parquet differ