From 420bed1fe66687410470b285ab2e450b23223365 Mon Sep 17 00:00:00 2001 From: Zio Gabber <78922322+Gabrymi93@users.noreply.github.com> Date: Fri, 16 Oct 2026 12:16:21 +0100 Subject: [PATCH 1/2] =?UTF-8?q?breaking:=20rimosso=20il=20server=20MCP=20?= =?UTF-8?q?=E2=80=94=20migrato=20a=20dataciviclab/dcl-mcp?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Il lato serving (mcp_server.py, entry point agent-context-mcp, extra [mcp]) e il query-side di navigation.py (search_index, explore_ref, lab_map_summary + helpers) sono migrati in dataciviclab/dcl-mcp insieme ai tool Discussions e al nuovo gcs_query (3 passate di audit newcomer). ACB resta il builder: CLI + CI che producono gli artifact sul branch context, consumati da remoto dal nuovo server. Rimosso anche: - tests/test_mcp_server.py (coperti da dcl-mcp/tests) - test_navigation.py: via i test query-side (coperti da dcl-mcp) - lab-connectors[mcp] → lab-connectors base (serve solo HttpClient) --- README.md | 56 +- pyproject.toml | 3 +- src/agent_context_builder/mcp_server.py | 501 ----------------- src/agent_context_builder/navigation.py | 704 +----------------------- tests/test_mcp_server.py | 681 ----------------------- tests/test_navigation.py | 193 ------- 6 files changed, 19 insertions(+), 2119 deletions(-) delete mode 100644 src/agent_context_builder/mcp_server.py delete mode 100644 tests/test_mcp_server.py diff --git a/README.md b/README.md index 730d7c3..0954945 100644 --- a/README.md +++ b/README.md @@ -53,52 +53,26 @@ python scripts/discover_registries.py --org dataciviclab --config dataciviclab.c python scripts/discover_registries.py --org dataciviclab --config dataciviclab.config.yml --apply ``` -## Tool MCP - -Esposti via `agent-context-mcp` (server MCP `dataciviclab-context`). - -| Tool | Quando usarlo | -|---|---| -| `session_bootstrap()` | Prima chiamata — orientamento rapido | -| `lab_map(domain=)` | Struttura del Lab: domains → repos → sources | -| `find(query, type=, domain=)` | Ricerca ranked su dataset/repo/domain/source/analysis | -| `explore(ref)` | Deep-dive di qualsiasi entità (slug repo, dataset, domain, source) | -| `workspace_triage(section=)` | Stato precisi: radar, prs, issues, registry, pipeline | -| `topic_index(resolve=)` | Legacy summary (resolve delega a explore) | -| `search(query)` | Legacy search (wraps find + GitHub issues) | -| `refresh_context()` | Trigger rebuild CI | - -### Esempi - -```python -# Orientamento -session_bootstrap() -# → markdown con radar, PR, issues, discussions - -# Struttura del Lab -lab_map() -# → {domains: [...], repos: [...], sources: {...}, totals: {...}} - -# Trovare un dataset -find("rifiuti", type="dataset") -# → results[{type, slug, name, parent_repo, parent_domain, score}] - -# Contesto di un repo -explore("open-ispra") -# → {type: repo, role, domain, datasets: [...], sources: [...]} - -# Stato radar -workspace_triage(section="radar") -# → {"green": 35, "yellow": 0, "red": 1, ...} -``` +## Tool MCP — migrato a dcl-mcp + +Il server MCP (`session_bootstrap`, `lab_map`, `find`, `explore`, +`workspace_triage`, `refresh_context`) non vive più qui: è migrato in +[`dataciviclab/dcl-mcp`](https://github.com/dataciviclab/dcl-mcp) insieme +ai tool Discussions e al nuovo `gcs_query`. ACB resta il **builder**: +CLI + CI che producono gli artifact sul branch `context`, che dcl-mcp +consuma da remoto. + +Per l'uso degli artefatti via MCP, vedi il README di dcl-mcp. Le env +`ACB_REPO` / `ACB_BRANCH` / `ACB_ARTIFACT_TTL_SECONDS` continuano a +valere lì (puntano a questa repo come sorgente). -### Configurazione +### Configurazione (dcl-mcp) ```json { "mcpServers": { - "dataciviclab-context": { - "command": "agent-context-mcp", + "dataciviclab": { + "command": "dcl-mcp", "env": { "GITHUB_TOKEN": "" } } } diff --git a/pyproject.toml b/pyproject.toml index 2d12748..846602e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ dependencies = [ "click>=8.0", "pydantic>=2.0", "pyyaml>=6.0", - "lab-connectors[mcp] @ git+https://github.com/dataciviclab/lab-connectors.git", + "lab-connectors @ git+https://github.com/dataciviclab/lab-connectors.git", ] [project.optional-dependencies] @@ -32,7 +32,6 @@ dev = [ [project.scripts] agent-context = "agent_context_builder.cli:main" -agent-context-mcp = "agent_context_builder.mcp_server:main" [project.urls] Repository = "https://github.com/dataciviclab/agent-context-builder" diff --git a/src/agent_context_builder/mcp_server.py b/src/agent_context_builder/mcp_server.py deleted file mode 100644 index b835d29..0000000 --- a/src/agent_context_builder/mcp_server.py +++ /dev/null @@ -1,501 +0,0 @@ -"""Public MCP server — serves DataCivicLab context artifacts from the context branch. - -Tools: - session_bootstrap — quick orientation (markdown) - workspace_triage — machine-readable state (compact + sections) - lab_map — navigation tree: domains → repos → sources - find — ranked cross-entity search (dataset/repo/domain/source/analysis) - explore — deep-dive any Lab entity by ref - topic_index — compact summary (resolve delegates to explore) - search — legacy cross-cutting search (wraps find + GitHub issues) - refresh_context — trigger CI rebuild (action) - -Configuration: - ACB_REPO GitHub repo (default: dataciviclab/agent-context-builder) - ACB_BRANCH Branch where artifacts are published (default: context) - GITHUB_TOKEN Required only for refresh_context - ACB_LOG_LEVEL Logging level (default: INFO) -""" - -from __future__ import annotations - -import json -import os -import time -from pathlib import Path -from typing import Any - -from lab_connectors.http import HttpClient -from lab_connectors.mcp import TtlCache, create_mcp_server, get_mcp_logger, guard_timed - -from .navigation import explore_ref, lab_map_summary, search_index - -_REPO = os.environ.get("ACB_REPO", "dataciviclab/agent-context-builder") -_BRANCH = os.environ.get("ACB_BRANCH", "context") -_RAW_BASE = f"https://raw.githubusercontent.com/{_REPO}/{_BRANCH}" -_API_BASE = f"https://api.github.com/repos/{_REPO}" - -_REFRESH_MIN_INTERVAL = 60 -_last_refresh_attempt: float | None = None - -# Artifact cache: agents often call lab_map/find/explore in sequence on the -# same topic_index.json. TTL keeps freshness after CI rebuilds (~6h cadence). -_ARTIFACT_TTL_SECONDS = int(os.environ.get("ACB_ARTIFACT_TTL_SECONDS", "120")) -_artifact_cache: TtlCache[str, str] = TtlCache(ttl_seconds=_ARTIFACT_TTL_SECONDS) - -_log = get_mcp_logger("agent-context-builder", level=os.environ.get("ACB_LOG_LEVEL", "INFO")) - -mcp = create_mcp_server( - name="dataciviclab-context", - instructions=( - "DataCivicLab context artifacts, generated from GitHub every 6 hours. " - "Start with session_bootstrap for orientation, then lab_map for structure, " - "find for discovery, explore for deep-dive, workspace_triage for state." - ), -) - - -# ── Env / HTTP helpers ───────────────────────────────────────────────────────── -_ENV_LOADED = False - - -def _parse_env_line(line: str) -> tuple[str, str] | None: - stripped = line.strip() - if not stripped or stripped.startswith("#") or "=" not in stripped: - return None - key, value = stripped.split("=", 1) - key = key.strip() - if not key: - return None - value = value.strip().strip('"').strip("'") - return key, value - - -def _candidate_env_paths() -> list[Path]: - explicit = os.environ.get("ACB_ENV_FILE", "").strip() - paths: list[Path] = [] - if explicit: - paths.append(Path(explicit).expanduser()) - starts = [Path.cwd(), Path(__file__).resolve()] - for start in starts: - current = start if start.is_dir() else start.parent - paths.extend(parent / ".env" for parent in [current] + list(current.parents)) - seen: set[Path] = set() - unique: list[Path] = [] - for path in paths: - resolved = path.resolve() - if resolved not in seen: - seen.add(resolved) - unique.append(resolved) - return unique - - -def _load_dotenv_if_present() -> bool: - global _ENV_LOADED - if _ENV_LOADED: - return True - _ENV_LOADED = True - loaded = False - for path in _candidate_env_paths(): - if not path.is_file(): - continue - try: - for line in path.read_text(encoding="utf-8").splitlines(): - parsed = _parse_env_line(line) - if parsed is None: - continue - key, value = parsed - if key not in os.environ or not os.environ[key]: - os.environ[key] = value - loaded = True - except OSError: - continue - return loaded - - -def _get_env(name: str) -> str | None: - _load_dotenv_if_present() - return os.environ.get(name) or None - - -_http: HttpClient | None = None - - -def _get_http() -> HttpClient: - global _http - if _http is None: - _http = HttpClient(timeout=15) - return _http - - -def _fetch(path: str, retries: int = 1, backoff: float = 1.0, use_cache: bool = True) -> str: - """Fetch a context-branch artifact, with optional TTL cache.""" - if use_cache: - cached = _artifact_cache.get(path) - if cached is not None: - return cached - - url = f"{_RAW_BASE}/{path}" - client = _get_http() - last_err: Exception | None = None - for attempt in range(retries + 1): - try: - result = client.get(url) - except Exception as exc: - last_err = exc - _log.warning("fetch", "exception", path=path, attempt=attempt, error=str(exc)) - if attempt < retries: - time.sleep(backoff * (2**attempt)) - continue - if result.is_ok and result.response is not None: - text = result.response.text - if use_cache: - _artifact_cache.set(path, text) - return text - last_err = Exception( - f"HTTP {result.response.status_code if result.response else 'N/A'}: {result.err}" - ) - _log.warning("fetch", "http_error", path=path, error=result.err) - if attempt < retries: - time.sleep(backoff * (2**attempt)) - raise last_err or Exception(f"Failed to fetch {path}") - - -# ── Tools ────────────────────────────────────────────────────────────────────── - - -@mcp.tool( - description="Quick orientation — compact markdown overview of Lab state.", - structured_output=True, -) -def session_bootstrap() -> dict[str, object]: - def _exec() -> dict[str, object]: - content = _fetch("session_bootstrap.md") - return {"content": content, "format": "markdown", "ok": True} - - return guard_timed(_exec, "session_bootstrap") - - -@mcp.tool( - description=( - "Machine-readable Lab state. Default: compact summary. " - "Use section= to get a specific part: radar, prs, issues, " - "discussions, registry, git, warnings." - ), - structured_output=True, -) -def workspace_triage(section: str | None = None) -> dict[str, object]: - def _exec() -> dict[str, object]: - raw = _fetch("workspace_triage.json") - data: dict = json.loads(raw) - - # Default: compact summary - if section is None: - radar = data.get("radar", {}) - registry = data.get("registry_summary", []) - total_ds = sum(r.get("datasets", 0) for r in registry if r.get("available")) - total_signals = sum(r.get("signals", 0) for r in registry if r.get("available")) - return { - "radar": { - "green": radar.get("green", 0), - "yellow": radar.get("yellow", 0), - "red": radar.get("red", 0), - "persistent_red": radar.get("persistent_red", 0), - }, - "datasets": total_ds, - "signals": total_signals, - "repos_active": sum(1 for r in registry if r.get("available")), - "prs": data.get("open_prs") or len(data.get("prs", [])), - "issues": data.get("open_issues") or len(data.get("issues", [])), - "discussions": data.get("open_discussions") or len(data.get("discussions", [])), - "warnings": data.get("warnings", []), - "ok": True, - } - - # Section filter - sections = { - "radar": lambda d: d.get("radar", {}), - "prs": lambda d: {"prs": d.get("prs", []), "count": len(d.get("prs", []))}, - "issues": lambda d: {"issues": d.get("issues", []), "count": len(d.get("issues", []))}, - "discussions": lambda d: { - "discussions": d.get("discussions", []), - "count": len(d.get("discussions", [])), - }, - "registry": lambda d: {"registry_summary": d.get("registry_summary", [])}, - "git": lambda d: {"git_state": d.get("git_state", {})}, - "warnings": lambda d: {"warnings": d.get("warnings", [])}, - "pipeline": lambda d: {"pipeline_state": d.get("pipeline_state", {})}, - "source_health": lambda d: {"source_health": d.get("source_health", {})}, - } - - extractor = sections.get(section) - if extractor is None: - return { - "ok": False, - "error": f"Sezione '{section}' non valida. Opzioni: {list(sections.keys())}", - } - - return {"content": extractor(data), "ok": True} - - return guard_timed(_exec, "workspace_triage") - - -def _load_topic_index() -> dict[str, Any]: - raw = _fetch("topic_index.json") - data = json.loads(raw) - return data if isinstance(data, dict) else {} - - -@mcp.tool( - description=( - "Lab navigation map — domains → repos → sources. " - "Use domain= to focus on one editorial theme." - ), - structured_output=True, -) -def lab_map(domain: str | None = None) -> dict[str, object]: - def _exec() -> dict[str, object]: - data = _load_topic_index() - summary = lab_map_summary(data, domain=domain) - return {"content": summary, "ok": True} - - return guard_timed(_exec, "lab_map") - - -@mcp.tool( - description=( - "Ranked search across datasets, repos, domains, sources, and analyses. " - "Filters: type= (dataset|repo|domain|source|analysis), domain=, limit=. " - "Returns type + parent pointers. Use explore(ref) for details." - ), - structured_output=True, -) -def find( - query: str, - type: str | None = None, - domain: str | None = None, - limit: int = 10, - include_github: bool = False, -) -> dict[str, object]: - def _exec() -> dict[str, object]: - data = _load_topic_index() - results = search_index(data, query, entity_type=type, domain=domain, limit=limit) - payload: dict[str, Any] = { - "query": query, - "total": len(results), - "results": results, - "ok": True, - } - if include_github: - token = _get_env("GITHUB_TOKEN") - payload["issues"] = _search_github_issues(query, token, limit) - payload["total"] += len(payload["issues"]) - return {"content": payload, "ok": True} - - return guard_timed(_exec, "find") - - -@mcp.tool( - description=( - "Deep-dive any Lab entity by ref: dataset slug, repo name, domain slug, " - "source id, or analysis slug. Returns card + children + routing hints." - ), - structured_output=True, -) -def explore(ref: str) -> dict[str, object]: - def _exec() -> dict[str, object]: - data = _load_topic_index() - return {"content": explore_ref(data, ref), "ok": True} - - return guard_timed(_exec, "explore") - - -@mcp.tool( - description=( - "Dataset/analysis summary (legacy). " - "Without resolve: returns compact summary (counts by source, stage). " - "With resolve: delegates to explore(). Prefer explore() and find()." - ), - structured_output=True, -) -def topic_index(resolve: str | None = None) -> dict[str, object]: - def _exec() -> dict[str, object]: - raw = _fetch("topic_index.json") - data: dict = json.loads(raw) - - # Default: compact summary - if not resolve: - by_source: dict[str, int] = {} - by_stage: dict[str, int] = {} - for source, ds_list in data.get("datasets", {}).items(): - for ds in ds_list: - by_source[source] = by_source.get(source, 0) + 1 - stage = ds.get("stage", "unknown") - by_stage[stage] = by_stage.get(stage, 0) + 1 - top_sources = sorted(by_source.items(), key=lambda x: -x[1])[:10] - return { - "total": sum(by_stage.values()), - "by_stage": by_stage, - "top_sources": {s: n for s, n in top_sources}, - "n_sources": len(by_source), - "analyses": len(data.get("analyses", [])), - "domains": len(data.get("domains", [])), - "ok": True, - } - - # Resolve always delegates to the navigation index (schema v7) - card = explore_ref(data, resolve) - return {"content": card, "ok": True} - - return guard_timed(_exec, "topic_index") - - -@mcp.tool( - description=( - "Legacy cross-cutting search across issues, PRs, datasets, and analyses. " - "Prefer find() for ranked navigation. Returns compact results." - ), - structured_output=True, -) -def search(query: str, limit: int = 10) -> dict[str, object]: - def _exec() -> dict[str, object]: - token = _get_env("GITHUB_TOKEN") - - # GitHub Issues + PRs - issues = _search_github_issues(query, token, limit) - - try: - topic_data = _load_topic_index() - except Exception: - topic_data = {} - - ranked = search_index(topic_data, query, limit=limit) if topic_data else [] - datasets = [ - { - "slug": r["slug"], - "name": r.get("name", ""), - "stage": r.get("stage", "published"), - } - for r in ranked - if r.get("type") == "dataset" - ] - analyses = [ - { - "slug": r["slug"], - "name": r.get("name", ""), - "datasets": r.get("datasets") or [], - } - for r in ranked - if r.get("type") == "analysis" - ] - - return { - "query": query, - "total": len(issues) + len(datasets) + len(analyses), - "issues": issues, - "datasets": datasets, - "analyses": analyses, - "ok": True, - } - - return guard_timed(_exec, "search") - - -@mcp.tool( - description=( - "Trigger a new context build on CI. " - "Requires GITHUB_TOKEN with workflow scope. " - "Artifacts updated within ~1 minute. Rate limit: 2 dispatches/hour." - ), - structured_output=True, -) -def refresh_context() -> dict[str, object]: - def _exec() -> dict: - global _last_refresh_attempt - - token = _get_env("GITHUB_TOKEN") - if not token: - return {"ok": False, "error": "GITHUB_TOKEN non impostato."} - - now = time.monotonic() - if _last_refresh_attempt is not None: - elapsed = now - _last_refresh_attempt - if elapsed < _REFRESH_MIN_INTERVAL: - wait = _REFRESH_MIN_INTERVAL - elapsed - return { - "ok": False, - "error": f"Rate limit. Aspetta ~{int(wait)}s.", - "retry_after": int(wait), - } - - _last_refresh_attempt = now - client = _get_http() - result = client.post( - f"{_API_BASE}/actions/workflows/build-context.yml/dispatches", - json={"ref": "main"}, - headers={"Authorization": f"token {token}", "Accept": "application/vnd.github+json"}, - ) - - if not result.is_ok or result.response is None: - return {"ok": False, "error": f"Errore di rete: {result.err}"} - - if result.response.status_code == 204: - _artifact_cache.clear() - return {"ok": True, "message": "Build triggerato. Aggiornamento entro ~1 minuto."} - elif result.response.status_code == 422: - return {"ok": False, "error": "Build rifiutato (422). Verifica workflow su main."} - else: - return {"ok": False, "error": f"Errore {result.response.status_code}"} - - return guard_timed(_exec, "refresh_context") - - -# ── Search helpers ───────────────────────────────────────────────────────────── - - -def _search_github_issues( - query: str, token: str | None, limit: int = 10 -) -> list[dict[str, object]]: - url = "https://api.github.com/search/issues" - headers = ( - {"Authorization": f"token {token}", "Accept": "application/vnd.github+json"} - if token - else {} - ) - params = { - "q": f"{query} org:dataciviclab", - "sort": "updated", - "order": "desc", - "per_page": min(limit, 50), - } - client = _get_http() - try: - result = client.get(url, params=params, headers=headers) - except Exception: - return [] - if not result.is_ok or result.response is None: - return [] - try: - items = result.response.json().get("items", []) - except Exception: - return [] - return [ - { - "repo": item.get("repository_url", "").replace("https://api.github.com/repos/", ""), - "number": item.get("number"), - "title": item.get("title", ""), - "state": item.get("state", ""), - "url": item.get("html_url", ""), - "type": "pr" if "pull_request" in item else "issue", - } - for item in items - ] - - -def main() -> None: - _log.info("main", "starting", repo=_REPO, branch=_BRANCH) - mcp.run() - - -if __name__ == "__main__": - main() diff --git a/src/agent_context_builder/navigation.py b/src/agent_context_builder/navigation.py index 4929ac4..ea13523 100644 --- a/src/agent_context_builder/navigation.py +++ b/src/agent_context_builder/navigation.py @@ -1,7 +1,9 @@ """Navigation graph builders for topic_index.json (schema v7). Pure functions: no I/O. Input is the flat dataset list + themes + repo info; -output are the navigation sections consumed by MCP tools and dashboards. +output are the navigation sections consumed by dashboards and by the +dcl-mcp server (query-side: search/lab_map/explore — migrated to +dataciviclab/dcl-mcp with the MCP server). Hierarchy: @@ -14,8 +16,6 @@ from __future__ import annotations -import re -import unicodedata from collections import Counter, defaultdict from typing import Any @@ -209,701 +209,3 @@ def build_reverse_indexes( "by_domain": {k: sorted(v) for k, v in sorted(by_domain.items())}, "by_slug": by_slug, } - - -def flatten_datasets(datasets_by_source: dict[str, list[dict[str, Any]]]) -> list[dict[str, Any]]: - """Flatten topic_index ``datasets`` dict into a list of dataset entries.""" - flat: list[dict[str, Any]] = [] - for _source, items in (datasets_by_source or {}).items(): - for item in items or []: - if isinstance(item, dict) and item.get("slug"): - flat.append(item) - return flat - - -# ── Query helpers (work on a built topic_index dict) ───────────────────────── - -# Function words that pollute natural-language queries ("bandi di gara", -# "appalti pubblici anac"). Matched after de-accenting, so "piu" covers "più". -# Curated minimal set — not a full Italian stopword list. -_STOPWORDS = frozenset( - { - "a", - "ad", - "agli", - "ai", - "al", - "alle", - "allo", - "anche", - "chi", - "ci", - "come", - "cui", - "da", - "dagli", - "dai", - "dal", - "dalle", - "del", - "dello", - "delle", - "dei", - "degli", - "di", - "dove", - "ed", - "e", - "fra", - "gli", - "ha", - "hanno", - "i", - "il", - "in", - "la", - "le", - "lo", - "ma", - "nei", - "negli", - "nel", - "nello", - "non", - "o", - "oppure", - "per", - "piu", - "quale", - "quali", - "quanti", - "quando", - "si", - "sono", - "sugli", - "sulle", - "sullo", - "sul", - "su", - "sui", - "tra", - "un", - "una", - "uno", - } -) - -# Field weights for the token tier (all tokens matched across the entity). -_FIELD_TOKEN_WEIGHTS = {"slug": 50, "name": 55, "text": 35, "tag": 28} -# Tier weights for exact whole-query and substring matches. -_EXACT_WEIGHTS = {"slug": 100, "name": 80, "text": 40, "tag": 30} -_SUBSTR_WEIGHTS = {"slug": 80, "name": 60, "text": 30, "tag": 25} - - -def _norm(text: str | None) -> str: - return (text or "").strip().lower() - - -def _deaccent(text: str) -> str: - # Fast path: most Lab fields (slugs, tags, semantic_types, ASCII names) - # have no accented characters, so skip normalize entirely. - if text.isascii(): - return text - return "".join(c for c in unicodedata.normalize("NFKD", text) if not unicodedata.combining(c)) - - -# Per-search cache: field value → word list. Passed down through -# search_index → _score_entity → _token_in_words so each distinct field -# string is tokenized once per query, not once per (token × entity). -_WordCache = dict[str, list[str]] - - -def _field_words(value: str | None, cache: _WordCache | None = None) -> list[str]: - """Lowercase alphanumeric words; underscores/punctuation are separators. - - Makes snake_case slugs word-matchable: "anac_bandi_gara" → - ["anac", "bandi", "gara"]. When a cache dict is supplied, repeated - values reuse the precomputed word list. - """ - if value is None: - return [] - if cache is not None and value in cache: - return cache[value] - v = _deaccent(_norm(value)) - words = [w for w in re.split(r"[^a-z0-9]+", v) if w] - if cache is not None: - cache[value] = words - return words - - -def _tokenize_query(query: str) -> list[str]: - """Query tokens minus Italian stopwords (order preserved, deduped).""" - tokens: list[str] = [] - for word in _field_words(query): - if len(word) < 2 or word in _STOPWORDS: - continue - if word not in tokens: - tokens.append(word) - return tokens - - -def _word_forms(word: str) -> set[str]: - """Word plus a light morphological variant (final-vowel strip). - - Covers common Italian singular/plural and gender pairs without a full - stemmer: gara≈gare, pubblico≈pubblici, appalto≈appalti. - """ - forms = {word} - if len(word) >= 4 and word[-1] in "aeiou": - forms.add(word[:-1]) - return forms - - -def _token_in_words(token: str, words: list[str], token_forms: set[str]) -> bool: - """Match a query token against a pre-tokenized field (word list). - - Uses a token_forms set computed once per query — the common case, since - the same token is tested against every field of every entity. - """ - for field_word in words: - if _word_forms(field_word) & token_forms: - return True - if len(token) >= 3 and field_word.startswith(token): - return True - return False - - -def query_matches(query: str, *texts: str | None) -> bool: - """True if every non-stopword query token matches at least one text. - - Token match is cross-text: words may be spread across the values. - Empty/stopword-only queries never match. - """ - tokens = _tokenize_query(query) - if not tokens: - return False - token_forms_list = [_word_forms(t) for t in tokens] - field_words_list = [_field_words(text) for text in texts] - return all( - any(_token_in_words(tok, words, tf) for words in field_words_list) - for tok, tf in zip(tokens, token_forms_list) - ) - - -def _score_entity( - query: str, - fields: list[tuple[str | None, str]], - word_cache: _WordCache | None = None, -) -> int: - """Score an entity against a query. fields: (value, weight_name). - - Tiers: - 1. exact whole-query match on a field - 2. whole-query substring on a field - 3. all non-stopword tokens match across the entity's fields (cross-field - AND) — natural-language queries like "appalti pubblici anac" - """ - q = _norm(query) - if not q: - return 0 - score = 0 - # Length guards: a 1-char query ("e") is a substring of almost everything; - # exact match needs at least 2 chars to be meaningful. - for value, kind in fields: - v = _norm(value) - if not v: - continue - if len(q) >= 2 and v == q: - score = max(score, _EXACT_WEIGHTS.get(kind, 20)) - elif len(q) >= 3 and q in v: - score = max(score, _SUBSTR_WEIGHTS.get(kind, 15)) - - tokens = _tokenize_query(query) - if not tokens: - return score - token_forms_list = [_word_forms(t) for t in tokens] - # Tokenize each field once per entity (not once per token×field). Empty - # fields are dropped so they are never re-scanned. The shared word_cache - # dedupes identical field values across entities in one search. - prepared = [] - for value, kind in fields: - if not value: - continue - words = _field_words(value, word_cache) - if words: - prepared.append((kind, words)) - matched_kinds: set[str] = set() - all_matched = True - for tok, tok_forms in zip(tokens, token_forms_list): - token_hit = False - for kind, words in prepared: - if _token_in_words(tok, words, tok_forms): - matched_kinds.add(kind) - token_hit = True - if not token_hit: - all_matched = False - break - if all_matched and matched_kinds: - best = max(_FIELD_TOKEN_WEIGHTS.get(kind, 18) for kind in matched_kinds) - score = max(score, best) - return score - - -def _columns_index_text(columns: list[dict[str, Any]] | None) -> tuple[str, str]: - """Join column names+descriptions and semantic types for search indexing. - - The topic_index already carries full column defs (~6k across the Lab); - indexing them lets find() answer "which dataset has a CIG column?" - without a separate pipeline. - """ - if not columns: - return "", "" - names = " ".join( - filter( - None, - (str(c.get("name") or "") for c in columns), - ) - ) - descs = " ".join( - filter( - None, - (str(c.get("description") or "") for c in columns), - ) - ) - sem = " ".join( - filter( - None, - (str(c.get("semantic_type") or "") for c in columns), - ) - ) - return f"{names} {descs}".strip(), sem - - -def search_index( - topic_index: dict[str, Any], - query: str, - entity_type: str | None = None, - domain: str | None = None, - limit: int = 10, -) -> list[dict[str, Any]]: - """Ranked cross-entity search over a topic_index dict. - - entity_type: dataset | repo | domain | source | analysis | None (all) - Returns items: {type, slug, name, parent_repo, parent_domain, score, stage?} - """ - results: list[dict[str, Any]] = [] - type_filter = entity_type.lower() if entity_type else None - # Shared word cache for the whole query: identical field values (e.g. - # source_id "anac" repeated across 9 datasets) tokenize once. - word_cache: _WordCache = {} - - by_slug = topic_index.get("by_slug") or {} - datasets_by_source = topic_index.get("datasets") or {} - - # Datasets — full entries carry tags/description (the main discovery - # fields); by_slug meta enriches with registry_source and domain. Both - # are built from the same flat list in render_topic_index, so by_slug - # never holds slugs missing from datasets. - dataset_entries: list[dict[str, Any]] = [] - seen_slugs: set[str] = set() - for ds in flatten_datasets(datasets_by_source): - slug = ds.get("slug") - if not slug or slug in seen_slugs: - continue - seen_slugs.add(slug) - meta = by_slug.get(slug) or {} - dataset_entries.append( - { - "slug": slug, - "name": ds.get("name") or slug, - "stage": ds.get("stage") or meta.get("stage"), - "source_id": ds.get("source_id") or meta.get("source_id"), - "registry_source": ds.get("registry_source") or meta.get("registry_source") or "", - "domain": meta.get("domain"), - "category": ds.get("category") or meta.get("category"), - "tags": ds.get("tags") or meta.get("tags"), - "description": ds.get("description") or meta.get("description"), - "columns": ds.get("columns") or meta.get("columns"), - } - ) - - if type_filter in (None, "dataset", "datasets"): - for ds in dataset_entries: - col_text, col_sem = _columns_index_text(ds.get("columns")) - score = _score_entity( - query, - [ - (ds.get("slug"), "slug"), - (ds.get("name"), "name"), - (ds.get("source_id"), "tag"), - (ds.get("category"), "tag"), - (ds.get("registry_source"), "tag"), - (ds.get("description"), "text"), - (" ".join(ds.get("tags") or []), "tag"), - # Column schema already lives in the artifact — names + - # descriptions as discovery text, semantic_type as a - # structured tag ("cig_code" → query "cig"). - (col_text, "text"), - (col_sem, "tag"), - ], - word_cache, - ) - if score <= 0: - continue - d_domain = ds.get("domain") - if domain and d_domain != domain: - continue - results.append( - { - "type": "dataset", - "slug": ds.get("slug"), - "name": ds.get("name"), - "parent_repo": ds.get("registry_source") or "", - "parent_domain": d_domain, - "stage": ds.get("stage"), - "score": score, - } - ) - - if type_filter in (None, "repo", "repos"): - for repo, card in (topic_index.get("repos") or {}).items(): - if not isinstance(card, dict): - continue - score = _score_entity( - query, - [ - (repo, "slug"), - (card.get("description"), "text"), - (card.get("role"), "tag"), - (card.get("domain"), "tag"), - ], - word_cache, - ) - if score <= 0: - continue - if domain and card.get("domain") != domain: - continue - results.append( - { - "type": "repo", - "slug": repo, - "name": repo, - "parent_repo": repo, - "parent_domain": card.get("domain"), - "role": card.get("role"), - "n_datasets": card.get("n_datasets"), - "score": score, - } - ) - - if type_filter in (None, "domain", "domains", "theme", "themes"): - domains = topic_index.get("domains") or topic_index.get("explorer_themes") or [] - for dom in domains: - if not isinstance(dom, dict): - continue - score = _score_entity( - query, - [ - (dom.get("slug"), "slug"), - (dom.get("name"), "name"), - (dom.get("description"), "text"), - ], - word_cache, - ) - if score <= 0: - continue - if domain and dom.get("slug") != domain: - continue - results.append( - { - "type": "domain", - "slug": dom.get("slug"), - "name": dom.get("name"), - "parent_repo": "", - "parent_domain": dom.get("slug"), - "dataset_count": dom.get("dataset_count") or len(dom.get("datasets") or []), - "score": score, - } - ) - - if type_filter in (None, "analysis", "analyses"): - for a in topic_index.get("analyses") or []: - score = _score_entity( - query, - [ - (a.get("slug"), "slug"), - (a.get("name"), "name"), - (a.get("description"), "text"), - (a.get("source"), "tag"), - ], - word_cache, - ) - if score <= 0: - continue - results.append( - { - "type": "analysis", - "slug": a.get("slug"), - "name": a.get("name"), - "parent_repo": "", - "parent_domain": None, - "datasets": a.get("datasets") or [], - "score": score, - } - ) - - if type_filter in (None, "source", "sources"): - source_counts: dict[str, int] = {} - for ds in dataset_entries: - sid = ds.get("source_id") or "" - if sid: - source_counts[sid] = source_counts.get(sid, 0) + 1 - for sid, count in source_counts.items(): - score = _score_entity(query, [(sid, "slug")], word_cache) - if score <= 0: - continue - results.append( - { - "type": "source", - "slug": sid, - "name": sid, - "parent_repo": "", - "parent_domain": None, - "n_datasets": count, - "score": score, - } - ) - - results.sort(key=lambda r: (-r.get("score", 0), r.get("slug") or "")) - return results[: max(limit, 1)] - - -def lab_map_summary(topic_index: dict[str, Any], domain: str | None = None) -> dict[str, Any]: - """Compact navigation tree: domains → repos → sources.""" - repos = topic_index.get("repos") or {} - domains = topic_index.get("domains") or [] - by_domain = topic_index.get("by_domain") or {} - by_repo = topic_index.get("by_repo") or {} - by_slug = topic_index.get("by_slug") or {} - - total_datasets = ( - sum(len(v) for v in by_repo.values()) if by_repo else sum(len(v) for v in by_slug.values()) - ) - published = sum(1 for m in by_slug.values() if m.get("stage") == "published") - - domain_nodes: list[dict[str, Any]] = [] - selected = [d for d in domains if not domain or d.get("slug") == domain] - for d in selected: - domain_nodes.append( - { - "slug": d.get("slug"), - "name": d.get("name"), - "description": d.get("description", ""), - "dataset_count": d.get("dataset_count") or len(d.get("datasets") or []), - "repos": d.get("repos") or [], - } - ) - - repo_nodes: list[dict[str, Any]] = [] - for name, card in sorted(repos.items()): - if not isinstance(card, dict): - continue - if domain and card.get("domain") != domain: - continue - repo_nodes.append( - { - "repo": name, - "role": card.get("role"), - "domain": card.get("domain"), - "n_datasets": card.get("n_datasets") or len(by_repo.get(name) or []), - "n_published": card.get("n_published", 0), - "sources": card.get("sources") or [], - } - ) - - sources: dict[str, int] = {} - if domain: - ds_slugs = by_domain.get(domain) or [] - for slug in ds_slugs: - sid = (by_slug.get(slug) or {}).get("source_id") or "" - if sid: - sources[sid] = sources.get(sid, 0) + 1 - else: - for meta in by_slug.values(): - sid = meta.get("source_id") or "" - if sid: - sources[sid] = sources.get(sid, 0) + 1 - - return { - "domains": domain_nodes, - "repos": repo_nodes, - "sources": dict(sorted(sources.items(), key=lambda x: (-x[1], x[0]))), - "totals": { - "repos": len(repo_nodes), - "domains": len(domain_nodes), - "datasets": total_datasets, - "published": published, - "analyses": len(topic_index.get("analyses") or []), - }, - "ok": True, - } - - -def explore_ref(topic_index: dict[str, Any], ref: str) -> dict[str, Any]: - """Resolve any Lab entity (repo, dataset, domain, source, analysis).""" - ref_l = (ref or "").strip().lower() - result: dict[str, Any] = {"ref": ref, "found": False} - - by_slug = topic_index.get("by_slug") or {} - datasets_by_source = topic_index.get("datasets") or {} - repos = topic_index.get("repos") or {} - domains = topic_index.get("domains") or topic_index.get("explorer_themes") or [] - analyses = topic_index.get("analyses") or [] - analyses_by_dataset = topic_index.get("analyses_by_dataset") or {} - by_repo = topic_index.get("by_repo") or {} - by_domain = topic_index.get("by_domain") or {} - - def _dataset_card(slug: str) -> dict[str, Any] | None: - # by_slug is always built from the same flat list as datasets - # (render_topic_index) — no scan fallback needed. - meta = by_slug.get(slug) - if meta is None: - return None - return { - "type": "dataset", - "slug": slug, - "name": meta.get("name") or slug, - "stage": meta.get("stage"), - "source_id": meta.get("source_id"), - "registry_source": meta.get("registry_source") or "", - "domain": meta.get("domain"), - "category": meta.get("category"), - "analyses": analyses_by_dataset.get(slug) or [], - } - - # Dataset - ds_card = _dataset_card(ref) - if ds_card is not None: - result.update(ds_card) - result["found"] = True - repo = ds_card.get("registry_source") or "" - if repo and repo in repos: - card = repos[repo] - result["repo"] = { - "name": repo, - "role": card.get("role") if isinstance(card, dict) else None, - "domain": card.get("domain") if isinstance(card, dict) else None, - "url": card.get("url") if isinstance(card, dict) else "", - } - return result - - # Repo - repo_key = ref if ref in repos else None - if repo_key is None: - for name in repos: - if name.lower() == ref_l: - repo_key = name - break - if repo_key is not None: - card = repos[repo_key] if isinstance(repos[repo_key], dict) else {} - ds_slugs = by_repo.get(repo_key) or [] - result.update( - { - "type": "repo", - "slug": repo_key, - "name": repo_key, - "role": card.get("role"), - "domain": card.get("domain"), - "description": card.get("description", ""), - "url": card.get("url", ""), - "n_datasets": card.get("n_datasets") or len(ds_slugs), - "n_published": card.get("n_published", 0), - "sources": card.get("sources") or [], - "datasets": ds_slugs[:50], - "found": True, - } - ) - return result - - # Domain - domain_key = None - for d in domains: - slug = d.get("slug") if isinstance(d, dict) else None - if slug and slug.lower() == ref_l: - domain_key = d - break - if domain_key is not None: - slug = domain_key.get("slug") - ds_slugs = domain_key.get("datasets") or by_domain.get(slug) or [] - result.update( - { - "type": "domain", - "slug": slug, - "name": domain_key.get("name"), - "description": domain_key.get("description", ""), - "repos": domain_key.get("repos") or [], - "datasets": ds_slugs[:50], - "dataset_count": domain_key.get("dataset_count") or len(ds_slugs), - "found": True, - } - ) - return result - - # Source (source_id) - source_ds: list[str] = [] - seen_source_ds: set[str] = set() - for source_key, items in datasets_by_source.items(): - if source_key.lower() == ref_l: - for item in items or []: - slug = item.get("slug") - if slug and slug not in seen_source_ds: - seen_source_ds.add(slug) - source_ds.append(slug) - result.update( - { - "type": "source", - "slug": source_key, - "name": source_key, - "n_datasets": len(source_ds), - "datasets": sorted(source_ds)[:50], - "found": True, - } - ) - return result - for slug, meta in by_slug.items(): - sid = meta.get("source_id") or "" - if sid and sid.lower() == ref_l and slug not in seen_source_ds: - seen_source_ds.add(slug) - source_ds.append(slug) - if source_ds: - result.update( - { - "type": "source", - "slug": ref, - "name": ref, - "n_datasets": len(source_ds), - "datasets": sorted(source_ds)[:50], - "found": True, - } - ) - return result - - # Analysis - for a in analyses: - if (a.get("slug") or "").lower() == ref_l: - result.update( - { - "type": "analysis", - "slug": a.get("slug"), - "name": a.get("name"), - "status": a.get("status"), - "description": a.get("description", ""), - "source": a.get("source", ""), - "period": a.get("period"), - "datasets": a.get("datasets") or [], - "found": True, - } - ) - return result - - return result diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py deleted file mode 100644 index a3b18d7..0000000 --- a/tests/test_mcp_server.py +++ /dev/null @@ -1,681 +0,0 @@ -"""Tests for MCP server resources — uses FakeHttpClient for HTTP boundary.""" - -from __future__ import annotations - -import json -from unittest.mock import MagicMock, patch - -import pytest -from lab_connectors.http import HttpResult -from lab_connectors.testing import FakeHttpClient, fake_response - -import agent_context_builder.mcp_server as mcp_server - -# --------------------------------------------------------------------------- -# _fetch-based tools — inject FakeHttpClient via patching HttpClient class -# --------------------------------------------------------------------------- - - -def _patch_fetch(fake: FakeHttpClient, path: str, text: str = "", status: int = 200) -> None: - """Register a response for one of the context-branch artifact URLs.""" - url = f"https://raw.githubusercontent.com/dataciviclab/agent-context-builder/context/{path}" - resp = fake_response(status, text=text) - err = Exception(f"HTTP {status}") if status >= 400 else None - fake.responses[url] = HttpResult(response=resp, err=err) - # Drop any cached body so tests always see the newly registered payload. - mcp_server._artifact_cache.invalidate(path) - - -@pytest.mark.contract -def test_session_bootstrap_resource(): - """session_bootstrap fetches session_bootstrap.md from context branch.""" - fake = FakeHttpClient() - _patch_fetch(fake, "session_bootstrap.md", text="# Session Bootstrap\n") - - mcp_server._http = fake - result = mcp_server.session_bootstrap() - mcp_server._http = None - - assert result["format"] == "markdown" - assert "Session Bootstrap" in result["content"] - - -@pytest.mark.contract -def test_workspace_triage_resource(): - """workspace_triage fetches workspace_triage.json from context branch.""" - fake = FakeHttpClient() - _patch_fetch(fake, "workspace_triage.json", text='{"open_prs": 2}') - - mcp_server._http = fake - result = mcp_server.workspace_triage() - mcp_server._http = None - - assert result["ok"] is True - assert result["ok"] is True - - -@pytest.mark.contract -def test_topic_index_resource(): - """topic_index fetches topic_index.json from context branch.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text='{"topics": {}}') - - mcp_server._http = fake - result = mcp_server.topic_index() - mcp_server._http = None - - assert result["ok"] is True - assert "total" in result - - -@pytest.mark.contract -def test_session_bootstrap_http_error(): - """session_bootstrap returns error dict on HTTP failure instead of raising.""" - fake = FakeHttpClient() - _patch_fetch(fake, "session_bootstrap.md", status=403) - - mcp_server._http = fake - result = mcp_server.session_bootstrap() - mcp_server._http = None - - assert "error" in result - - -@pytest.mark.contract -def test_workspace_triage_http_error(): - """workspace_triage returns error dict on HTTP failure instead of raising.""" - fake = FakeHttpClient() - _patch_fetch(fake, "workspace_triage.json", status=404) - - mcp_server._http = fake - result = mcp_server.workspace_triage() - mcp_server._http = None - - assert "error" in result - - -@pytest.mark.contract -def test_topic_index_http_error(): - """topic_index returns error dict on HTTP failure instead of raising.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", status=500) - - mcp_server._http = fake - result = mcp_server.topic_index() - mcp_server._http = None - - assert "error" in result - - -# --------------------------------------------------------------------------- -# topic_index(resolve=...) — delegates to explore (schema v7 only) -# --------------------------------------------------------------------------- - - -# --------------------------------------------------------------------------- -# refresh_context — via HttpClient.post -# --------------------------------------------------------------------------- - - -def _reset_refresh_state(monkeypatch) -> None: - """Reset global state for refresh_context tests. - - Sets ``_ENV_LOADED = False`` so that ``_load_dotenv_if_present()`` - actually loads from .env files during the test. - """ - monkeypatch.setattr(mcp_server, "_last_refresh_attempt", None) - monkeypatch.setattr(mcp_server, "_ENV_LOADED", False) - - -def _mock_http_result(status: int = 204): - """Build an HttpResult-like object for mocking HttpClient.post.""" - resp = MagicMock() - resp.status_code = status - resp.text = "mock response" - from lab_connectors.http import HttpResult - - return HttpResult(response=resp, err=None, ssl_fallback_used=None) - - -@pytest.mark.adapter -def test_refresh_context_no_token(monkeypatch): - """refresh_context returns error message when GITHUB_TOKEN not set.""" - monkeypatch.delenv("GITHUB_TOKEN", raising=False) - monkeypatch.delenv("ACB_ENV_FILE", raising=False) - monkeypatch.setattr(mcp_server, "_ENV_LOADED", True) - result = mcp_server.refresh_context() - - assert result["ok"] is False - assert "GITHUB_TOKEN" in result["error"] - - -@pytest.mark.adapter -def test_refresh_context_loads_token_from_env_file(monkeypatch, tmp_path): - """refresh_context can use a local .env when the host does not export env vars.""" - env_file = tmp_path / ".env" - env_file.write_text("GITHUB_TOKEN=file-token\n", encoding="utf-8") - - monkeypatch.delenv("GITHUB_TOKEN", raising=False) - monkeypatch.setenv("ACB_ENV_FILE", str(env_file)) - _reset_refresh_state(monkeypatch) - - with patch("agent_context_builder.mcp_server.HttpClient.post") as mock_post: - mock_post.return_value = _mock_http_result(204) - result = mcp_server.refresh_context() - - assert result["ok"] is True - assert mock_post.call_args.kwargs["headers"]["Authorization"] == "token file-token" - - -@pytest.mark.adapter -def test_refresh_context_loads_token_when_env_is_empty(monkeypatch, tmp_path): - """A blank inherited value should not block the local .env fallback.""" - env_file = tmp_path / ".env" - env_file.write_text("GITHUB_TOKEN=file-token\n", encoding="utf-8") - - monkeypatch.setenv("GITHUB_TOKEN", "") - monkeypatch.setenv("ACB_ENV_FILE", str(env_file)) - _reset_refresh_state(monkeypatch) - - with patch("agent_context_builder.mcp_server.HttpClient.post") as mock_post: - mock_post.return_value = _mock_http_result(204) - result = mcp_server.refresh_context() - - assert result["ok"] is True - assert mock_post.call_args.kwargs["headers"]["Authorization"] == "token file-token" - - -@pytest.mark.adapter -def test_refresh_context_continues_after_partial_env(monkeypatch, tmp_path): - """A partial explicit .env should not prevent later candidates from filling tokens.""" - explicit_env = tmp_path / "partial.env" - explicit_env.write_text("ACB_BRANCH=context\n", encoding="utf-8") - workspace_env = tmp_path / ".env" - workspace_env.write_text("GITHUB_TOKEN=file-token\n", encoding="utf-8") - - monkeypatch.chdir(tmp_path) - monkeypatch.delenv("GITHUB_TOKEN", raising=False) - monkeypatch.setenv("ACB_ENV_FILE", str(explicit_env)) - _reset_refresh_state(monkeypatch) - - with patch("agent_context_builder.mcp_server.HttpClient.post") as mock_post: - mock_post.return_value = _mock_http_result(204) - result = mcp_server.refresh_context() - - assert result["ok"] is True - assert mock_post.call_args.kwargs["headers"]["Authorization"] == "token file-token" - - -@pytest.mark.adapter -def test_refresh_context_success(monkeypatch): - """refresh_context triggers workflow dispatch and reports success.""" - monkeypatch.setenv("GITHUB_TOKEN", "fake-token") - _reset_refresh_state(monkeypatch) - - with patch("agent_context_builder.mcp_server.HttpClient.post") as mock_post: - mock_post.return_value = _mock_http_result(204) - result = mcp_server.refresh_context() - - assert result["ok"] is True - - -@pytest.mark.adapter -def test_refresh_context_api_error(monkeypatch): - """refresh_context reports API errors without raising.""" - monkeypatch.setenv("GITHUB_TOKEN", "fake-token") - _reset_refresh_state(monkeypatch) - - with patch("agent_context_builder.mcp_server.HttpClient.post") as mock_post: - mock_post.return_value = _mock_http_result(403) - result = mcp_server.refresh_context() - - assert result["ok"] is False - assert "error" in result - - -# ── search ───────────────────────────────────────────────────────────────── - - -@pytest.mark.adapter -def test_search_github_issues_api_error(): - """_search_github_issues returns empty list on API error instead of raising.""" - fake = FakeHttpClient() - search_url = "https://api.github.com/search/issues" - fake.responses[search_url] = HttpResult( - response=None, - err=RuntimeError("403 rate limit exceeded"), - ) - mcp_server._http = fake - result = mcp_server._search_github_issues("test", "fake-token", limit=5) - mcp_server._http = None - - assert result == [] - - -@pytest.mark.adapter -def test_search_github_issues_malformed_json(): - """_search_github_issues handles malformed JSON response.""" - fake = FakeHttpClient() - url = "https://api.github.com/search/issues" - fake.responses[url] = HttpResult( - response=fake_response(200, text="not json"), - err=None, - ) - mcp_server._http = fake - result = mcp_server._search_github_issues("test", "fake-token", limit=5) - mcp_server._http = None - - assert result == [] - - -@pytest.mark.adapter -def test_search_github_issues_success(): - """_search_github_issues parses valid search results correctly.""" - fake = FakeHttpClient() - url = "https://api.github.com/search/issues" - fake.responses[url] = HttpResult( - response=fake_response( - 200, - json_data={ - "total_count": 1, - "items": [ - { - "number": 42, - "title": "Test issue about rifiuti", - "state": "open", - "html_url": "https://github.com/dataciviclab/test-repo/issues/42", - "repository_url": "https://api.github.com/repos/dataciviclab/test-repo", - "updated_at": "2026-07-14T10:00:00Z", - }, - ], - }, - ), - err=None, - ) - mcp_server._http = fake - result = mcp_server._search_github_issues("rifiuti", "fake-token", limit=5) - mcp_server._http = None - - assert len(result) == 1 - assert result[0]["number"] == 42 - assert result[0]["repo"] == "dataciviclab/test-repo" - assert result[0]["type"] == "issue" - assert result[0]["state"] == "open" - - -@pytest.mark.adapter -def test_search_github_issues_detects_pr(): - """_search_github_issues marks items with pull_request field as type 'pr'.""" - fake = FakeHttpClient() - url = "https://api.github.com/search/issues" - fake.responses[url] = HttpResult( - response=fake_response( - 200, - json_data={ - "total_count": 1, - "items": [ - { - "number": 99, - "title": "feat: add new dataset", - "state": "open", - "html_url": "https://github.com/dataciviclab/repo/pull/99", - "repository_url": "https://api.github.com/repos/dataciviclab/repo", - "pull_request": {"url": "..."}, - "updated_at": "2026-07-14T10:00:00Z", - }, - ], - }, - ), - err=None, - ) - mcp_server._http = fake - result = mcp_server._search_github_issues("dataset", "fake-token", limit=5) - mcp_server._http = None - - assert len(result) == 1 - assert result[0]["type"] == "pr" - - -@pytest.mark.contract -def test_search_tool_no_token(monkeypatch): - """search() works without token (just topic_index search).""" - monkeypatch.delenv("GITHUB_TOKEN", raising=False) - monkeypatch.setenv("ACB_ENV_FILE", "") - monkeypatch.setattr(mcp_server, "_ENV_LOADED", True) - - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text='{"datasets": {}, "analyses": []}') - - mcp_server._http = fake - result = mcp_server.search(query="test", limit=5) - mcp_server._http = None - - assert result["ok"] is True - assert result["query"] == "test" - assert "issues" in result - assert "datasets" in result - assert "analyses" in result - - -# --------------------------------------------------------------------------- -# Navigation tools: lab_map / find / explore (schema v7) -# --------------------------------------------------------------------------- - -_SAMPLE_V7_INDEX = json.dumps( - { - "schema_version": 7, - "repos": { - "rifiuti-urbani": { - "description": "Rifiuti urbani ISPRA", - "url": "https://github.com/dataciviclab/rifiuti-urbani", - "role": "dati", - "domain": "territorio-ambiente", - "n_datasets": 1, - "n_published": 1, - "sources": ["ispra"], - }, - "toolkit": { - "description": "Pipeline engine", - "url": "https://github.com/dataciviclab/toolkit", - "role": "infra", - "domain": None, - "n_datasets": 0, - "n_published": 0, - "sources": [], - }, - }, - "datasets": { - "ispra": [ - { - "slug": "ispra_ru_base", - "name": "Rifiuti Urbani", - "stage": "published", - "source_id": "ispra", - "registry_source": "rifiuti-urbani", - "category": "ambiente", - } - ] - }, - "domains": [ - { - "slug": "territorio-ambiente", - "name": "Territorio e ambiente", - "description": "Ambiente", - "repos": ["rifiuti-urbani"], - "datasets": ["ispra_ru_base"], - "dataset_count": 1, - } - ], - "explorer_themes": [ - { - "slug": "territorio-ambiente", - "name": "Territorio e ambiente", - "datasets": ["ispra_ru_base"], - } - ], - "analyses": [ - { - "slug": "rifiuti-urbani", - "name": "Rifiuti urbani", - "datasets": ["ispra_ru_base"], - "status": "published", - } - ], - "analyses_by_dataset": {"ispra_ru_base": ["rifiuti-urbani"]}, - "by_repo": {"rifiuti-urbani": ["ispra_ru_base"]}, - "by_domain": {"territorio-ambiente": ["ispra_ru_base"]}, - "by_slug": { - "ispra_ru_base": { - "name": "Rifiuti Urbani", - "stage": "published", - "source_id": "ispra", - "registry_source": "rifiuti-urbani", - "domain": "territorio-ambiente", - "category": "ambiente", - } - }, - } -) - - -@pytest.mark.contract -def test_topic_index_resolve_delegates_to_explore(): - """resolve= uses explore_ref on the v7 index (no legacy sub-graph path).""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.topic_index(resolve="ispra_ru_base") - mcp_server._http = None - - data = result["content"] - assert data["found"] is True - assert data["type"] == "dataset" - assert data["slug"] == "ispra_ru_base" - - -@pytest.mark.contract -def test_topic_index_resolve_not_found(): - """resolve returns found=False via explore when ref does not exist.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.topic_index(resolve="nonexistent") - mcp_server._http = None - - assert result["content"]["found"] is False - - -@pytest.mark.contract -def test_lab_map_resource(): - """lab_map returns navigation tree from topic_index.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.lab_map() - mcp_server._http = None - - content = result["content"] - assert content["ok"] is True - assert content["totals"]["datasets"] == 1 - assert content["totals"]["published"] == 1 - assert any(d["slug"] == "territorio-ambiente" for d in content["domains"]) - assert any(r["repo"] == "rifiuti-urbani" for r in content["repos"]) - assert content["sources"]["ispra"] == 1 - - -@pytest.mark.contract -def test_lab_map_domain_filter(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.lab_map(domain="territorio-ambiente") - mcp_server._http = None - - content = result["content"] - assert len(content["domains"]) == 1 - assert content["domains"][0]["slug"] == "territorio-ambiente" - - -@pytest.mark.contract -def test_find_ranks_dataset_with_parents(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.find(query="ispra_ru_base", limit=5) - mcp_server._http = None - - content = result["content"] - assert content["ok"] is True - assert content["results"] - top = content["results"][0] - assert top["type"] == "dataset" - assert top["slug"] == "ispra_ru_base" - assert top["parent_repo"] == "rifiuti-urbani" - assert top["parent_domain"] == "territorio-ambiente" - - -@pytest.mark.contract -def test_find_type_filter_repo(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.find(query="toolkit", type="repo") - mcp_server._http = None - - content = result["content"] - assert content["results"] - assert content["results"][0]["type"] == "repo" - assert content["results"][0]["role"] == "infra" - - -@pytest.mark.contract -def test_find_include_github_off_by_default(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.find(query="rifiuti") - mcp_server._http = None - assert "issues" not in result["content"] - - -@pytest.mark.contract -def test_explore_dataset(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.explore(ref="ispra_ru_base") - mcp_server._http = None - - card = result["content"] - assert card["found"] is True - assert card["type"] == "dataset" - assert card["registry_source"] == "rifiuti-urbani" - assert card["analyses"] == ["rifiuti-urbani"] - assert card["repo"]["role"] == "dati" - - -@pytest.mark.contract -def test_explore_repo(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.explore(ref="rifiuti-urbani") - mcp_server._http = None - - card = result["content"] - assert card["found"] is True - assert card["type"] == "repo" - assert card["datasets"] == ["ispra_ru_base"] - assert card["n_published"] == 1 - - -@pytest.mark.contract -def test_explore_domain(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.explore(ref="territorio-ambiente") - mcp_server._http = None - - card = result["content"] - assert card["found"] is True - assert card["type"] == "domain" - assert "ispra_ru_base" in card["datasets"] - - -@pytest.mark.contract -def test_explore_not_found(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.explore(ref="does-not-exist") - mcp_server._http = None - assert result["content"]["found"] is False - - -@pytest.mark.contract -def test_topic_index_v7_summary_includes_domains(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.topic_index() - mcp_server._http = None - assert result["domains"] == 1 - assert result["total"] == 1 - - -@pytest.mark.contract -def test_search_uses_navigation_index_when_v7(): - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - result = mcp_server.search(query="rifiuti") - mcp_server._http = None - content = result - assert content["ok"] is True - # search returns top-level keys (not nested content) - assert "datasets" in content - slugs = [d["slug"] for d in content["datasets"]] - assert "ispra_ru_base" in slugs - - -# --------------------------------------------------------------------------- -# Artifact TTL cache -# --------------------------------------------------------------------------- - - -@pytest.mark.policy -def test_artifact_cache_reuses_topic_index_fetch(): - """Second tool call within TTL must not re-fetch topic_index.json.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - try: - mcp_server.lab_map() - mcp_server.lab_map() - mcp_server.find(query="ispra_ru_base", limit=1) - mcp_server.explore(ref="toolkit") - topic_fetches = [ - c - for c in fake.requests - if getattr(c, "url", "") and "topic_index.json" in str(getattr(c, "url", "")) - ] - # FakeHttpClient request log shape may vary; fall back to cache stats. - stats = mcp_server._artifact_cache.stats - assert stats.entries >= 1 - # If request objects expose url, enforce single fetch; otherwise cache hit path is enough. - if topic_fetches: - assert len(topic_fetches) == 1 - result = mcp_server.topic_index() - assert result["ok"] is True - assert result["domains"] == 1 - finally: - mcp_server._http = None - mcp_server._artifact_cache.clear() - - -@pytest.mark.policy -def test_artifact_cache_invalidate_on_refresh(monkeypatch): - """refresh_context success clears the artifact cache.""" - fake = FakeHttpClient() - _patch_fetch(fake, "topic_index.json", text=_SAMPLE_V7_INDEX) - mcp_server._http = fake - mcp_server.lab_map() - assert mcp_server._artifact_cache.stats.entries >= 1 - - monkeypatch.setenv("GITHUB_TOKEN", "fake-token") - monkeypatch.setattr(mcp_server, "_ENV_LOADED", True) - monkeypatch.setattr(mcp_server, "_last_refresh_attempt", None) - - dispatch_url = ( - "https://api.github.com/repos/dataciviclab/agent-context-builder" - "/actions/workflows/build-context.yml/dispatches" - ) - fake.responses[dispatch_url] = HttpResult( - response=fake_response(204, text=""), - err=None, - ) - result = mcp_server.refresh_context() - mcp_server._http = None - assert result["ok"] is True - assert mcp_server._artifact_cache.stats.entries == 0 diff --git a/tests/test_navigation.py b/tests/test_navigation.py index 3011f99..314dd87 100644 --- a/tests/test_navigation.py +++ b/tests/test_navigation.py @@ -7,15 +7,11 @@ import pytest from agent_context_builder.navigation import ( - _tokenize_query, build_domains, build_explorer_themes, build_repo_cards, build_reverse_indexes, category_to_domain, - explore_ref, - lab_map_summary, - search_index, ) from agent_context_builder.sources.explorer import ExplorerTheme @@ -153,192 +149,3 @@ def test_reverse_indexes(): assert idx["by_domain"]["finanza-pubblica"] == ["bdap_entrate_stato"] assert idx["by_slug"]["ispra_ru_base"]["registry_source"] == "rifiuti-urbani" assert idx["by_slug"]["ispra_ru_base"]["domain"] == "territorio-ambiente" - - -def test_search_index_finds_dataset_by_slug(): - results = search_index(_index(), "ispra_ru_base") - assert results - top = results[0] - assert top["type"] == "dataset" - assert top["slug"] == "ispra_ru_base" - assert top["parent_repo"] == "rifiuti-urbani" - assert top["parent_domain"] == "territorio-ambiente" - - -def test_search_index_type_filter_repo(): - results = search_index(_index(), "toolkit", entity_type="repo") - assert results - assert results[0]["type"] == "repo" - assert results[0]["slug"] == "toolkit" - assert results[0]["role"] == "infra" - - -def test_search_index_domain_filter(): - results = search_index(_index(), "rifiuti", domain="territorio-ambiente") - assert any(r["slug"] == "ispra_ru_base" for r in results) - results2 = search_index(_index(), "entrate", domain="territorio-ambiente") - assert not any(r["slug"] == "bdap_entrate_stato" for r in results2) - - -def test_search_index_source_and_analysis(): - src = search_index(_index(), "ispra", entity_type="source") - assert src and src[0]["type"] == "source" - ana = search_index(_index(), "rifiuti", entity_type="analysis") - assert ana and ana[0]["type"] == "analysis" - - -def test_lab_map_summary_tree(): - summary = lab_map_summary(_index()) - assert summary["totals"]["datasets"] == 3 - assert summary["totals"]["published"] == 2 - assert any(d["slug"] == "territorio-ambiente" for d in summary["domains"]) - assert any(r["repo"] == "rifiuti-urbani" for r in summary["repos"]) - assert summary["sources"]["ispra"] == 1 - - -def test_lab_map_summary_domain_filter(): - summary = lab_map_summary(_index(), domain="finanza-pubblica") - assert summary["totals"]["datasets"] == 3 # totals remain global - assert len(summary["domains"]) == 1 - assert summary["repos"] == [] or all( - r["domain"] == "finanza-pubblica" for r in summary["repos"] - ) - - -def test_explore_ref_dataset_repo_domain(): - idx = _index() - ds = explore_ref(idx, "ispra_ru_base") - assert ds["found"] is True - assert ds["type"] == "dataset" - assert ds["registry_source"] == "rifiuti-urbani" - assert ds["analyses"] == ["rifiuti-urbani"] - - repo = explore_ref(idx, "open-conto-annuale") - assert repo["found"] is True - assert repo["type"] == "repo" - assert repo["datasets"] == ["bdap_entrate_stato"] - - dom = explore_ref(idx, "territorio-ambiente") - assert dom["found"] is True - assert dom["type"] == "domain" - assert "ispra_ru_base" in dom["datasets"] - - -def test_explore_ref_analysis_and_source(): - idx = _index() - ana = explore_ref(idx, "rifiuti-urbani") - # slug collides with repo name — repo wins if listed after dataset miss; - # analysis slug is also rifiuti-urbani. Dataset miss → repo hit. - assert ana["found"] is True - - src = explore_ref(idx, "ispra") - assert src["found"] is True - assert src["type"] == "source" - assert src["datasets"] == ["ispra_ru_base"] - - -def test_explore_ref_source_dedupes_slugs(): - """Same dataset slug under one source must appear once.""" - idx = _index() - # Inject duplicate entry under ispra source - idx["datasets"]["ispra"].append(dict(FLAT[0])) - src = explore_ref(idx, "ispra") - assert src["found"] is True - assert src["datasets"] == ["ispra_ru_base"] - assert src["n_datasets"] == 1 - - -def test_explore_ref_not_found(): - assert explore_ref(_index(), "nope")["found"] is False - - -# ── Tokenized natural-language queries (regression: multi-word discovery) ──── - - -def test_tokenize_query_drops_stopwords_and_accents(): - from agent_context_builder.navigation import _tokenize_query - - assert _tokenize_query("bandi di gara") == ["bandi", "gara"] - assert _tokenize_query("appalti pubblici ANAC") == ["appalti", "pubblici", "anac"] - assert _tokenize_query("spesa più recente") == ["spesa", "recente"] - assert _tokenize_query("di delle dei") == [] - - -def test_field_words_split_snake_case(): - from agent_context_builder.navigation import _field_words - - assert _field_words("anac_bandi_gara") == ["anac", "bandi", "gara"] - - -def test_word_forms_match_plural_gender(): - from agent_context_builder.navigation import _word_forms - - assert _word_forms("gara") & _word_forms("gare") - assert _word_forms("pubblico") & _word_forms("pubblici") - assert _word_forms("appalto") & _word_forms("appalti") - - -def test_query_matches_cross_text_tokens(): - from agent_context_builder.navigation import query_matches - - # tokens spread across separate texts (old per-field AND would fail) - assert query_matches("appalti anac", "appalti", "anac") - assert not query_matches("appalti anac banane", "appalti", "anac") - # stopword-only query never matches - assert not query_matches("di dei") - - -def test_search_index_natural_query_cross_field(): - """Regression: 'appalti pubblici anac' must find anac_bandi_gara. - - Tokens live in different fields (category, description, source_id) — - the old matcher required all words inside a single field. - """ - results = search_index(_index(), "appalti pubblici anac", entity_type="dataset") - assert results - assert results[0]["slug"] == "anac_bandi_gara" - - -def test_search_index_stopword_query_finds_snake_case_slug(): - """Regression: 'bandi di gara' must find anac_bandi_gara despite 'di'.""" - results = search_index(_index(), "bandi di gara", entity_type="dataset") - slugs = [r["slug"] for r in results] - assert "anac_bandi_gara" in slugs - - -def test_search_index_morphology_plural(): - """'gare' matches name/tags containing 'gara' (final-vowel strip).""" - results = search_index(_index(), "bandi gare anac", entity_type="dataset") - assert results and results[0]["slug"] == "anac_bandi_gara" - - -def test_search_index_column_name(): - """Regression: a query matching a column name (not in slug/name/tags/desc) - must find the dataset — columns are now an indexed field.""" - # 'importo' lives only in the column name, not in any other field - results = search_index(_index(), "importo aggiudicazione", entity_type="dataset") - assert results and results[0]["slug"] == "anac_bandi_gara" - - -def test_search_index_column_semantic_type(): - """semantic_type 'cig_code' is indexed as a tag: query 'cig' finds it.""" - results = search_index(_index(), "cig", entity_type="dataset") - assert results and results[0]["slug"] == "anac_bandi_gara" - - -def test_search_index_single_word_still_works(): - results = search_index(_index(), "rifiuti", entity_type="dataset") - assert results and results[0]["slug"] == "ispra_ru_base" - - -def test_search_index_single_stopword_returns_nothing(): - """Regression: 'e' used to substring-match every slug (score 80).""" - assert search_index(_index(), "e", entity_type="dataset") == [] - assert search_index(_index(), "di dei delle", entity_type="dataset") == [] - - -def test_search_index_natural_question_drops_function_words(): - """Regression: question/function words are stopwords, not required tokens.""" - assert _tokenize_query("quanti bandi di gara ha anac") == ["bandi", "gara", "anac"] - results = search_index(_index(), "quanti bandi di gara ha anac", entity_type="dataset") - assert results and results[0]["slug"] == "anac_bandi_gara" From 40c1d0f584bd752e1232879f2e0dbad66645ced5 Mon Sep 17 00:00:00 2001 From: Zio Gabber <78922322+Gabrymi93@users.noreply.github.com> Date: Fri, 16 Oct 2026 12:19:12 +0100 Subject: [PATCH 2/2] chore: rimuovi vecchia procedura --- workflows/lab-check.md | 60 ------------------------------------------ 1 file changed, 60 deletions(-) delete mode 100644 workflows/lab-check.md diff --git a/workflows/lab-check.md b/workflows/lab-check.md deleted file mode 100644 index 00f5ec6..0000000 --- a/workflows/lab-check.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -name: lab-check -description: Check rapido delle novità e dello stato del Lab tramite MCP dataciviclab-context. -license: MIT -metadata: - version: "0.3" - owner: "DataCivicLab" - tags: [context, triage, check, mcp] ---- - -# Workflow: lab-check - -Obiettivo: orientarsi rapidamente su PR, issue, discussion, dataset, fonti e blocchi operativi via MCP. - -## Quando usarlo - -- Inizio sessione senza quadro chiaro. -- Controllo warning, PR, issue o discussion per una macro-area. -- Triage tematico prima di aprire file o repo specifici. -- **Non usare** per task tecnici isolati, build dati o editing di codice. - -## Profilo operativo - -- **Shared-mode MCP**: percorso primario di questo workflow. -- **Local-mode git**: se hai il workspace locale, leggi stato git direttamente (`git status`, branch, dirty) — il triage MCP resta utile per PR/issue/discussion, non per lo stato dei branch locali. Non invocare `refresh_context` se il tuo quadro viene già dal git locale. - -Precondizione: server MCP `dataciviclab-context` accessibile. - -## Passi canonici - -1. **session_bootstrap**: prima chiamata. Leggi repo attivi, PR, issue, discussion e warning. -2. **lab_map**: struttura del Lab (domains → repos → sources). Usa `domain=` per una macro-area. -3. **find**: discovery con query naturale (es. `find("bandi anac", type="dataset")`). Restituisce type + parent pointer. -4. **explore**: deep-dive su un ref trovato (slug repo, dataset, domain, source, analysis). -5. **workspace_triage**: se il problema riguarda stato incrociato di repo o issue (`section=radar|prs|issues|discussions|registry|source_health|pipeline`). -6. **refresh_context**: solo se i dati sono palesemente obsoleti dopo merge/push. Attendere la CI. - -## Stop rules ed errori - -- Fermati se i tool MCP falliscono o restituiscono JSON non leggibile. -- Non usare `refresh_context` a ogni sessione: e' eccezione, non default. -- Non scrivere commenti o aprire issue solo da triage: valida l'attualità. -- Ricorda il lag fisiologico tra GitHub/CI e stato git locale. - -## Output minimo - -Il workflow è completo quando hai uno di questi esiti: - -- artifact identificato: PR, issue, discussion, dataset o fonte; -- nessuna novità critica: puoi continuare il task già noto; -- contesto bloccato: serve refresh o debug MCP. - -## Stati finali -- `checked`: Orientamento completato, pronto al lavoro. -- `waiting-for-refresh`: Rebuild CI in corso. -- `blocked-on-context`: Errore tecnico dei tool context. - -## Riferimenti -- README `agent-context-builder` -- MCP `dataciviclab-context`