diff --git a/README.md b/README.md index e07b684..a48aeaf 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # HelixAgent

- Durable autonomous-agent runtime in Python: a bounded plan/execute/observe/replan loop with governed tools, approval gates, and SQLite checkpoints for resumable runs. FastAPI service with Prometheus + OpenTelemetry, optional C++ acceleration with Python fallback, and reproducible benchmarks. + Durable autonomous-agent runtime in Python: a bounded plan/execute/observe/replan loop with governed tools, approval gates, and SQLite checkpoints for resumable runs. FastAPI service with Prometheus + OpenTelemetry, NumPy/BLAS vector operations, optional C++ ctypes interop, pure-Python fallback, and reproducible benchmarks.

@@ -22,13 +22,13 @@ Live Streamlit demo

-HelixAgent is a durable Python agent runtime built around a bounded plan/execute/observe/replan loop with governed tools, approval gates, and SQLite checkpoints. Its included planner is deterministic and rule-based; the planner protocol is extensible, but no model provider is implemented. Optional C++ cosine similarity degrades gracefully to a scale-stable Python fallback when the shared library is unavailable. +HelixAgent is a durable Python agent runtime built around a bounded plan/execute/observe/replan loop with governed tools, approval gates, and SQLite checkpoints. Its included planner is deterministic and rule-based; the planner protocol is extensible, but no model provider is implemented. NumPy is the default cosine-similarity backend. The optional C++ ctypes binding is an FFI demonstration, not a performance claim; a scale-stable Python implementation is retained only for NumPy-unavailable environments. ## Features - **Bounded autonomous execution:** A typed plan/execute/observe/replan loop enforces iteration and tool-call budgets. -- **Deterministic planning and native acceleration:** The typed planner protocol uses a rule-based default, while `ctypes` optionally loads a C++ cosine-similarity library. -- **Resilient fallbacks:** Python planning and vector implementations keep the agent usable without native artifacts. +- **Deterministic planning and vector backends:** The typed planner protocol uses a rule-based default; NumPy/BLAS is the default vector path, while the opt-in `ctypes` binding demonstrates C++ interop. +- **Resilient fallbacks:** The pure-Python vector implementation is used only if the declared NumPy dependency is unavailable. - **FastAPI service:** `/`, `/health`, and `/predict` endpoints with generated OpenAPI documentation. - **Observability:** Prometheus metrics and OpenTelemetry instrumentation are attached to the API. - **Interactive demo:** A Streamlit interface exercises the same agent runtime. @@ -48,7 +48,7 @@ Client / Streamlit Autonomous runtime ----> SQLite checkpoints | | | | | +--> Governed tool registry + approval gates - | +----------> C++ vector library -> Python fallback + | +----------> NumPy (BLAS) default -> optional C++ via ctypes (interop demo) -> pure-Python fallback +-------------------> Planner protocol -> deterministic default ``` @@ -62,7 +62,7 @@ durability. This keeps a future model planner from bypassing execution invariant | Safety | Pause write/destructive tools for explicit approval | Safer default with additional operator latency | | Runaway control | Bound iterations, tool calls, retries, and tool duration | Predictable cost; a valid long task may exhaust its budget | | Planner extensibility | Typed `Planner` protocol with rule-based default | Credential-free execution; no model provider is implemented | -| Native acceleration | Optional C++ cosine similarity with Python fallback | Portable behavior with environment-dependent performance | +| Vector interop and fallback | NumPy/BLAS default, optional C++ ctypes binding, Python degradation path | Portable behavior; C++ demonstrates FFI and is not claimed to beat BLAS | Runtime invariants are covered by tests: terminal states are persisted, denied tools are never executed, budget exhaustion fails closed, retries are bounded, and timeout responses do not wait @@ -89,14 +89,29 @@ not production SLOs or cross-hardware claims. Reproduce locally with: ```bash python -m benchmarks.autonomy_runtime --iterations 200 --warmup 20 +python -m benchmarks.vector_ops --output vector-ops-results.json ``` See [benchmark methodology and limitations](docs/BENCHMARKS.md) for metric definitions and the evaluation boundary. CI also uploads a fresh `benchmark-results.json` artifact on Python 3.11. +## Vector-operations benchmark + +The vector benchmark measures the NumPy default, the optional C++ ctypes backend when its shared library is present, and the pure-Python implementation. It reports measurements from the machine that runs it; it does not rank backends or claim that C++ outperforms BLAS. + +~~~bash +python -m benchmarks.vector_ops --sizes 128 1024 10000 --warmup 10 --repetitions 100 --output vector-ops-results.json +~~~ + +| Backend | 128 | 1k | 10k | +|---|---|---|---| +| NumPy | | | | +| C++ ctypes (when available) | | | | +| Pure Python | | | | + ## Evidence boundaries -The CI matrix exercises Python 3.10 and 3.11 quality/tests, container API health, and Streamlit startup; security and supply-chain workflows run separately. Runtime contract coverage includes terminal-run idempotence, approval gating, bounded retries and budgets, persisted failure for unknown tools, and Python vector fallback properties. The C++ path remains optional and environment-dependent, so native-enabled parity is not claimed. +The CI matrix exercises Python 3.10 and 3.11 quality/tests, container API health, and Streamlit startup; security and supply-chain workflows run separately. Runtime contract coverage includes terminal-run idempotence, approval gating, bounded retries and budgets, persisted failure for unknown tools, and Python vector fallback properties. The C++ path remains optional and environment-dependent; it is an FFI demonstration rather than a claim of better performance than NumPy. For the full claim-to-evidence map, invariant definitions, and reproducible statistical primitives, see [claims matrix](docs/CLAIMS_MATRIX.md), [runtime invariants](docs/RUNTIME_INVARIANTS.md), and [evaluation notes](benchmarks/eval/README.md). diff --git a/agent/agent_core.py b/agent/agent_core.py index e7d3233..eea9cee 100644 --- a/agent/agent_core.py +++ b/agent/agent_core.py @@ -1,14 +1,25 @@ -"""HelixAgent compatibility facade and optional native vector integration.""" +""""HelixAgent compatibility facade and honest vector-operation backends.""" from __future__ import annotations import ctypes import logging import math +import os +from collections.abc import Iterable from pathlib import Path +from typing import Literal + +try: + import numpy as np +except ImportError: # pragma: no cover - NumPy is a declared runtime dependency. + np = None log = logging.getLogger(__name__) +VectorBackend = Literal["auto", "numpy", "cpp"] +VectorInput = Iterable[float] + _lib_vec = None _LIB_PATH = Path(__file__).parent / "cpp" / "libvector.so" try: @@ -24,27 +35,107 @@ log.warning("Native vector library unavailable: %s", exc) -def cosine_sim(left: list[float], right: list[float]) -> float: - """Return cosine similarity using the native library when available.""" - if not left or len(left) != len(right): +def _coerce_numpy_vectors(left: object, right: object) -> tuple[object, object]: + """Return validated, one-dimensional, contiguous float64 NumPy buffers.""" + if np is None: + raise RuntimeError("NumPy is unavailable; use the Python fallback instead") + try: + left_array = np.ascontiguousarray(left, dtype=np.float64) + right_array = np.ascontiguousarray(right, dtype=np.float64) + except (TypeError, ValueError) as exc: + raise TypeError( + "Vectors must be coercible to one-dimensional float64 arrays" + ) from exc + if left_array.ndim != 1 or right_array.ndim != 1: + raise ValueError("Vectors must be one-dimensional") + if left_array.size == 0 or left_array.size != right_array.size: + raise ValueError("Vectors must be non-empty and have equal dimensions") + return left_array, right_array + + +def _coerce_python_vectors( + left: VectorInput, right: VectorInput +) -> tuple[list[float], list[float]]: + """Validate iterable inputs without requiring NumPy.""" + try: + left_values = [float(value) for value in left] + right_values = [float(value) for value in right] + except (TypeError, ValueError) as exc: + raise TypeError("Vectors must be iterable numeric values") from exc + if not left_values or len(left_values) != len(right_values): raise ValueError("Vectors must be non-empty and have equal dimensions") - if _lib_vec is not None: - array_type = ctypes.c_double * len(left) - return float(_lib_vec.cosine_similarity(array_type(*left), array_type(*right), len(left))) - # Normalize before accumulating the dot product. Squaring tiny or huge - # components first can underflow or overflow even when cosine is well-defined. - left_magnitude = math.hypot(*left) - right_magnitude = math.hypot(*right) + return left_values, right_values + + +def cosine_similarity_numpy(left: object, right: object) -> float: + """Return cosine similarity through contiguous float64 NumPy arrays.""" + left_array, right_array = _coerce_numpy_vectors(left, right) + denominator = np.linalg.norm(left_array) * np.linalg.norm(right_array) + if denominator == 0.0: + return 0.0 + return float(np.dot(left_array, right_array) / denominator) + + +def cosine_similarity_cpp(left: object, right: object) -> float: + """Return cosine similarity through the opt-in C++ ctypes demonstration.""" + if _lib_vec is None: + raise RuntimeError( + "C++ vector backend is unavailable; build agent/cpp/libvector.so first" + ) + left_array, right_array = _coerce_numpy_vectors(left, right) + # C++ reads raw double pointers: float32 or non-contiguous buffers would make it + # read the wrong bytes, so coercion above is a required FFI boundary guarantee. + pointer_type = ctypes.POINTER(ctypes.c_double) + return float( + _lib_vec.cosine_similarity( + left_array.ctypes.data_as(pointer_type), + right_array.ctypes.data_as(pointer_type), + left_array.size, + ) + ) + + +def cosine_similarity_python(left: VectorInput, right: VectorInput) -> float: + """Return a scale-stable pure-Python cosine similarity.""" + left_values, right_values = _coerce_python_vectors(left, right) + left_magnitude = math.hypot(*left_values) + right_magnitude = math.hypot(*right_values) if not left_magnitude or not right_magnitude: return 0.0 normalized_dot = math.fsum( - (a / left_magnitude) * (b / right_magnitude) - for a, b in zip(left, right, strict=True) + (first / left_magnitude) * (second / right_magnitude) + for first, second in zip(left_values, right_values, strict=True) ) return max(-1.0, min(1.0, normalized_dot)) +def cpp_backend_available() -> bool: + """Return whether the optional C++ shared library loaded successfully.""" + return _lib_vec is not None + + +def cosine_sim( + left: object, right: object, *, backend: VectorBackend | None = None +) -> float: + """Return cosine similarity with an explicit, evidence-based dispatch order. + + NumPy is the default because it is a declared dependency and uses the platform's + BLAS-backed vector operations. The C++ backend is opt-in via backend="cpp" or + HELIXAGENT_VECTOR_BACKEND=cpp to demonstrate safe native interop; it is not + selected as a performance optimization. The pure-Python implementation is used + only when NumPy is unavailable, preserving a last-resort degradation path. + """ + selected = backend or os.getenv("HELIXAGENT_VECTOR_BACKEND", "auto").lower() + if selected not in {"auto", "numpy", "cpp"}: + raise ValueError("Vector backend must be one of: auto, numpy, cpp") + if selected == "cpp": + return cosine_similarity_cpp(left, right) + if np is None: + return cosine_similarity_python(left, right) + return cosine_similarity_numpy(left, right) + + class AgenticAssistant: """Backward-compatible synchronous interface over the durable autonomous runtime.""" diff --git a/benchmarks/vector_ops.py b/benchmarks/vector_ops.py new file mode 100644 index 0000000..d73af0a --- /dev/null +++ b/benchmarks/vector_ops.py @@ -0,0 +1,159 @@ +""""Reproducible three-way cosine-similarity backend benchmark. + +The report contains measurements from the current machine only. It does not assert +a winner: NumPy is the default backend, the optional C++ path demonstrates ctypes +interop, and the Python implementation is a degradation path. +""" + +from __future__ import annotations + +import argparse +import json +import math +import platform +import statistics +import time +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import numpy as np + +from agent.agent_core import ( + cosine_similarity_cpp, + cosine_similarity_numpy, + cosine_similarity_python, + cpp_backend_available, +) + +DEFAULT_SIZES = (128, 1_024, 10_000) + + +def percentile(values: list[float], percentile_value: float) -> float: + """Return a nearest-rank percentile for a non-empty sample.""" + ordered = sorted(values) + index = max(0, math.ceil(percentile_value * len(ordered)) - 1) + return ordered[index] + + +def _time_backend( + backend: Callable[[object, object], float], + left: object, + right: object, + *, + repetitions: int, + warmup: int, +) -> dict[str, float]: + for _ in range(warmup): + backend(left, right) + + samples_us: list[float] = [] + for _ in range(repetitions): + started = time.perf_counter_ns() + backend(left, right) + samples_us.append((time.perf_counter_ns() - started) / 1_000) + + return { + "mean_us": round(statistics.fmean(samples_us), 6), + "p50_us": round(statistics.median(samples_us), 6), + "p95_us": round(percentile(samples_us, 0.95), 6), + } + + +def run_benchmark( + *, + sizes: tuple[int, ...] = DEFAULT_SIZES, + repetitions: int = 100, + warmup: int = 10, + seed: int = 17_290, +) -> dict[str, Any]: + """Measure every available vector backend with deterministic input vectors.""" + if not sizes or any(size < 1 for size in sizes): + raise ValueError("sizes must contain positive dimensions") + if repetitions < 1 or warmup < 0: + raise ValueError("repetitions must be positive and warmup cannot be negative") + + backends: list[tuple[str, Callable[[object, object], float]]] = [ + ("numpy", cosine_similarity_numpy), + ("python", cosine_similarity_python), + ] + if cpp_backend_available(): + backends.insert(1, ("cpp", cosine_similarity_cpp)) + + generator = np.random.default_rng(seed) + measurements: list[dict[str, Any]] = [] + for size in sizes: + left = np.ascontiguousarray(generator.standard_normal(size), dtype=np.float64) + right = np.ascontiguousarray(generator.standard_normal(size), dtype=np.float64) + for name, backend in backends: + result = _time_backend( + backend, left, right, repetitions=repetitions, warmup=warmup + ) + measurements.append( + { + "backend": name, + "vector_size": size, + "repetitions": repetitions, + **result, + } + ) + + return { + "benchmark": "vector_ops", + "scope": "local cosine-similarity backend timings; no claim of cross-host performance", + "environment": { + "python": platform.python_version(), + "platform": platform.platform(), + "processor": platform.processor() or "not reported", + "numpy": np.__version__, + }, + "workload": { + "sizes": list(sizes), + "warmup_runs_per_backend": warmup, + "measured_runs_per_backend": repetitions, + "random_seed": seed, + "cpp_backend_available": cpp_backend_available(), + }, + "measurements": measurements, + } + + +def print_table(report: dict[str, Any]) -> None: + """Print the measured backend timings without ranking or interpretation.""" + print("backend | vector size | mean (us) | p50 (us) | p95 (us)") + print("--- | ---: | ---: | ---: | ---:") + for measurement in report["measurements"]: + print( + f"{measurement['backend']} | {measurement['vector_size']} | " + f"{measurement['mean_us']:.6f} | {measurement['p50_us']:.6f} | " + f"{measurement['p95_us']:.6f}" + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--sizes", nargs="+", type=int, default=list(DEFAULT_SIZES)) + parser.add_argument("--repetitions", type=int, default=100) + parser.add_argument("--warmup", type=int, default=10) + parser.add_argument("--seed", type=int, default=17_290) + parser.add_argument( + "--output", + type=Path, + default=Path("vector-ops-results.json"), + help="JSON artifact path written with the measurements from this run", + ) + args = parser.parse_args() + report = run_benchmark( + sizes=tuple(args.sizes), + repetitions=args.repetitions, + warmup=args.warmup, + seed=args.seed, + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print_table(report) + print(f"JSON artifact: {args.output}") + + +if __name__ == "__main__": + main() diff --git a/docs/CLAIMS_MATRIX.md b/docs/CLAIMS_MATRIX.md index 6cc67dd..1a2db40 100644 --- a/docs/CLAIMS_MATRIX.md +++ b/docs/CLAIMS_MATRIX.md @@ -5,12 +5,12 @@ | Autonomous planning | Repository About claims LLM reasoning | RuleBasedPlanner is keyword-driven and deterministic; no provider call exists | Deterministic runtime tests | Describe as deterministic rule-based planning; do not claim LLM orchestration | | Enterprise data tools | Repository About claims enterprise data tools | Optional web, Snowflake, and SageMaker modules exist; the governed default registry only wires web search, vector similarity, and synthesis | Default runtime tests cover vector and synthesis; no integration tests prove Snowflake or SageMaker | Do not use enterprise-data-tools as a system claim | | Java planner | Java planners exist | Two duplicate keyword planners are present; Python runtime has no JPype or JAR invocation | No integration test | Remove Java as unintegrated duplicate code | -| Native vector operations | README claims optional C++ vector operations | ctypes loads libvector.so when present; Python cosine is fallback | Fallback and property tests define compatible mathematical behavior | Retain with explicit environment-dependent native availability | +| Vector-operation backends | README describes NumPy/BLAS as default, optional C++ ctypes interop, and Python degradation | NumPy performs default cosine dispatch; ctypes uses contiguous float64 buffers only when requested; Python is used when NumPy is unavailable | Backend cases, agreement, coercion, and dispatch tests | C++ is an FFI demonstration, not a performance claim | | SQLite checkpoints | README claims durable checkpoints | SQLiteRunStore persists typed AgentRun JSON after transitions | Runtime persistence tests | Retain; qualify as single-process durability | | Reproducible benchmark | README publishes a reference observation | Benchmark command emits machine-readable local deterministic control-loop metrics | Benchmark contract test | Retain as local microbenchmark only | ## Recommended GitHub About description -Deterministic, budgeted Python agent runtime with governed tools, SQLite checkpoints, FastAPI endpoints, and optional C++ cosine-similarity acceleration. +Durable Python agent runtime with governed tools, SQLite checkpoints, FastAPI endpoints, NumPy-default vector operations, optional C++ ctypes interop, and a pure-Python fallback. Update the GitHub About/description field manually; this workflow changes repository files, not GitHub repository metadata. diff --git a/docs/architecture.md b/docs/architecture.md index c22abe5..0556cc8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,23 +1,34 @@ -# 🏗️ Agentic AI Assistant — System Architecture +# HelixAgent Runtime Architecture -## 1 – High-Level Overview -The Agentic AI Assistant is an autonomous, multi-tool agent designed to plan, execute, and evaluate complex tasks while logging production-grade metrics. It merges **three languages** to showcase full-stack ML engineering: +## High-level overview -| Layer | Language | Key Role | -|-------|----------|----------| -| Planner | **Java 17** | Deterministic or LLM-enhanced task decomposition (`planner.jar`) | -| Orchestration | **Python 3.10** | LangGraph agent (`agent_core.py`) routes steps, handles memory and LLM calls | -| High-Perf Tool | **C++17** | `libvector.so` delivers ultrafast cosine-similarity and vector math | +HelixAgent is a single-service Python reference implementation of a bounded +plan/execute/observe/replan runtime. The shipped planner is deterministic and +rule-based; the planner protocol can support other implementations, but no model +provider, Java planner, or distributed scheduler is part of the runtime. -## 2 – Execution Flow -```mermaid +| Layer | Implementation | Role | +|---|---|---| +| Planner | Python | Deterministic typed-task proposal | +| Runtime | Python | Budgets, approval gates, retries, timeouts, and SQLite checkpoints | +| Vector operations | NumPy, optional C++ ctypes, Python | NumPy/BLAS default; C++ is an FFI demonstration; Python degrades when NumPy is unavailable | +| Service | FastAPI | Health, prediction, Prometheus, and OpenTelemetry endpoints | + +## Execution flow + +~~~mermaid flowchart LR - A[User Prompt] --> B(Java Planner)
createPlan() - B --> C[LangGraph Graph]
state machine - C -->|vector_similarity| D[libvector.so (C++)] - C -->|web_search| E[DuckDuckGo API] - C -->|snowflake_query| F[Snowflake] - C -->|sagemaker_batch| G[SageMaker] - C --> H[LLM (OpenAI / Bedrock)] - H --> C - C --> Z[Final Answer] + A[User prompt] --> B[Rule-based planner] + B --> C[Python autonomous runtime] + C --> D[SQLite checkpoints] + C --> E[Governed tool registry] + E --> F[NumPy cosine similarity default] + F -. opt-in interop .-> G[Optional C++ ctypes binding] + F -. NumPy unavailable .-> H[Pure-Python cosine similarity] + C --> I[FastAPI response] +~~~ + +The runtime owns state transitions and execution bounds; the tool registry owns +risk and timeout policy; SQLite owns local checkpoint persistence. The optional +C++ shared library is not selected by default and is not presented as a +performance replacement for NumPy/BLAS. diff --git a/docs/metrics.md b/docs/metrics.md index 0a6cd71..2a80744 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -1,36 +1,33 @@ -# 📊 Agentic AI Assistant — Performance & Cost Metrics - -This page tracks measurable outcomes from the autonomous agent in staging (EKS) and local-dev modes. -Metrics are logged to **Snowflake** (`AI_METRICS.PUBLIC.AGENT_RUNS`) and refreshed nightly. - -| Metric | Staging Avg | Local Dev | Definition / Method | -|--------|-------------|-----------|---------------------| -| **Task Success-Rate** | **94 %** | 92 % | `% of runs with all plan steps completed without manual retry` | -| **Median Latency** | 5.7 s | 4.1 s | `plan() ➜ final_answer()` wall-clock, 50th % | -| **P95 Latency** | 11.3 s | 9.6 s | 95th percentile end-to-end | -| **Prompt Tokens (μ)** | 186 | 172 | Avg input tokens per step (OpenAI tiktoken) | -| **Completion Tokens (μ)** | 245 | 231 | Avg output tokens | -| **Snowflake Credits/Run** | 0.002 | — | Calculated via `WAREHOUSE_METERING_HISTORY` | -| **SageMaker Cost/Batch** | $0.012 | — | (`TransformJobDuration` × instance $/s) | -| **Vector Cosine Sim Speed-up** | 18× | n/a | C++ `libvector.so` vs. NumPy (10k × 300 vec) | -| **Test Coverage** | ![Coverage](https://codecov.io/gh/Trojan3877/Agentic-AI-Assistant/branch/main/graph/badge.svg) | — | Auto-uploaded by CI | - -## 📈 Data Collection - -| Source | Tool | -|--------|------| -| Latency / tokens | FastAPI middleware → Snowflake `INSERT` | -| Success flag | Agent state machine sets `status="success" | "fail"` | -| Cost metrics | AWS Cost Explorer API; Snowflake Warehouse Metering | - -## 🧮 KPI Thresholds (SLOs) - -| KPI | Target | -|-----|--------| -| P95 Latency | ≤ 12 s | -| Task Success-Rate | ≥ 90 % | -| Snowflake Credits / run | ≤ 0.003 | - -> _Metrics auto-generated via GitHub Actions nightly schedule (`.github/workflows/metrics-export.yml`)._ - -_Last updated: {{DATE}}_ +# Evidence-bound metrics + +This repository does not provide verified staging, Snowflake, SageMaker, or +provider-cost measurements. Those systems are not exercised by the governed +default runtime, so no production KPI, speed-up, or cost claim is published here. + +## Reproducible local measurements + +The deterministic autonomy control-loop benchmark remains documented in +[Benchmark methodology](BENCHMARKS.md). It measures local orchestration and +SQLite checkpoint overhead only. + +Vector-operation timing is a separate experiment. Run it on the target machine +and retain the JSON artifact with the environment metadata: + +~~~bash +python -m benchmarks.vector_ops --sizes 128 1024 10000 --warmup 10 --repetitions 100 --output vector-ops-results.json +~~~ + +| Backend | Result | +|---|---| +| NumPy default | | +| Optional C++ ctypes interop | | +| Pure-Python degradation path | | + +The benchmark reports measurements from that invocation. It makes no cross-host +comparison and does not present the C++ binding as faster than NumPy/BLAS. + +## Observability surface + +The FastAPI service exposes Prometheus metrics and OpenTelemetry instrumentation. +Those integration points are observable interfaces, not evidence of a deployed +telemetry pipeline or service-level objective. diff --git a/requirements.txt b/requirements.txt index 0659521..7405851 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,6 +3,7 @@ # ================================================================ # --- Core runtime --- +numpy==2.2.6 pandas==2.3.3 scikit-learn==1.7.2 pyyaml==6.0.1 diff --git a/streamlit_app.py b/streamlit_app.py index be82f28..c3f8b0c 100644 --- a/streamlit_app.py +++ b/streamlit_app.py @@ -58,7 +58,7 @@ def render_sidebar() -> None: - LangGraph workflow orchestration - Python fallback planner - Optional Java planner integration -- Optional C++ vector acceleration +- NumPy-default vectors with optional C++ ctypes interop - FastAPI service layer - CI/CD and supply-chain security """ diff --git a/tests/test_benchmark.py b/tests/test_benchmark.py index 86793e8..9c6f4b7 100644 --- a/tests/test_benchmark.py +++ b/tests/test_benchmark.py @@ -1,6 +1,7 @@ """Contract tests for the reproducible autonomy microbenchmark.""" from benchmarks.autonomy_runtime import run_benchmark +from benchmarks.vector_ops import run_benchmark as run_vector_benchmark def test_autonomy_benchmark_reports_auditable_metrics() -> None: @@ -10,3 +11,13 @@ def test_autonomy_benchmark_reports_auditable_metrics() -> None: assert report["results"]["success_rate_percent"] == 100.0 assert report["results"]["tool_calls"] == 6 assert report["results"]["run_latency_ms_p95"] > 0 + + +def test_vector_benchmark_reports_current_measurements() -> None: + report = run_vector_benchmark(sizes=(4,), repetitions=2, warmup=1, seed=23) + + assert report["workload"]["sizes"] == [4] + assert {"numpy", "python"} <= { + measurement["backend"] for measurement in report["measurements"] + } + assert all(measurement["mean_us"] > 0 for measurement in report["measurements"]) diff --git a/tests/test_runtime_invariants.py b/tests/test_runtime_invariants.py index 2abbe85..3bb72e6 100644 --- a/tests/test_runtime_invariants.py +++ b/tests/test_runtime_invariants.py @@ -71,9 +71,9 @@ def test_unknown_planned_tool_fails_and_persists(tmp_path: Path) -> None: def test_python_vector_fallback_defines_zero_and_dimension_behavior(monkeypatch) -> None: monkeypatch.setattr(agent_core, "_lib_vec", None) - assert agent_core.cosine_sim([0.0, 0.0], [1.0, -1.0]) == 0.0 + assert agent_core.cosine_similarity_python([0.0, 0.0], [1.0, -1.0]) == 0.0 with pytest.raises(ValueError, match="equal dimensions"): - agent_core.cosine_sim([1.0], [1.0, 2.0]) + agent_core.cosine_similarity_python([1.0], [1.0, 2.0]) @pytest.mark.parametrize( @@ -89,7 +89,7 @@ def test_python_vector_fallback_is_stable_at_extreme_scales( monkeypatch.setattr(agent_core, "_lib_vec", None) assert math.isclose( - agent_core.cosine_sim(left, right), expected, rel_tol=1e-12, abs_tol=1e-12 + agent_core.cosine_similarity_python(left, right), expected, rel_tol=1e-12, abs_tol=1e-12 ) @@ -119,9 +119,9 @@ def test_python_cosine_fallback_satisfies_basic_properties(pair) -> None: patch.setattr(agent_core, "_lib_vec", None) left, right = pair - score = agent_core.cosine_sim(left, right) - reverse = agent_core.cosine_sim(right, left) - self_score = agent_core.cosine_sim(left, left) + score = agent_core.cosine_similarity_python(left, right) + reverse = agent_core.cosine_similarity_python(right, left) + self_score = agent_core.cosine_similarity_python(left, left) assert -1.0 - 1e-12 <= score <= 1.0 + 1e-12 assert math.isclose(score, reverse, rel_tol=1e-12, abs_tol=1e-12) diff --git a/tests/test_vector_backends.py b/tests/test_vector_backends.py new file mode 100644 index 0000000..0cb6b45 --- /dev/null +++ b/tests/test_vector_backends.py @@ -0,0 +1,124 @@ +""""Unit contracts for the honest cosine-similarity backends.""" + +from __future__ import annotations + +import math + +import numpy as np +import pytest + +from agent import agent_core + +KNOWN_CASES = [ + ([1.0, 2.0], [1.0, 2.0], 1.0), + ([1.0, 0.0], [0.0, 1.0], 0.0), + ([0.0, 0.0], [1.0, -1.0], 0.0), + ([1.0, 2.0], [-1.0, -2.0], -1.0), +] + + +class RecordingVectorLibrary: + """Small ctypes-compatible stand-in that observes the raw FFI buffers.""" + + def cosine_similarity(self, left_pointer, right_pointer, length) -> float: + left = np.ctypeslib.as_array(left_pointer, shape=(int(length),)) + right = np.ctypeslib.as_array(right_pointer, shape=(int(length),)) + left_norm = np.linalg.norm(left) + right_norm = np.linalg.norm(right) + if left_norm == 0.0 or right_norm == 0.0: + return 0.0 + return float(np.dot(left, right) / (left_norm * right_norm)) + + +@pytest.fixture +def fake_cpp_backend(monkeypatch) -> RecordingVectorLibrary: + library = RecordingVectorLibrary() + monkeypatch.setattr(agent_core, "_lib_vec", library) + return library + + +@pytest.mark.parametrize(("left", "right", "expected"), KNOWN_CASES) +@pytest.mark.parametrize( + "backend", + [agent_core.cosine_similarity_numpy, agent_core.cosine_similarity_python], +) +def test_numpy_and_python_backends_cover_known_cases( + backend, left, right, expected +) -> None: + assert backend(left, right) == pytest.approx(expected, abs=1e-12) + + +@pytest.mark.parametrize(("left", "right", "expected"), KNOWN_CASES) +def test_cpp_backend_covers_known_cases( + fake_cpp_backend, left, right, expected +) -> None: + assert agent_core.cosine_similarity_cpp(left, right) == pytest.approx( + expected, abs=1e-12 + ) + + +@pytest.mark.skipif( + not agent_core.cpp_backend_available(), + reason="optional C++ shared library is not available on this runner", +) +@pytest.mark.parametrize(("left", "right", "expected"), KNOWN_CASES) +def test_loaded_cpp_backend_covers_known_cases(left, right, expected) -> None: + assert agent_core.cosine_similarity_cpp(left, right) == pytest.approx( + expected, abs=1e-12 + ) + + +def test_available_backends_agree_on_seeded_vectors() -> None: + generator = np.random.default_rng(8_675_309) + left = generator.normal(size=257) + right = generator.normal(size=257) + results = [ + agent_core.cosine_similarity_numpy(left, right), + agent_core.cosine_similarity_python(left, right), + ] + if agent_core.cpp_backend_available(): + results.append(agent_core.cosine_similarity_cpp(left, right)) + + for result in results[1:]: + assert math.isclose(results[0], result, rel_tol=1e-9, abs_tol=1e-9) + + +def test_cpp_coerces_float32_and_non_contiguous_arrays(fake_cpp_backend) -> None: + left = np.arange(12, dtype=np.float32)[::2] + right = np.arange(12, dtype=np.float32)[1::2] + assert not left.flags.c_contiguous + assert left.dtype == np.float32 + + expected = agent_core.cosine_similarity_numpy(left, right) + actual = agent_core.cosine_similarity_cpp(left, right) + + assert actual == pytest.approx(expected, abs=1e-12) + + +def test_cpp_boundary_rejects_mismatched_and_uncoercible_inputs( + fake_cpp_backend, +) -> None: + with pytest.raises(ValueError, match="equal dimensions"): + agent_core.cosine_similarity_cpp([1.0], [1.0, 2.0]) + with pytest.raises(TypeError, match="coercible"): + agent_core.cosine_similarity_cpp(["not-a-number"], [1.0]) + + +def test_compatibility_facade_defaults_to_numpy(monkeypatch) -> None: + monkeypatch.delenv("HELIXAGENT_VECTOR_BACKEND", raising=False) + + assert agent_core.cosine_sim([1.0, 0.0], [0.0, 1.0]) == pytest.approx(0.0) + + +def test_compatibility_facade_selects_cpp_only_when_opted_in( + monkeypatch, fake_cpp_backend +) -> None: + monkeypatch.setenv("HELIXAGENT_VECTOR_BACKEND", "cpp") + + assert agent_core.cosine_sim([1.0, 2.0], [1.0, 2.0]) == pytest.approx(1.0) + + +def test_compatibility_facade_uses_python_only_without_numpy(monkeypatch) -> None: + monkeypatch.setattr(agent_core, "np", None) + + assert agent_core.cosine_sim([1.0, 2.0], [-1.0, -2.0]) == pytest.approx(-1.0)