Add Dataset mode for comparing algorithms on real series

README.md

@@ -39,6 +39,16 @@ inv web-start
 
 Then open http://127.0.0.1:8100 in your browser.
 
+### Compare algorithms on a real dataset
+
+A third mode, **Dataset**, lets you load a bundled or pasted time series and
+see which change points each Otava algorithm variant
+(`compute_change_points`, `compute_change_points_orig`,
+`compute_change_points_deterministic`) detects on the same data. The TigerBeetle
+benchmark dataset ships as the default preset. The Otava analysis panel now
+exposes the same algorithm checkboxes in all three modes. See
+`docs/visualizer.md` for details.
+
 ## Installation
 
 ```bash

docs/index.md

@@ -105,6 +105,13 @@ otava-gen list
 otava-gen info step_function
 ```
 
+## Compare algorithms on a real dataset
+
+A third mode, **Dataset**, lets you load a bundled or pasted time series and
+see which change points each Otava algorithm variant detects on the same data.
+The TigerBeetle benchmark dataset ships as the default preset. See the
+[Web Visualizer](visualizer.md) doc for details.
+
 ## Contents
 
 ```{toctree}

docs/visualizer.md

@@ -355,3 +355,72 @@ The visualizer calculates these metrics when comparing Otava's results to ground
 ## Show All Graphs
 
 Click "Show All Graphs" to run analysis on all generators simultaneously. This produces a grid of charts showing how well the analysis methods perform across different types of change point patterns, allowing you to quickly compare detection accuracy across pattern types.
+
+## Dataset Mode — compare algorithms on a real series
+
+The mode toggle in the top section has three options: **Single Pattern**, **Mix
+Patterns**, and **Dataset**. Dataset mode replaces the synthetic generator grid
+with a dataset picker and a "Custom (paste below)" textarea so you can load a
+real time series and see where each Otava algorithm variant places change
+points on it.
+
+Ground-truth-only UI (accuracy metrics, comparison tables, chart legend) is
+hidden in Dataset mode because real data doesn't come with known change points.
+
+### Bundled datasets
+
+| Name | Length | Description |
+|------|--------|-------------|
+| `tigerbeetle` | 365 | Same series used by `apache/otava`'s `perf/perf_test.py`. Has a couple of distinctive ups and downs, an anomalous drop, then an upward slope, then normal variance. |
+
+Add more presets by dropping a file in `src/otava_test_data/datasets/` and
+registering it in `datasets/__init__.py::DATASETS`.
+
+### Algorithm checkboxes
+
+The Otava Analysis panel exposes a checkbox per algorithm variant in all three
+modes:
+
+| Key | Otava function | CLI flag |
+|-----|----------------|----------|
+| `split` | `compute_change_points` | (default) |
+| `orig` | `compute_change_points_orig` | `--orig-edivisive` |
+| `deterministic` | `compute_change_points_deterministic` | `--deterministic-edivisive` ([PR](https://github.com/apache/otava/pull/154)) |
+
+If `compute_change_points_deterministic` isn't importable in the installed
+otava version, the checkbox is disabled with a "(not in installed otava)"
+indicator next to it.
+
+### Custom data
+
+Pick "Custom (paste below)" from the source dropdown and paste a series as
+either a JSON array (`[1.2, 3.4, ...]`) or whitespace/comma-separated numbers.
+Non-numeric tokens are filtered out and the UI tells you how many were dropped.
+
+### HTTP endpoints
+
+- `GET /api/datasets` — bundled-preset metadata.
+- `GET /api/datasets/{name}` — one preset's series + metadata.
+- `GET /api/algorithms` — change-point algorithms exposed by the installed
+  otava version.
+- `POST /api/compare?window_len=...&max_pvalue=...&min_magnitude=...` —
+  run selected algorithms on a series. Body:
+  ```json
+  {"data": [1.2, 3.4, ...], "algorithms": ["split", "orig"]}
+  ```
+  Response:
+  ```json
+  {
+    "results": {
+      "split": {"indices": [15, 71, ...], "count": 8},
+      "orig":  {"indices": [15, 71, ...], "count": 5}
+    },
+    "parameters": {"window_len": 50, "max_pvalue": 0.001, "min_magnitude": 0.0}
+  }
+  ```
+  If `algorithms` is omitted, every available algorithm runs.
+
+The synthetic-pattern endpoints `/api/generate/{name}` and
+`/api/analyze/{name}` also accept an `otava_algorithm` query parameter
+(`split` | `orig` | `deterministic`) so synthetic-pattern flows can pick an
+algorithm variant too.

pyproject.toml

@@ -33,6 +33,7 @@ dev = [
     "pytest-cov>=4.0.0",
     "invoke>=2.0.0",
     "ruff>=0.1.0",
+    "httpx>=0.27.0",  # required by fastapi.testclient
 ]
 docs = [
     "sphinx>=7.0.0",

src/otava_test_data/datasets/__init__.py

@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Real-world performance datasets bundled for algorithm comparison."""
+
+from otava_test_data.datasets.tigerbeetle import TIGERBEETLE
+
+DATASETS = {
+    TIGERBEETLE["name"]: TIGERBEETLE,
+}
+
+
+def list_datasets() -> list[dict]:
+    """Return metadata for every bundled dataset (without the series itself)."""
+    return [
+        {"name": ds["name"], "title": ds["title"], "description": ds["description"]}
+        for ds in DATASETS.values()
+    ]
+
+
+def get_dataset(name: str) -> dict | None:
+    """Return the full dataset entry (series + metadata) or None if unknown."""
+    return DATASETS.get(name)

src/otava_test_data/datasets/tigerbeetle.py

@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+TigerBeetle benchmark series — same values used by apache/otava's
+perf/perf_test.py for algorithm performance benchmarking.
+
+Source: https://github.com/apache/otava/blob/master/perf/perf_test.py
+"""
+
+TIGERBEETLE_SERIES = [
+    26705, 26475, 26641, 26806, 26835, 26911, 26564, 26812, 26874, 26682,
+    15672, 26745, 26460, 26977, 26851, 23412, 23547, 23674, 23519, 23670,
+    23662, 23462, 23750, 23717, 23524, 23588, 23687, 23793, 23937, 23715,
+    23570, 23730, 23690, 23699, 23670, 23860, 23988, 23652, 23681, 23798,
+    23728, 23604, 23523, 23412, 23685, 23773, 23771, 23718, 23409, 23739,
+    23674, 23597, 23682, 23680, 23711, 23660, 23990, 23938, 23742, 23703,
+    23536, 24363, 24414, 24483, 24509, 24944, 24235, 24560, 24236, 24667,
+    24730, 28346, 28437, 28436, 28057, 28217, 28456, 28427, 28398, 28250,
+    28331, 28222, 28726, 28578, 28345, 28274, 28514, 28590, 28449, 28305,
+    28411, 28788, 28404, 28821, 28580, 27483, 26805, 27487, 27124, 26898,
+    27295, 26951, 27312, 27660, 27154, 27050, 26989, 27193, 27503, 27326,
+    27375, 27513, 27057, 27421, 27574, 27609, 27123, 27824, 27644, 27394,
+    27836, 27949, 27702, 27457, 27272, 28207, 27802, 27516, 27586, 28005,
+    27768, 28543, 28237, 27915, 28437, 28342, 27733, 28296, 28524, 28687,
+    28258, 28611, 29360, 28590, 29641, 28965, 29474, 29256, 28611, 28205,
+    28539, 27962, 28398, 28509, 28240, 28592, 28102, 28461, 28578, 28669,
+    28507, 28535, 28226, 28536, 28561, 28087, 27953, 28398, 28007, 28518,
+    28337, 28242, 28607, 28545, 28514, 28377, 28010, 28412, 28633, 28576,
+    28195, 28637, 28724, 28466, 28287, 28719, 28425, 28860, 28842, 28604,
+    28327, 28216, 28946, 28918, 29287, 28725, 29148, 29541, 29137, 29628,
+    29087, 28612, 29154, 29108, 28884, 29234, 28695, 28969, 28809, 28695,
+    28634, 28916, 29852, 29389, 29757, 29531, 29363, 29251, 29552, 29561,
+    29046, 29795, 29022, 29395, 28921, 29739, 29257, 29455, 29376, 29528,
+    28909, 29492, 28984, 29621, 29026, 29457, 29102, 29114, 28924, 29162,
+    29259, 29554, 29616, 29211, 29367, 29460, 28836, 29645, 29586, 28848,
+    29324, 28969, 29150, 29243, 29081, 29312, 28923, 29272, 29117, 29072,
+    29529, 29737, 29652, 29612, 29856, 29012, 30402, 29969, 29309, 29439,
+    29285, 29421, 29023, 28772, 29692, 29416, 29267, 29542, 29904, 30045,
+    29739, 29945, 29141, 29163, 29765, 29197, 29441, 28910, 29504, 29614,
+    29643, 29506, 29420, 29672, 29432, 29784, 29888, 29309, 29247, 29816,
+    29254, 29813, 29451, 29382, 29618, 28558, 29845, 29499, 29283, 29184,
+    29246, 28790, 29952, 29145, 29415, 30437, 29227, 29605, 29859, 29156,
+    29807, 29406, 29734, 29861, 29140, 29983, 29832, 29919, 29896, 29991,
+    29266, 29001, 29459, 29548, 29310, 29042, 29303, 29894, 29091, 29018,
+    29537, 29614, 29180, 29736, 29500, 29218, 29581, 28906, 28542, 29306,
+    28987, 29878, 28865, 30272, 29707, 29662, 29815, 30492, 29347, 30096,
+    29054, 30238, 28813, 31895, 28915,
+]
+
+TIGERBEETLE = {
+    "name": "tigerbeetle",
+    "title": "TigerBeetle",
+    "description": (
+        "TigerBeetle dataset used for demo purposes at Nyrkiö and as the "
+        "performance benchmark series in apache/otava's perf_test.py. "
+        "Contains a couple of distinctive ups and downs, an anomalous drop, "
+        "then an upward slope, with the rest as normal variance."
+    ),
+    "series": TIGERBEETLE_SERIES,
+}

src/otava_test_data/tests/test_compare_api.py

@@ -0,0 +1,161 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for the /api/datasets and /api/compare endpoints."""
+
+from fastapi.testclient import TestClient
+
+from otava_test_data.web.main import app
+
+client = TestClient(app)
+
+
+def test_list_datasets():
+    r = client.get("/api/datasets")
+    assert r.status_code == 200
+    body = r.json()
+    assert "datasets" in body
+    names = [d["name"] for d in body["datasets"]]
+    assert "tigerbeetle" in names
+
+
+def test_get_tigerbeetle():
+    r = client.get("/api/datasets/tigerbeetle")
+    assert r.status_code == 200
+    body = r.json()
+    assert body["name"] == "tigerbeetle"
+    assert len(body["series"]) == 365
+
+
+def test_get_unknown_dataset_404():
+    r = client.get("/api/datasets/nope")
+    assert r.status_code == 404
+
+
+def test_list_algorithms_includes_split():
+    r = client.get("/api/algorithms")
+    assert r.status_code == 200
+    body = r.json()
+    assert "split" in body["algorithms"]
+    # split must be available because apache-otava is a hard dependency.
+    assert body["algorithms"]["split"]["available"] is True
+
+
+def test_compare_runs_split_on_tigerbeetle():
+    series = client.get("/api/datasets/tigerbeetle").json()["series"]
+    r = client.post(
+        "/api/compare?max_pvalue=0.001",
+        json={"data": series, "algorithms": ["split"]},
+    )
+    assert r.status_code == 200
+    body = r.json()
+    assert "split" in body["results"]
+    assert "indices" in body["results"]["split"]
+    # TigerBeetle has clear change points — split should detect at least one.
+    assert body["results"]["split"]["indices"]
+
+
+def test_compare_rejects_empty_data():
+    r = client.post("/api/compare", json={"data": [], "algorithms": ["split"]})
+    assert r.status_code == 400
+
+
+def test_compare_rejects_unknown_algorithm():
+    """Unknown algorithm names should fail validation, not return 200 with an inline error."""
+    r = client.post(
+        "/api/compare",
+        json={"data": [1.0, 2.0, 1.0, 2.0, 1.0, 2.0], "algorithms": ["nonexistent"]},
+    )
+    assert r.status_code == 422
+
+
+def test_compare_default_runs_all_available_algorithms():
+    """When `algorithms` is omitted, every available algorithm in ALGORITHMS runs."""
+    available = {
+        name for name, info in client.get("/api/algorithms").json()["algorithms"].items()
+        if info["available"]
+    }
+    r = client.post(
+        "/api/compare",
+        json={"data": [1.0, 1.1, 1.0, 1.2, 5.0, 5.1, 5.0, 5.2, 5.3, 5.1]},
+    )
+    assert r.status_code == 200
+    assert set(r.json()["results"].keys()) == available
+
+
+def test_compare_accepts_floats_and_round_trips_through_json():
+    """Float series shouldn't blow up at JSON serialization."""
+    series = [1.5, 2.5, 1.7, 2.3, 1.4, 3.9, 4.1, 4.0, 4.2, 4.3, 4.1]
+    r = client.post("/api/compare", json={"data": series, "algorithms": ["split"]})
+    assert r.status_code == 200
+    body = r.json()
+    # Indices must be JSON-serializable plain ints, not numpy types.
+    for idx in body["results"]["split"]["indices"]:
+        assert isinstance(idx, int)
+
+
+def test_index_exposes_dataset_mode():
+    """The main page should expose the dataset-mode button and section."""
+    r = client.get("/")
+    assert r.status_code == 200
+    assert 'id="mode-dataset-btn"' in r.text
+    assert 'id="dataset-section"' in r.text
+    assert 'otava-algo-checkbox' in r.text
+
+
+def test_generate_with_otava_algorithm_orig():
+    """The /api/generate endpoint should accept otava_algorithm=orig."""
+    r = client.get(
+        "/api/generate/step_function",
+        params={
+            "length": 100, "seed": 42,
+            "run_otava": True, "otava_algorithm": "orig",
+            "max_pvalue": 0.001,
+        },
+    )
+    assert r.status_code == 200
+    body = r.json()
+    assert "otava" in body
+    # No error path expected here.
+    assert "error" not in body["otava"]
+    assert "parameters" in body["otava"], body["otava"]
+    assert body["otava"]["parameters"]["algorithm"] == "orig"
+
+
+def test_generate_rejects_unknown_otava_algorithm():
+    r = client.get(
+        "/api/generate/step_function",
+        params={"length": 50, "seed": 42, "otava_algorithm": "nonexistent"},
+    )
+    assert r.status_code == 422
+
+
+def test_analyze_accepts_otava_algorithm():
+    """`/api/analyze` documents the otava_algorithm param — verify it actually works."""
+    r = client.get(
+        "/api/analyze/step_function",
+        params={
+            "length": 100, "seed": 42,
+            "otava_algorithm": "orig",
+            "max_pvalue": 0.001,
+        },
+    )
+    assert r.status_code == 200
+    body = r.json()
+    assert "otava" in body
+    assert "error" not in body["otava"]
+    assert body["otava"]["parameters"]["algorithm"] == "orig"