feat: publish privacy-first GitHub Traffic observer

.github/workflows/validate.yml

@@ -0,0 +1,30 @@
+name: Validate
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+      - name: Compile
+        run: python -m py_compile scripts/archive_traffic.py
+      - name: Run offline regression tests
+        run: python -m unittest discover -s tests -v
+      - name: Ensure no real archive directory is tracked
+        shell: bash
+        run: |
+          set -euo pipefail
+          if git ls-files 'traffic/**' '.traffic-raw/**' 'raw/**' | grep -q .; then
+            echo "Real traffic/raw data must not be tracked in the public repository."
+            exit 1
+          fi

.gitignore

@@ -0,0 +1,19 @@
+# Collected data must stay out of the public repository
+traffic/
+data/
+raw/
+.traffic-raw/
+
+# Local secrets/config
+.env
+.env.*
+secrets.json
+config.local.json
+
+# Python
+__pycache__/
+*.py[cod]
+
+# Temporary/log files
+*.tmp
+*.log

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Loic Leclair
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1 +1,136 @@
-# github-traffic-observer
+# GitHub Traffic Observer
+
+Privacy-first archival and analysis for GitHub's aggregated repository Traffic metrics.
+
+GitHub Traffic Observer keeps a durable history beyond GitHub's rolling window, records late corrections, follows repository renames by stable repository ID, tracks endpoint freshness, and emits simple explainable anomaly signals.
+
+It **does not identify individual visitors**. GitHub's Traffic API exposes aggregate counters and top referrers/pages, not visitor identities, IP addresses, cookies, or user-agent fingerprints.
+
+## Why this project exists
+
+GitHub Traffic metrics are useful but short-lived and can be revised after the fact. This collector stores normalized history so you can answer questions such as:
+
+- how daily clone/view counts changed over time;
+- whether GitHub retroactively corrected a previous day;
+- whether a repository rename fragmented historical reporting;
+- which referrers or popular paths entered or left GitHub's top lists;
+- whether an endpoint is fresh, stale, or temporarily unavailable.
+
+Clones are **not people**, and a unique cloner is not a verified unique human. Treat the output as repository telemetry, not identity tracking.
+
+## Recommended architecture
+
+Keep the engine public and the real data private:
+
+```text
+PUBLIC
+github-traffic-observer
+  ├── scripts/
+  ├── tests/
+  ├── docs/
+  └── action.yml
+        │
+        │ pinned version / commit
+        ▼
+PRIVATE
+your-github-traffic-archive
+  ├── traffic/*.csv
+  ├── traffic/latest.json
+  └── GitHub Actions artifacts (raw gzip snapshots)
+```
+
+The collector refuses, by default, to persist real traffic from GitHub Actions when the archive repository is public or its visibility cannot be verified.
+
+## Features
+
+- standard-library-only Python collector;
+- stable `repository_id` continuity across renames;
+- daily clones, views and GitHub-reported unique counters;
+- explicit late-revision history;
+- top referrer and popular-path changes;
+- repository metadata snapshots;
+- endpoint-level freshness/state;
+- raw gzip snapshots outside Git;
+- anomaly events for revisions, spikes, new referrers and clone concentration;
+- offline regression tests;
+- reusable composite GitHub Action.
+
+## Quick start
+
+Create a **private repository** for the collected data, add a secret named `TRAFFIC_GITHUB_TOKEN`, and use the workflow in `examples/private-archive-workflow.yml` as a starting point.
+
+The caller should pin this project to a trusted tag or commit:
+
+```yaml
+- name: Collect GitHub Traffic
+  uses: loic31000/github-traffic-observer@<trusted-tag-or-commit>
+  env:
+    TRAFFIC_OWNER: your-github-username
+    TRAFFIC_GITHUB_TOKEN: ${{ secrets.TRAFFIC_GITHUB_TOKEN }}
+```
+
+For a fine-grained token, GitHub's Traffic endpoints require repository access compatible with the Traffic API. Grant only the permissions and repository scope you actually need.
+
+## Configuration
+
+| Variable | Purpose | Default |
+|---|---|---|
+| `TRAFFIC_OWNER` | account whose owned repositories are discovered | required |
+| `TRAFFIC_GITHUB_TOKEN` | token used for GitHub API requests | required |
+| `TRAFFIC_DATA_DIR` | normalized archive directory | `traffic` |
+| `TRAFFIC_ARCHIVE_REPOSITORY` | repository excluded from discovery; falls back to `GITHUB_REPOSITORY` | caller repository |
+| `RAW_OUTPUT_DIR` | raw gzip output directory, ideally runner temp storage | `.traffic-raw` |
+| `RAW_RETENTION_DAYS` | local raw cleanup window | `30` |
+| `GITHUB_API_VERSION` | REST API version header | `2026-03-10` |
+| `ALLOW_PUBLIC_TRAFFIC_ARCHIVE` | explicit override of the public-archive safety check | unset / false |
+
+The public-archive override is intentionally opt-in. Publishing real traffic can expose private repository names, activity timing and referrer/path information.
+
+## Data model
+
+The normalized archive can contain:
+
+- `daily.csv`
+- `snapshots.csv`
+- `revisions.csv`
+- `repositories.csv`
+- `repository-metadata.csv`
+- `referrers.csv`
+- `popular-paths.csv`
+- `anomalies.csv`
+- `errors.csv`
+- `latest.json`
+- optional `migration-conflicts.csv`
+
+See [docs/architecture.md](docs/architecture.md) for the model and [docs/privacy.md](docs/privacy.md) for the privacy rules.
+
+## Raw snapshots
+
+Raw Traffic responses are written outside `TRAFFIC_DATA_DIR` to `RAW_OUTPUT_DIR/YYYY-MM-DD/HHMMSS.json.gz`.
+
+The example private workflow uploads that directory as a GitHub Actions artifact with a bounded retention period. This avoids permanently growing Git history with compressed raw payloads.
+
+## Validation
+
+```bash
+python -m py_compile scripts/archive_traffic.py
+python -m unittest discover -s tests -v
+```
+
+The tests are offline and use temporary directories and mocked API responses.
+
+## Privacy principles
+
+This project is intended for aggregate repository observability.
+
+It should not be extended to fingerprint, deanonymize or correlate an individual visitor across services. Do not add tracking pixels, cookies, IP collection, browser fingerprinting, or speculative identity attribution.
+
+See [docs/privacy.md](docs/privacy.md).
+
+## Security
+
+Never commit GitHub tokens or real Traffic archives to this public repository. See [SECURITY.md](SECURITY.md).
+
+## License
+
+MIT. See [LICENSE](LICENSE).

SECURITY.md

@@ -0,0 +1,19 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+Use a private GitHub security-reporting channel when available. Do not post active credentials, private repository names, private Traffic archives or sensitive exploit details in a public issue.
+
+## Secrets
+
+Never commit API credentials, personal access credentials, private archive CSV/JSON files, or raw Traffic payloads from real repositories.
+
+Use GitHub Actions Secrets or another secret manager.
+
+## Token scope
+
+Grant only the repositories and permissions needed to read GitHub Traffic. Rotate a credential if it is ever exposed.
+
+## Data repository
+
+The recommended deployment uses a separate **private** repository for normalized history and raw artifacts. The public collector repository should contain synthetic examples only.

action.yml

@@ -0,0 +1,23 @@
+name: GitHub Traffic Observer
+description: Archive aggregated GitHub Traffic metrics into a caller-controlled data directory.
+author: Loic Leclair
+
+inputs:
+  data-dir:
+    description: Directory where normalized CSV/JSON history is stored.
+    required: false
+    default: traffic
+  raw-output-dir:
+    description: Directory for compressed raw API snapshots. Prefer runner temp storage.
+    required: false
+    default: .traffic-raw
+
+runs:
+  using: composite
+  steps:
+    - name: Collect GitHub Traffic
+      shell: bash
+      env:
+        TRAFFIC_DATA_DIR: ${{ inputs.data-dir }}
+        RAW_OUTPUT_DIR: ${{ inputs.raw-output-dir }}
+      run: python "${{ github.action_path }}/scripts/archive_traffic.py"

docs/architecture.md

@@ -0,0 +1,42 @@
+# Architecture
+
+GitHub Traffic Observer separates **collector code** from **collected data**.
+
+## Components
+
+1. **Collector** — `scripts/archive_traffic.py`, public and reusable.
+2. **Caller workflow** — runs from a private archive repository.
+3. **Normalized archive** — CSV/JSON files stored in the private repository.
+4. **Raw snapshots** — gzip payloads stored outside Git and optionally uploaded as short-retention Actions artifacts.
+
+## Identity
+
+Repositories are tracked primarily by GitHub's stable numeric `repository_id`. Names are retained as observations so a rename does not split the historical series.
+
+## Endpoint isolation
+
+Clones, views, referrers, popular paths and metadata are fetched independently. A failed endpoint does not become a false zero. The last successful value remains in state, while the current snapshot records the endpoint as unavailable/stale.
+
+## Late revisions
+
+GitHub may revise an earlier day. The collector re-reads the returned daily window, updates the normalized value and writes an explicit revision event for completed days.
+
+## Ranked data
+
+Referrers and popular paths are top-list snapshots, not visit logs. Changes are recorded as `new`, `existing` or `dropped_from_top10`.
+
+## Anomalies
+
+Signals are deliberately simple and explainable:
+
+- `late_revision`
+- `repository_renamed`
+- `new_referrer`
+- `clone_concentration`
+- `traffic_spike`
+
+They are telemetry hints, not conclusions about a human visitor or malicious actor.
+
+## Storage boundary
+
+The public repository must contain only code, documentation, tests and synthetic examples. Real data belongs in a private archive repository.

docs/privacy.md

@@ -0,0 +1,29 @@
+# Privacy
+
+GitHub Traffic Observer is designed for **aggregate repository observability**, not person-level tracking.
+
+## What GitHub Traffic provides
+
+Depending on the endpoint, GitHub exposes aggregate counts, GitHub-reported unique counts, top referrers and top popular paths.
+
+The collector does not receive a visitor's GitHub account identity, IP address, browser fingerprint, cookies, personal profile, or exact per-person browsing history.
+
+## Rules for this project
+
+1. **Private by default.** Store real collected data in a private repository.
+2. **No deanonymization.** Do not infer that a specific named person generated a visit or clone.
+3. **No cross-service identity correlation.** A social referrer and a GitHub visit do not prove they came from the same person.
+4. **No fingerprinting.** Do not add tracking pixels, cookies, IP logging or browser/device fingerprinting.
+5. **Aggregate language.** Use terms such as "GitHub-reported unique visitor" rather than "person" when interpreting metrics.
+6. **Bound raw retention.** Prefer temporary Actions artifacts or external storage with explicit deletion/retention controls.
+7. **Minimize secrets.** Keep credentials in GitHub Actions Secrets or another secret manager and grant the narrowest practical scope.
+
+## Public-repository safety check
+
+When running in GitHub Actions, the collector checks the caller archive repository. Unless `ALLOW_PUBLIC_TRAFFIC_ARCHIVE=true` is explicitly set, it refuses to continue if the archive is public or its visibility cannot be verified.
+
+The override exists for advanced users who intentionally accept publication of the data. It is not recommended.
+
+## Interpreting metrics
+
+Clones may be generated by humans, CI systems, agents, indexers or other Git clients. GitHub-reported unique counters are not proof of distinct real-world people. Referrers and popular paths are incomplete top-list data and may refresh on a different cadence from clones/views.

examples/anomalies.example.csv

@@ -0,0 +1,2 @@
+detected_at,repository_id,repository,type,severity,date_utc,metric,value,baseline,details,signature
+2026-01-03T09:17:00Z,1001,example-owner/sample-api,late_revision,info,2026-01-02,views,10,8,GitHub revised views from 8 to 10,1001|late_revision|2026-01-02|views|8|10

examples/daily.example.csv

@@ -0,0 +1,3 @@
+date,repository_id,repository,clones,unique_cloners,views,unique_visitors
+2026-01-01,1001,example-owner/sample-api,4,3,8,5
+2026-01-02,1001,example-owner/sample-api,6,4,10,6

examples/latest.example.json

@@ -0,0 +1,19 @@
+{
+  "schema_version": 2,
+  "snapshot_utc": "2026-01-03T09:17:00Z",
+  "owner": "example-owner",
+  "repositories": {
+    "example-owner/sample-api": {
+      "repository_id": 1001,
+      "repository_name": "example-owner/sample-api",
+      "clones_14d": 10,
+      "unique_cloners_14d": 7,
+      "views_14d": 18,
+      "unique_visitors_14d": 9,
+      "endpoints": {
+        "clones": {"status": "ok", "stale": false, "last_attempt_utc": "2026-01-03T09:17:00Z", "last_success_utc": "2026-01-03T09:17:00Z"},
+        "views": {"status": "ok", "stale": false, "last_attempt_utc": "2026-01-03T09:17:00Z", "last_success_utc": "2026-01-03T09:17:00Z"}
+      }
+    }
+  }
+}