Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,35 @@ LOG_LEVEL=INFO
# Optional Google Sheets sync (free service account + Sheets API):
# pip install -e ".[sheets]"
# GOOGLE_SHEET_ID= # from docs.google.com/spreadsheets/d/<ID>/edit
# GOOGLE_SHEET_URL= # optional full share/deep link for Teams/Slack cards (else built from ID)
# GOOGLE_SHEET_TAB=Ingest
# GOOGLE_SHEET_RANK_TAB=Ranked # Grok interpret-rank full-replace tab (never Ingest)
# GOOGLE_SERVICE_ACCOUNT_FILE=secrets/google-service-account.json
# # or inline JSON (e.g. CI secret):
# GOOGLE_SERVICE_ACCOUNT_JSON=

# Optional Grok interpret-rank (xAI API; does not mutate ingest rows):
# pip install -e ".[ai]"
# XAI_API_KEY=
# XAI_BASE_URL=https://api.x.ai/v1
# XAI_MODEL=grok-4.5
# OBJECTIVES_PATH=config/objectives.yaml

# Teams Workflows webhooks (ops alerts + high-match capture pings):
# TEAMS_WEBHOOK_URL= # hard fail / streak / partial (ops)
# TEAMS_MATCH_WEBHOOK_URL= # optional dedicated capture channel; else ops URL
# TEAMS_MATCH_NOTIFY_ENABLED=true
# TEAMS_MATCH_SCORE_THRESHOLD=40 # 0–100; RelevanceScore (ingest) or Grok fit
# TEAMS_MATCH_MAX_ITEMS=8
# NOTIFY_CONFIG_PATH=config/notify.yaml

# Slack match pings — prefer Slack CLI / Bolt tokens (https://docs.slack.dev/tools/slack-cli/):
# slack login → create/install app with chat:write → paste:
# SLACK_BOT_TOKEN=xoxb-... # Bot User OAuth Token (required for Web API)
# SLACK_CHANNEL_ID=C... # or #channel-name
# SLACK_APP_TOKEN=xapp-... # optional; Socket Mode / slack run only
# SLACK_MATCH_NOTIFY_ENABLED=true
# Setup guide: scripts/slack_cli_setup.md
# Legacy Incoming Webhook fallback (discouraged):
# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
# SLACK_MATCH_WEBHOOK_URL=
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ build/
# Local data / state / logs (keep directory placeholders)
data/*.db
data/*.csv
data/rankings/
data/atlas_crawl/
!data/.gitkeep
logs/*
!logs/.gitkeep
Expand Down
45 changes: 30 additions & 15 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
|-------|--------|
| **Project** | Contract Bidding System — Phase 1 opportunity ingest |
| **Package** | `opportunity_ingest` (`src/opportunity_ingest/`) |
| **As-of** | 2026-08-10 (as-built; Google Sheets sync included) |
| **As-of** | 2026-08-11 (as-built; Sheets sync + optional Grok interpret-rank) |
| **Primary language** | Python 3.11+ |
| **Default store** | SQLite (`STORAGE_BACKEND=sqlite`) |

Expand All @@ -24,8 +24,9 @@ Daily (or on-demand) pipeline:
6. Dedupe by `OpportunityID` + normalized `Link`.
7. **Create-only** writes to SQLite (default) or SharePoint (optional).
8. Optional: export CSV, **full-replace sync to Google Sheets `Ingest` tab**, Teams alerts.
9. Optional post-ingest: **Grok interpret-rank** — plain-English rewrite + fit ranking vs `config/objectives.yaml` (report only).

**SQLite is the system of record for day-1.** Google Sheets is a **view** (overwritten on sync). SharePoint is a **pluggable store**, not required for go-live.
**SQLite is the system of record for day-1.** Google Sheets is a **view** (overwritten on sync). SharePoint is a **pluggable store**, not required for go-live. Grok ranking writes reports under `data/rankings/`; it does **not** mutate store Status/Notes/RelevanceScore.

---

Expand All @@ -35,14 +36,18 @@ Daily (or on-demand) pipeline:
|----------|------|---------|
| 1 | **This file (`AGENTS.md`)** | Non-negotiable rules + orientation |
| 2 | [`docs/STATUS.md`](docs/STATUS.md) | Living readiness snapshot (what works / next) |
| 3 | [`docs/DATA_UPDATE_DIRECTIVES.md`](docs/DATA_UPDATE_DIRECTIVES.md) | **MUST / MUST NOT** for data writes, schema, sync |
| 4 | [`docs/AS_BUILT.md`](docs/AS_BUILT.md) | Current architecture, modules, CLI, env |
| 5 | [`docs/CHANGE_PLAYBOOK.md`](docs/CHANGE_PLAYBOOK.md) | How to implement common change types |
| 6 | [`docs/INDEX.md`](docs/INDEX.md) | Full documentation index |
| 7 | [`docs/phase1-canadabuys-sharepoint-implementation-schema.md`](docs/phase1-canadabuys-sharepoint-implementation-schema.md) | Original design (rev 4); prefer AS_BUILT if conflict |
| 8 | [`scripts/ops_runbook.md`](scripts/ops_runbook.md) | Human operator procedures |
| 9 | [`scripts/google_sheets_setup.md`](scripts/google_sheets_setup.md) | Sheets service-account setup |
| 10 | [`scripts/provision_sharepoint_list.md`](scripts/provision_sharepoint_list.md) | SharePoint activation (deferred) |
| 3 | [`docs/PLUG_AND_PLAY.md`](docs/PLUG_AND_PLAY.md) | Human keys + plug-and-play steps for built features |
| 3b | [`docs/PROCESSOR_OVERVIEW.md`](docs/PROCESSOR_OVERVIEW.md) | Plain language: this app as one processor — triggers, I/O, system place |
| 4 | [`docs/DATA_UPDATE_DIRECTIVES.md`](docs/DATA_UPDATE_DIRECTIVES.md) | **MUST / MUST NOT** for data writes, schema, sync |
| 5 | [`docs/AS_BUILT.md`](docs/AS_BUILT.md) | Current architecture, modules, CLI, env |
| 6 | [`docs/BACKLOG.md`](docs/BACKLOG.md) | Roadmap / discussed but **not built** |
| 7 | [`docs/CHANGE_PLAYBOOK.md`](docs/CHANGE_PLAYBOOK.md) | How to implement common change types |
| 8 | [`docs/INDEX.md`](docs/INDEX.md) | Full documentation index |
| 9 | [`docs/DECISIONS.md`](docs/DECISIONS.md) | ADR-style as-built decisions |
| 10 | [`docs/phase1-canadabuys-sharepoint-implementation-schema.md`](docs/phase1-canadabuys-sharepoint-implementation-schema.md) | Original design (rev 4); prefer AS_BUILT if conflict |
| 11 | [`scripts/ops_runbook.md`](scripts/ops_runbook.md) | Human operator procedures |
| 12 | [`scripts/google_sheets_setup.md`](scripts/google_sheets_setup.md) | Sheets service-account setup |
| 13 | [`scripts/provision_sharepoint_list.md`](scripts/provision_sharepoint_list.md) | SharePoint activation (deferred) |

---

Expand All @@ -56,8 +61,9 @@ Daily (or on-demand) pipeline:
6. **CanadaBuys download requires browser-like User-Agent** (CDN returns 403 otherwise). See `download.py` `DEFAULT_HEADERS`.
7. **Google Sheets `Ingest` tab is full-replace.** Manual edits on that tab are wiped next `sync-sheets`. Human work → separate tab (e.g. `Review`).
8. **Keywords:** engineering owns `config/keywords.yaml`. Ops request changes via issues.
9. **Secrets:** never commit `.env`, `secrets/`, service-account JSON, Azure secrets.
10. **Phase 1 non-goals:** multi-source (MERX/etc.), AI ranking, auto-bidding, historical backfill, dual-write SQLite+SharePoint, two-way Sheets Status sync.
9. **Secrets:** never commit `.env`, `secrets/`, service-account JSON, Azure secrets, `XAI_API_KEY`.
10. **Phase 1 non-goals:** multi-source (MERX/etc.), auto-bidding, historical backfill, dual-write SQLite+SharePoint, two-way Sheets Status sync.
11. **Grok interpret-rank is post-ingest only.** Never write AI fit scores into `contract_opportunities` Status/Notes/RelevanceScore. Objectives live in `config/objectives.yaml` (eng-owned).

---

Expand All @@ -75,15 +81,18 @@ src/opportunity_ingest/
models.py # TenderRecord, OpportunityFields, ExistingKeys
config.py # pydantic-settings
state.py # zero-new streak JSON
notify.py # Teams Workflows Adaptive Card
notify.py # Teams + Slack match pings (score≥threshold); Teams ops alerts
exit_codes.py # exit + notify matrix
sheets_sync.py # SQLite → Google Sheets full tab replace
interpret_rank.py # Grok rephrase + fit rank (report only)
storage/
base.py # OpportunityStore Protocol, normalize_link
factory.py # sqlite | sharepoint
sqlite_store.py # default backend + export_csv
sharepoint_store.py # Graph adapter (optional)
config/keywords.yaml
config/objectives.yaml # company objectives for interpret-rank
config/notify.yaml # Teams match threshold / card limits
.github/workflows/ci.yml
.github/workflows/daily-canadabuys-ingest.yml
```
Expand All @@ -98,9 +107,11 @@ python -m opportunity_ingest download-sample [--out PATH]
python -m opportunity_ingest check-store
python -m opportunity_ingest export-csv [--out PATH]
python -m opportunity_ingest sync-sheets [--sheet-id ID] [--tab NAME]
python -m opportunity_ingest interpret-rank [--status STATUS] [--limit N] [--sync-sheets|--no-sync-sheets] [--rank-tab NAME]
python -m opportunity_ingest sync-rank-sheets [--from-json PATH] [--rank-tab NAME]
```

Optional install extras: `pip install -e ".[sheets]"` for Google Sheets; SharePoint uses core deps (`msal`, `httpx`).
Optional install extras: `pip install -e ".[sheets]"` for Google Sheets; `pip install -e ".[ai]"` for Grok interpret-rank; SharePoint uses core deps (`msal`, `httpx`). Grok rankings sync to tab **`Ranked`** (never `Ingest`).

---

Expand All @@ -115,7 +126,9 @@ Optional install extras: `pip install -e ".[sheets]"` for Google Sheets; SharePo
| Store schema | `sqlite_store.py` DDL + `models.OpportunityFields` + design schema |
| Pipeline order / metrics | `pipeline.py` |
| Exit / notify policy | `exit_codes.py`, `notify.py` |
| Sheets sync | `sheets_sync.py`, `cli.py` `sync-sheets` |
| Sheets sync (opportunities) | `sheets_sync.py`, `cli.py` `sync-sheets` → tab `Ingest` |
| Sheets sync (Grok ranks) | `sheets_sync.py` `sync_rankings_to_sheet`, `interpret-rank --sync-sheets`, `sync-rank-sheets` → tab `Ranked` |
| Grok interpret / rank | `interpret_rank.py`, `config/objectives.yaml`, `cli.py` `interpret-rank` |
| SharePoint | `storage/sharepoint_store.py`, `scripts/provision_sharepoint_list.md` |
| Daily schedule | `.github/workflows/daily-canadabuys-ingest.yml` |
| Env vars | `config.py`, `.env.example`, `config/settings.example.env` |
Expand All @@ -128,6 +141,8 @@ Optional install extras: `pip install -e ".[sheets]"` for Google Sheets; SharePo
|------|------|
| `data/contract_opportunities.db` | SQLite system of record |
| `data/sample-openTenderNotice.csv` | Downloaded open-tender sample |
| `data/rankings/interpret-*.md` | Grok ranked brief (human) |
| `data/rankings/interpret-*.json` | Grok ranked brief (machine) |
| `data/export-opportunities.csv` | Human CSV export |
| `state/zero_new_streak.json` | Zero-new calendar streak |
| `logs/run-*.json` | Per-run metrics |
Expand Down
47 changes: 41 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,22 @@ Phase 1 pipeline: download CanadaBuys **Open Tender Notices** CSV, filter by key
| **Optional store** | SharePoint via Microsoft Graph (`STORAGE_BACKEND=sharepoint`) |
| **Optional view** | Google Sheets tab full-replace (`sync-sheets`) |
| **LLM / agent entry** | **[`AGENTS.md`](AGENTS.md)** ← start here for AI-assisted development |
| **Human plug-in** | **[`docs/PLUG_AND_PLAY.md`](docs/PLUG_AND_PLAY.md)** ← keys, steps, roadmap to go live |
| **Processor overview** | **[`docs/PROCESSOR_OVERVIEW.md`](docs/PROCESSOR_OVERVIEW.md)** ← role, triggers, I/O in the wider system |

---

## Documentation structure

| Doc | Audience | Purpose |
|-----|----------|---------|
| [`docs/PROCESSOR_OVERVIEW.md`](docs/PROCESSOR_OVERVIEW.md) | **Everyone** | This app as **one key processor** — role, triggers, outputs |
| [`docs/PLUG_AND_PLAY.md`](docs/PLUG_AND_PLAY.md) | **Operators / humans** | **Keys to plug, function list, go-live sequence, roadmap** |
| [`AGENTS.md`](AGENTS.md) | LLMs + devs | Hard rules, module map, task routing |
| [`docs/STATUS.md`](docs/STATUS.md) | Everyone | Living readiness / what works now |
| [`docs/INDEX.md`](docs/INDEX.md) | Everyone | Full doc index |
| [`docs/AS_BUILT.md`](docs/AS_BUILT.md) | Devs / LLMs | Implemented architecture |
| [`docs/BACKLOG.md`](docs/BACKLOG.md) | Everyone | Roadmap / not built yet |
| [`docs/DATA_UPDATE_DIRECTIVES.md`](docs/DATA_UPDATE_DIRECTIVES.md) | Devs / LLMs | **MUST / MUST NOT** for data writes & sync |
| [`docs/CHANGE_PLAYBOOK.md`](docs/CHANGE_PLAYBOOK.md) | Devs / LLMs | How to make common changes |
| [`docs/DECISIONS.md`](docs/DECISIONS.md) | Devs / LLMs | Key ADRs |
Expand All @@ -42,8 +47,11 @@ If design doc and as-built conflict → **prefer code + AS_BUILT + DATA_UPDATE_D

## Quick start (Windows)

**Full human checklist (keys, Sheets, Grok, Teams, roadmap):**
→ **[`docs/PLUG_AND_PLAY.md`](docs/PLUG_AND_PLAY.md)**

```powershell
cd "C:\Users\Vikrant\Documents\Agentic AI Learning\contractbiddingsystem"
cd <repository-root>
python -m venv .venv
.\.venv\Scripts\Activate.ps1

Expand Down Expand Up @@ -71,7 +79,23 @@ pip install -e ".[sheets]"
python -m opportunity_ingest sync-sheets
```

**Notes learned in setup:**
### Grok interpret-rank (optional AI brief)

Plain-English rewrite + rank against `config/objectives.yaml` (does **not** change SQLite rows).
When `GOOGLE_SHEET_ID` is set (your live sheet), rankings are **full-replaced** into a **`Ranked`** tab (separate from `Ingest`):

```powershell
pip install -e ".[ai]"
pip install -e ".[sheets]" # same service account as sync-sheets
# Set XAI_API_KEY + existing GOOGLE_* in root .env
python -m opportunity_ingest interpret-rank --status New --limit 20
# Local: data/rankings/interpret-*.md
# Sheet: tab Ranked (created if missing)
# Skip sheet push: --no-sync-sheets
# Re-push last report without Grok: python -m opportunity_ingest sync-rank-sheets
```

**Setup notes:**

- `pip install -e .` needs the trailing `.`
- CanadaBuys may return **403** without a browser User-Agent (handled in `download.py`)
Expand All @@ -88,14 +112,18 @@ python -m opportunity_ingest download-sample [--out PATH]
python -m opportunity_ingest check-store
python -m opportunity_ingest export-csv [--out PATH]
python -m opportunity_ingest sync-sheets [--sheet-id ID] [--tab NAME]
python -m opportunity_ingest interpret-rank [--status STATUS] [--limit N] [--sync-sheets|--no-sync-sheets]
python -m opportunity_ingest sync-rank-sheets [--from-json PATH] [--rank-tab NAME]
```

| Command | Persist store? | Notes |
|---------|----------------|--------|
| `run` | Only with `--write` | Default dry-run |
| `export-csv` | No | Snapshot for Excel |
| `sync-sheets` | No SQLite write | **Full-replace** Sheet tab |
| `sync-sheets` | No SQLite write | **Full-replace** Sheet tab `Ingest` |
| `check-store` | No | Health + key count |
| `interpret-rank` | No | Grok rewrite + rank → `data/rankings/` + optional Sheets **Ranked** |
| `sync-rank-sheets` | No | Re-push ranking JSON → Sheets **Ranked** (no Grok call) |

---

Expand All @@ -107,15 +135,22 @@ python -m opportunity_ingest sync-sheets [--sheet-id ID] [--tab NAME]
| `DATA_DIR` | `data` | DB + exports |
| `MAX_CREATE` | `50` | Create-**attempt** budget; `0`=unlimited |
| `GOOGLE_SHEET_ID` | — | Sheets sync |
| `GOOGLE_SHEET_TAB` | `Ingest` | Tab overwritten on sync |
| `GOOGLE_SHEET_TAB` | `Ingest` | Opportunity tab overwritten on sync |
| `GOOGLE_SHEET_RANK_TAB` | `Ranked` | Grok ranking tab overwritten on interpret-rank |
| `GOOGLE_SERVICE_ACCOUNT_FILE` | — | Path to JSON key |
| `TEAMS_WEBHOOK_URL` | — | Failure / streak alerts |
| `TEAMS_WEBHOOK_URL` | — | Ops alerts + match fallback |
| `TEAMS_MATCH_WEBHOOK_URL` | — | Optional capture-channel webhook |
| `TEAMS_MATCH_SCORE_THRESHOLD` | `40` | Ping when score ≥ N |
| `XAI_API_KEY` | — | Grok interpret-rank |
| `OBJECTIVES_PATH` | `config/objectives.yaml` | AI ranking frame of reference |

Full list: `.env.example`, `config/settings.example.env`, `docs/AS_BUILT.md`.
Full list: `.env.example`, `config/settings.example.env`, [`docs/PLUG_AND_PLAY.md`](docs/PLUG_AND_PLAY.md), `docs/AS_BUILT.md`.

**Data rules (critical):** create-only; never silent-truncate Link; Sheets is a view.
→ [`docs/DATA_UPDATE_DIRECTIVES.md`](docs/DATA_UPDATE_DIRECTIVES.md)

**What exists vs not built:** [`docs/STATUS.md`](docs/STATUS.md) · [`docs/AS_BUILT.md`](docs/AS_BUILT.md) · [`docs/BACKLOG.md`](docs/BACKLOG.md)

---

## Development
Expand Down
Loading
Loading