Skip to content

feat: add gov source for central-government tenders (mr.gov.il) - #4

Merged
Zikithezikit merged 2 commits into
mainfrom
more-sources
Nov 5, 2026
Merged

Zikithezikit merged 2 commits into
mainfrom
more-sources

Conversation

@Zikithezikit

Copy link
Copy Markdown
Owner

What this adds

A gov source for central-government tenders (ministries and other bodies), read from the Government
Procurement Administration portal, mr.gov.il. The portal has no JSON API, so the source parses its server-rendered
search page. It is part of the default sources.

  • GovSource and GovExtra (public), SourceName.GOV.
  • HttpClient.get_text: same rate limit, cache, retries and error handling as the JSON methods, for HTML pages.
  • Only tenders with status published, updated or new are read (the portal lists 8 835 in total, 7 654 of them
    past their submission date).
  • docs/sources.md, docs/model.md, README and CHANGELOG are updated.

No city council sources are in this PR. data.gov.il has no live municipal feed besides Beer Sheva, and the other
city sites I probed each need their own parser. They can follow as separate sources.

Verified (how)

  • Live run against mr.gov.il on 2026-11-05: 419 tenders in about 21 s, 0 parse errors. The per-status counts
    (123 + 256 + 40) match the counts the portal reports itself. 183 are still open by deadline, 14 have no deadline.
  • make check passes: ruff, mypy --strict, 193 tests. Coverage 99% overall, 100% for gov.py.
  • Tests run offline against a fixture of 5 real result blocks. The embedded images were removed and the count
    sentence was edited to match the excerpt (noted in docs/sources.md).

Believed, not verified

  • The portal's license for reuse was not checked. The site has no robots.txt (empty response).
  • Statuses "being updated" (32) and "future" (15) are not read, because it is unclear whether they take offers.
  • Detail pages and attached documents are not fetched, so documents is empty.
  • Default Michraz() searches now include about 25 extra requests on a cold cache (around 20 s). Cached for an
    hour after that.

Tests

  • tests/test_gov.py: 12 unit tests (parsing, dates, dedupe across statuses, paging, skipped records, bad date,
    non-search page, HTTP 503).
  • tests/test_e2e.py: 3 new end-to-end tests. All three default sources feed one search and one CSV, gov keyword and
    closing-window filters, and the CLI printing gov tenders as JSON.
  • tests/conftest.py now fixes "now" to the recording date. Before this, tests that use open_only against the
    recorded fixtures would have started failing once the fixture deadlines passed (around 2026-11-10).
  • Sleeping in the rate limiter is skipped in tests (suite went from 26 s back to 1.5 s).

make check

tests

Real output from the live portal:

live output

@Zikithezikit
Zikithezikit merged commit 3dad784 into main Nov 5, 2026
6 checks passed
@Zikithezikit
Zikithezikit deleted the more-sources branch November 5, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant