Repository navigation
feat: add bids source for 12 municipalities - #5
Merged
Merged
Conversation
…nger fails the search
…ind filter, --kind)
… flowchart) with titles and descriptive names
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds tenders from 12 more cities: Akko, Arad, Beit Shemesh, Dimona, Kfar Saba, Kiryat Motzkin, Nesher, Netivot, Nof HaGalil, Savyon, Tamra and Yavne. They publish on the same kind of page, so one parser reads them all, and each city is a short class with just a name, a publisher and a domain. To keep every source uniform I also reworked the base: a source now only says how to read its records and how to turn one into a tender, and the shared
fetch()skips and reports bad records and drops repeats. RMI, Beer Sheva, the government portal and the cities all work this way, they are listed in one registry (BUILT_IN), and a normal search, the CLI choices andSourceNameall follow it. Four of these cities list staff vacancies in the same tables as their tenders, so every listing now has akind(tender or vacancy). Vacancies are hidden by default, and you can ask for them withkind=TenderKind.VACANCY, for both withkind=None, or with--kind tender|vacancy|allon the command line. That split is a guess based on the table caption and title, so an odd one can slip through. If one source is down the search keeps going and notes the error inlast_errors; it only fails when every source fails. I ran it against the live sites today and got 678 open tenders from 15 sources with no errors. Dates on these pages have no hour, so a deadline is treated as the end of that day, and Azor is left out because its robots.txt blocks everything.