Add government budget data sources: IMF SDMX and Togo revenueRevenue#28
Open
yukinko-iwasaki wants to merge 32 commits into
Open
Add government budget data sources: IMF SDMX and Togo revenueRevenue#28yukinko-iwasaki wants to merge 32 commits into
yukinko-iwasaki wants to merge 32 commits into
Conversation
…ture Renames budget/revenue.py to budget/government_budget.py and rewrites it to pull revenue and expenditure (national currency, XDC) from IMF's GFS_SOO (GFS Statement of Operations) dataflow via the SDMX v3 API, filtered to Budgetary Central Government (S1311B) and chunked by country. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both WEO (General Government, IMF.RES:WEO) and GFS_SOO (Budgetary Central Government, IMF.STA:GFS_SOO) now flow through a single chunked SDMX-JSON v3 fetcher driven by a SOURCES config list. The government_budget table gets one row per (country, year, data_source). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames country/revenue/expenditure/source to country_name/revenue_current_lcu/expenditure_current_lcu/data_source so the Togo PDF-extracted table can later be unioned with the IMF-sourced government_budget table without further reshaping. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PDF cells come through as French-formatted strings like '123 123, 00'. Adds a parse_amount() helper that strips whitespace and converts the comma decimal separator before casting to float, returning None on empty or unparseable input. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds government budget data ingestion from IMF SDMX APIs and Togo budget execution PDFs, while broadening World Bank indicator merges to preserve partial indicator coverage.
Changes:
- Adds IMF WEO/GFS SDMX extraction and writes
government_budget. - Adds Togo PDF download and revenue/expenditure extraction notebooks.
- Changes
wbgapi_fetchindicator merging from inner to outer joins.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
utils.py |
Preserves country-year rows when some World Bank indicators are missing. |
budget/government_budget.py |
Fetches and reshapes IMF SDMX government revenue/expenditure data. |
budget/togo/togo_budget_documents_download.py |
Downloads Togo budget PDFs into a Databricks volume. |
budget/togo/togo_revenue.py |
Parses Togo budget PDFs and saves extracted budget metrics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.