Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
28cd6fc
created training data dataset, added examples for pulling them and im…
Mijan Oct 27, 2025
ba7181d
wip: snapshot in-flight reorg before notebook-driven refactor
Mijan May 8, 2026
3387c09
Refactor warehouse_ops.io: typed config + table-schema registry
Mijan May 8, 2026
4279eb9
warehouse_ops population refactored
Mijan May 8, 2026
d92ccc4
warehouse tests added
Mijan May 8, 2026
73216eb
removed api client example
Mijan May 8, 2026
8e8baab
first warehouse population notebook added
Mijan May 8, 2026
7c97551
fixed bugs in loader coad and modified Enum
Mijan May 8, 2026
7785f35
added warehouse migrations scripts, readme and setup
Mijan May 8, 2026
e7f50f8
fixed variable typos
Mijan May 8, 2026
3292f3b
additional warehouse migrations performed
Mijan May 8, 2026
e7853ad
added timeout and logging to nasa and cams
Mijan May 8, 2026
8b6b0cc
ground station nasa cams ingestion step added
Mijan May 8, 2026
e072bf1
added fix for geolocation query bug
Mijan May 8, 2026
825ffb3
additional sanity tests added
Mijan May 8, 2026
ed74a33
loaders updated
Mijan May 8, 2026
bbd3f2b
adding merra 2 code
Mijan May 8, 2026
6b0d7a2
adding merra 2 code
Mijan May 8, 2026
d32e856
adding merra schema
Mijan May 8, 2026
be8adc4
dropped external relics
Mijan May 8, 2026
cb94e7c
fixed scaling for cams rad satelite data
Mijan May 8, 2026
1456a98
solar zenith angle removed
Mijan May 8, 2026
b77a944
a9 and a10 applied
Mijan May 8, 2026
12c47cb
minor bug fixed in features_service
Mijan May 8, 2026
565a89c
finally a useful version of the notebook
Mijan May 8, 2026
4c91528
merra data ingestion
Mijan May 8, 2026
1b68329
Readme updated
Mijan May 8, 2026
8c281f4
parallel merra fetcher
Mijan May 8, 2026
8cf9a70
added various fixes for merra 2
Mijan May 8, 2026
78fe3c1
next little bug fixed
Mijan May 8, 2026
27adf11
udpate of notebook
Mijan May 8, 2026
27f653a
removed dead code, massive cleanup
Mijan May 8, 2026
36775ce
updated modis code
Mijan May 8, 2026
136f595
c3 worked
Mijan May 8, 2026
b7b8c86
c7 is running
Mijan May 8, 2026
77b56d3
c8 is running
Mijan May 8, 2026
80d9a09
added datasets and dataset creation
Mijan May 8, 2026
5c3a44a
added everythong for preprocessing
Mijan May 8, 2026
43794f4
notebooks updated
Mijan May 8, 2026
aff3d08
initial madoel formulation including tests
Mijan May 8, 2026
0be7ff5
tests added
Mijan May 8, 2026
fe1786d
modeling notebook added
Mijan May 8, 2026
26a1694
all notebooks updated
Mijan May 9, 2026
c5d85fa
dead code removed
Mijan May 9, 2026
6f22423
additional descriptions added to notebooks
Mijan May 9, 2026
6298259
everything up to notebook 5 ready (for now)
Mijan May 9, 2026
cad9e1d
training added
Mijan May 10, 2026
b28e0eb
data folder update and minor restructuring
Mijan May 10, 2026
afdac9a
notebooks updated and readme added
Mijan May 10, 2026
3cd0960
refactored to include general derived features
Mijan May 10, 2026
4ac666d
udpated notebook files to match code changes
Mijan May 10, 2026
c06740e
updated merra datafetcher to fetch region span, speedup ~100x
Mijan May 10, 2026
fc9b3d2
adding features to recompute results for IEEE paper
Mijan May 10, 2026
df92068
added cleaning functionallity to clean data as necessary
Mijan May 10, 2026
9f8742a
some warehouse issues fixed (variable physical location added
Mijan May 10, 2026
cf6fa82
updated everything for paper notebook!
Mijan May 10, 2026
1fd210e
notebooks updated
Mijan May 11, 2026
5da54bf
final update and cleanup before pre-final commit
Mijan May 11, 2026
76e82a1
deleted unused data, added data imputation fixed readmes, added 28 gr…
Mijan May 11, 2026
23ad3df
added data visualization notebook
Mijan May 11, 2026
6f802f0
updated all notebooks and readmes
Mijan May 11, 2026
a351f88
formats and mypy fixed
Mijan May 11, 2026
46f6bed
katongole ingestion script added
Mijan May 11, 2026
55a1bda
plots for katongole data updated
Mijan May 11, 2026
9a3ae99
moved katongole data to external folder
Mijan May 12, 2026
a872855
added evaluation package and notebook (fixed previous notebooks to st…
Mijan May 12, 2026
dd0927c
fixed the plots
Mijan May 12, 2026
31ce4cd
formats updated, auxscripts updated and notebooks reran
Mijan May 12, 2026
5399823
typing fixed
Mijan May 12, 2026
239bcfd
inference notebook added and plots updated. predictor class for simpl…
Mijan May 13, 2026
e11aa77
cleanup and duplicated code removal
Mijan May 13, 2026
134ba78
.env credentials added
Mijan May 13, 2026
696c43b
clean upas, unified satellite-feature assembly and other minor cleanups
Mijan May 13, 2026
737298c
clean upas, unified satellite-feature assembly and other minor cleanups
Mijan May 13, 2026
0ac28ea
added schema and remove dhard coded dataframe columns
Mijan May 13, 2026
90f2248
track notebook build scripts as source-of-truth for tutorial chain
Mijan May 13, 2026
f2273f6
formats fixed
Mijan May 13, 2026
f059289
notebooks updated
Mijan May 13, 2026
2f39751
udpated docs and readme
Mijan May 13, 2026
9dae482
Remove CrossBoundary ground-measurement data from public repo
Mijan May 13, 2026
01034d6
updated notebooks and readme
Mijan May 13, 2026
316e6c8
fixed NASA ingestions and snapping to grid
Mijan May 18, 2026
99c079c
feature test added
Mijan May 18, 2026
2ffacfb
satelite loading added
Mijan May 18, 2026
0413c0e
satellite loading refactored
Mijan May 18, 2026
d9c4837
minor cleanup and fix
Mijan May 18, 2026
11aa615
Add human-facing metadata to DerivedFeature subclasses
Mijan May 19, 2026
913bf80
Add Predictor.feature_catalog for model-input metadata
Mijan May 19, 2026
06779df
added unit docstrings
Mijan May 20, 2026
7d58d25
Addding updated notebook
rogersmukiibi Jun 3, 2026
09eb7bb
updated notebooks
Mijan Jun 10, 2026
33cf692
Adding implementation notebook
rogersmukiibi Jun 10, 2026
61edeb3
Created using Colab
rogersmukiibi Jun 12, 2026
d969e46
updating notebook
rogersmukiibi Jun 12, 2026
86bd290
Merge branch 'jm/add_model' of https://github.com/Marconi-Lab/Solar_i…
rogersmukiibi Jun 12, 2026
ea57732
Reolving merge conflict
rogersmukiibi Jun 12, 2026
cc39129
Merge pull request #20 from Marconi-Lab/rm/repo_navigation
rogersmukiibi Jun 12, 2026
df54ae3
reformating with tox
rogersmukiibi Jun 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 41 additions & 60 deletions .github/workflows/tox.yml
Original file line number Diff line number Diff line change
@@ -1,90 +1,71 @@
name: Tox CI
name: CI

on:
push:
branches:
- main
branches: [main]
pull_request:
branches:
- main
branches: [main]

jobs:
tox:
test:
name: pytest (py${{ matrix.python-version }})
runs-on: ubuntu-latest

strategy:
matrix:
python-version: [3.12] # Specify the Python versions to test against, as per your tox envlist

python-version: ["3.12"]
steps:
# Step 1: Check out the repository
- name: Check out the repository
uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0 # setuptools_scm needs full history for versioning

# Step 2: Set up Python
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip

# Step 3: Install dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install tox

# Step 4: Run Tox
- name: Run Tox
env:
NASA_USERNAME: ${{ secrets.NASA_USERNAME }}
NASA_PASSWORD: ${{ secrets.NASA_PASSWORD }}
run: tox
- name: Install tox
run: python -m pip install --upgrade pip tox

# Add this to your tox job in tox.yml
- name: Run Tests with Coverage
- name: Run pytest
env:
NASA_USERNAME: ${{ secrets.NASA_USERNAME }}
NASA_PASSWORD: ${{ secrets.NASA_PASSWORD }}
run: tox -e py312
EARTHDATA_USERNAME: ${{ secrets.EARTHDATA_USERNAME }}
EARTHDATA_PASSWORD: ${{ secrets.EARTHDATA_PASSWORD }}
CAMS_EMAIL: ${{ secrets.CAMS_EMAIL }}
run: tox -e py${{ matrix.python-version }}

- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-py${{ matrix.python-version }}
path: test-results/coverage
if-no-files-found: ignore

format-check:
check-format:
name: black + isort + flake8 + interrogate
runs-on: ubuntu-latest

steps:
- name: Check out the repository
uses: actions/checkout@v3

- name: Set up Python 3.12
uses: actions/setup-python@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.12

- name: Install dependencies for formatting and linting
run: |
python -m pip install --upgrade pip
pip install tox

- name: Run Tox Check Format
run: tox -e check-format
python-version: "3.12"
cache: pip
- run: python -m pip install --upgrade pip tox
- run: tox -e check-format

static-analysis:
name: mypy
runs-on: ubuntu-latest

steps:
- name: Checkout full Git history
uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up Python 3.12
uses: actions/setup-python@v4
- uses: actions/setup-python@v5
with:
python-version: 3.12

- name: Install dependencies for static analysis
run: |
python -m pip install --upgrade pip
pip install tox

- name: Run Tox Static Analysis
run: tox -e static-analysis
python-version: "3.12"
cache: pip
- run: python -m pip install --upgrade pip tox
- run: tox -e static-analysis
24 changes: 21 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,27 @@ __pycache__/
/test-results/

version.py
data/

# Python bytecode files
# Generated artifacts under data/ — reproducible from the warehouse + recipes,
# kept off git so the repo stays small. data/ground_measurements/ IS tracked
# (canonical raw inputs), with one exception below.
/data/bundles/
/data/training_snapshots/

# CrossBoundary Energy ground measurements are covered by an NDA and must
# not be checked in. The data is retained in the BigQuery warehouse + on
# Google Drive for internal modelling; the repo does not redistribute it.
/data/ground_measurements/CBE_Data/

# Tool caches and editor / venv state
.mypy_cache/
.pytest_cache/
.vscode/
/data
.idea/
.venv/

# Ad-hoc run logs from long migrations (e.g. b8.log, c8.log).
*.log

# Local Claude config — per-developer prompt and agent state.
.claude/
Loading
Loading