Skip to content

Dask version of Validation Object Table for DC2 Run 2.2i - U/wmwv/dr6 dask refactor #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wmwv wants to merge 85 commits into
base: master
Choose a base branch
from
Open

Dask version of Validation Object Table for DC2 Run 2.2i - U/wmwv/dr6 dask refactor #159

Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
057dfdc
Start DASK version of DR6 validation script.
wmwv Jun 7, 2020
0cdb1a6
Fix errors in filters from load_data, data_release message.
wmwv Jun 7, 2020
68a255f
Read file with dask.
wmwv Jun 7, 2020
af9fa63
Fix sampling_factor, use pyarrow engine.
wmwv Jun 7, 2020
9032748
Fix up plotname to plot_gmr_cmodel, filters from load_data.
wmwv Jun 7, 2020
8b09f1c
Add plt.clf after plt.savefig s
wmwv Jun 7, 2020
a5a6f68
Run test plots. No ellipticity.
wmwv Jun 7, 2020
6f02af6
Rename df->ddf to remind me that this is Dask.
wmwv Jun 7, 2020
7fdeff3
Fix pddf->pdf and two missing {suffix}es
wmwv Jun 8, 2020
34ec095
Explicitly extract dask arrays for plot_mag_magerr.
wmwv Jun 8, 2020
c62305c
Write out arrays to dask arrays when making pltos with operations.
wmwv Jun 8, 2020
a08f828
Small trivial cleanup.py
wmwv Jun 8, 2020
a77929c
Explicitly convert dask arrays in plot_psf_cmodel.
wmwv Jun 8, 2020
8cf385f
Break out hist into individual calls in plot_psf_cmodel.
wmwv Jun 8, 2020
1275731
Use multiple workers. Default 8.
wmwv Jun 8, 2020
2496f5e
Use Agg backend to avoid trying to connect to X11.
wmwv Jun 20, 2020
c98c25c
Load Client with shared scheduler.json file.
wmwv Jun 20, 2020
d5fe348
Pass client object to run method.
wmwv Jun 20, 2020
61ddac5
Repartition Dask frame across the workers.
wmwv Jun 20, 2020
0aab2ec
Add simple instructions for running Dask version of validate script.
wmwv Jun 19, 2020
daa7a88
Generalize load_data to dr6a, dr6b.
wmwv Jul 7, 2020
5243f0d
Delay defining of good selection to help out Dask.
wmwv Jul 8, 2020
39d4935
Generalize Davenport stellar locus load location to reletiave to __fi…
wmwv Jul 8, 2020
cc82f3b
Use data_release instead of catalog_file in __main__, run, run_test.
wmwv Jul 8, 2020
e027084
Fix start_dask_cluster.
wmwv Jul 8, 2020
fb0447b
Clean up run, run_test.
wmwv Jul 8, 2020
1d9c764
Remove run_test.
wmwv Jul 8, 2020
6b5c034
Update default name to 'DR6a' for plot filenames.
wmwv Jul 9, 2020
06e7861
Start adapting Notebook to Dask.
wmwv Jul 10, 2020
c2d3519
Add Local Dask option.
wmwv Jul 10, 2020
92c5369
Got Dask+Holoviews plotting working through to Extendedness
wmwv Jul 10, 2020
117f0c2
Convert all plots to use Holoviews.
wmwv Jul 13, 2020
3ec1780
Add pedagogical notes on Dask, Holoviews, Parquet.
wmwv Jul 14, 2020
ea4699c
Clean up unused imports and functions.
wmwv Jul 14, 2020
496e8cf
Add LOCAL_FILE option to more clearly trace file.
wmwv Jul 14, 2020
42ca33a
Add a few more details on launching Jupyter Classic Notebook.
wmwv Jul 15, 2020
ce5739c
Update to desc-python-bleed kernel.
wmwv Jul 15, 2020
c187135
Add a few more notes about when Dask is doing actual work.
wmwv Jul 15, 2020
62da060
Trying to get tract, patch annotation on RA, Dec plot.
wmwv Jul 15, 2020
bf273ed
A bit more work on RA, Dec hover over.
wmwv Jul 16, 2020
6783464
Move LOCAL_DASK, LOCAL_FILE options to be more prominent.
wmwv Jul 16, 2020
b9a9a4e
Use Curve instead of Histogram for better plots.
wmwv Jul 16, 2020
a818a99
Add a few more links to HoloViz and HoloViews.
wmwv Jul 16, 2020
941e9c1
Improve explanations and Dask Cluster setup.
wmwv Jul 17, 2020
ccfdd55
Make cluster-based Dask and NERSC Parquet file location default.
wmwv Jul 17, 2020
db6ae9f
Define star, galaxy on a boolean column after good is persisted
wmwv Jul 17, 2020
1a453bf
Restore star color-color plot.
wmwv Jul 17, 2020
65c46f5
Generate PSF FWHM distribution using Curve.
wmwv Jul 17, 2020
e96079f
Show both styles of RA, Dec plots.
wmwv Jul 17, 2020
1e906e0
Adjust ellipticities ylim to start at 100.
wmwv Jul 17, 2020
7501007
Use dask.array operations in plot_data_curve.
wmwv Jul 17, 2020
660bcd2
Clean up some plotting.
wmwv Jul 17, 2020
e813006
Rework ellipticity calc to add the complex column
wmwv Jul 17, 2020
2717790
Fix plot_data_curve to normalize as advertized.
wmwv Jul 17, 2020
596b624
Clean up ellipticity calculations.
wmwv Jul 17, 2020
4eb9177
Add -L SCRATCH to salloc command.
wmwv Jul 18, 2020
9e02d56
First attempt at Mollweide display of RA, Dec
wmwv Jul 18, 2020
4c8e923
Make RA, Dec Molleweide project work.
wmwv Dec 9, 2020
81de98b
Start working on refactor of Dask reads.
wmwv May 27, 2021
af90a04
Start refactoring around set_index and carefully memory management.
wmwv Jun 2, 2021
a745012
Sort through the extended_df plots. Performance slow.
wmwv Jun 3, 2021
c663d9d
Trivial move of up one cell.
wmwv Jun 9, 2021
106445e
BUG FIX: Error in star subset dataframe definition.
wmwv Jun 10, 2021
7a814ff
Explain memory management. Use client.cancel.
wmwv Jun 14, 2021
f7afc70
Use da.histogram to replace np.histogram
wmwv Jun 14, 2021
1112cb7
Specify 'sorted=True' for 'set_index' runs.
wmwv Jun 16, 2021
337b175
Explain why canceling data frames yields blank datashaded frames.
wmwv Jun 16, 2021
38b07dc
Strip out astro quantity from function passed to Dask.
wmwv Jun 16, 2021
363e428
Explain LOCAL_DASK more.
wmwv Jun 16, 2021
98791d9
try/except graphviz.
wmwv Jun 16, 2021
b841c65
Make calculate_area robust to healpy.ang2pix nside in Dask
wmwv Jun 16, 2021
b4f53ed
Aggressively trim memory using client.run(libc.malloc_trim)
wmwv Jun 17, 2021
6b2002f
Use desc-python-bleed as default kernel.
wmwv Jun 17, 2021
3053dd9
Structure data paths for all 166 tracts or just one sample tract.
wmwv Jun 17, 2021
d69174d
Use MALLOC_TRIM_THRESHOLD to control memory usage. Document
wmwv Jun 17, 2021
1aab460
Add Learning Objectives and a bit more about memory.
wmwv Jun 17, 2021
07defec
Set workers at 8/node.
wmwv Jun 17, 2021
a912050
Reset data_dir to NERSC.
wmwv Jun 17, 2021
1bccec7
Bump back up to 16 workers/node.
wmwv Jun 17, 2021
161f98b
Cast dataframe columns in one go with dict passed to astype.
wmwv Jun 17, 2021
4a8f5d8
Remove counting GB.
wmwv Jun 17, 2021
9eb0157
Remove never-really-finished command-line script version.
wmwv Jun 17, 2021
4e6c45c
Fix some errors in explanatory notes about Bokeh plots.
wmwv Jun 17, 2021
0cab304
Remove mention of dask script in README.md
wmwv Jun 17, 2021
49a6c8c
Fix URL reference for RA, Dec with Mollweide Notebook.
wmwv Jun 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading