Repo size growth #288

Activity

1. roussel-ryan commented on Mar 21, 2025

   

   roussel-ryan

   on Mar 21, 2025

   Collaborator

   More actions

   I agree, we will discuss this during the software meeting today and hopefully @pluflou can take a look at migrating the notebooks to a separate repo in xopt-org

   👍React with 👍1Reacted by Sara A. Miskovich
2. pluflou commented on Apr 9, 2025

   

   pluflou

   on Apr 9, 2025

   Collaborator

   More actions

   I looked at the largest files in the repo, and at least 1GB of the size was due to assets from the docs site deployment that were being stored in the gh-pages branch. Up until 7 months ago, we were deploying the docs from a branch, but now we use the actions/deploy-pages to do so, which uploads artifacts that are generated during the workflow to GitHub’s internal infrastructure responsible for serving GitHub Pages.

   I can confirm that the last time these files were generated was September 2024, before we migrated to the actions deployment workflow. See below for the two largest files (the JSON site map and the ZIP file of all the executed Jupyter Notebooks):

   git rev-list --all | while read commit; do
     git ls-tree -r $commit | grep "search_index.json" | while read mode hash filename; do
       echo "$(git show -s --format="%ai" $commit) $hash $filename"
     done
   done
   2024-09-10 18:24:03 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
   2024-09-10 17:55:14 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
   

   and

   git rev-list --all | while read commit; do
     git ls-tree -r $commit | grep "search_index.json" | while read mode hash filename; do
       echo "$(git show -s --format="%ai" $commit) $hash $filename"
     done
   done
   2024-09-10 18:24:03 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
   2024-09-10 17:55:14 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
   

   I removed these two files from the commit history of the repo, and it brought it down to 350MB.

   My suggestion to resolve this issue is to do the same with all docs deployment related objects (HTML files are the next largest in the repo, along with .ipynb files), and force push a cleaned up commit history (someone with admin privileges will need to do this). Since we are no longer deploying from a branch, and we clear all notebooks before committing, we should not see any more crazy repo size growth after that. This would also avoid us having to move the docs to a separate repo (at least until there’s another reason to do so).

   The only drawback is that we’d be rewriting the commit history, but this seems like a necessary step anyway to shave off the repo size.
3. roussel-ryan commented on Apr 10, 2025

   

   roussel-ryan

   on Apr 10, 2025

   Collaborator

   More actions

   Thanks for the detailed report @pluflou , if there are no objections I think we should proceed with this plan, probably when I get back to the US later this month

   👍React with 👍1Reacted by Sara A. Miskovich
4. roussel-ryan commented on Apr 22, 2025

   

   roussel-ryan

   on Apr 22, 2025

   Collaborator

   More actions

   @pluflou and I just made an initial pass on reducing the repo size, git clone is now down to ~400MB. We will continue this effort over the next week or so