Skip to content

user dependent pixi paths #174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
j321m merged 7 commits into from
Apr 14, 2026
Merged

user dependent pixi paths #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9fc3ed5
user dependent pixi paths
j321m Apr 9, 2026
97aacbd
Merge branch 'main' of github.com:llm-random/nano into jm_fix_pixi_cl…
j321m Apr 9, 2026
5fc57db
fix
j321m Apr 9, 2026
adca52f
fix
j321m Apr 9, 2026
b415bf0
Update dataset paths in helios.yaml configurationfix helios dataset p…
j321m Apr 9, 2026
ce996fe
Merge branch 'jm_fix_pixi_cluster_paths' of github.com:llm-random/nan…
j321m Apr 13, 2026
e3cb6a1
fix PATH
j321m Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configs/_cluster/entropy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ infrastructure:
- 'export HYDRA_FULL_ERROR=1'

# export pixi variables
- 'export PIXI_HOME=/storage_nvme_4/nano/pixi_new'
- 'export PATH="$PIXI_HOME/bin:$PATH"'
- 'export PIXI_HOME=$PROJECT_HOME_PATH/$USER/pixi'
- 'export PATH="$HOME/.pixi/bin:$PATH"'
- 'export XDG_DATA_HOME="$PIXI_HOME/data"'
- 'export XDG_CACHE_HOME="$PIXI_HOME/cache"'
- 'export XDG_STATE_HOME="$PIXI_HOME/state"'
Expand Down
2 changes: 1 addition & 1 deletion configs/_cluster/entropy_a100.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ infrastructure:
- 'export HYDRA_FULL_ERROR=1'

# export pixi variables
- 'export PIXI_HOME=/storage_ssd_1/nano/pixi'
- 'export PIXI_HOME=$PROJECT_HOME_PATH/$USER/pixi'
- 'export PATH="$PIXI_HOME/bin:$PATH"'
- 'export XDG_DATA_HOME="$PIXI_HOME/data"'
- 'export XDG_CACHE_HOME="$PIXI_HOME/cache"'
Expand Down
12 changes: 6 additions & 6 deletions configs/_cluster/helios.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ infrastructure:
script:
- '${export_env_variables_placeholders:}'
- 'module load ML-bundle/25.04'
- 'export PROJECT_HOME_PATH=$PLG_GROUPS_STORAGE/plggllmeffi/nano'
- 'export PROJECT_HOME_PATH=$PLG_GROUPS_STORAGE/plggllmefficont3/nano'
- 'export HF_HOME=$PROJECT_HOME_PATH/hf_cache'

# hydra errors
- 'export HYDRA_FULL_ERROR=1'

# export pixi variables
- 'export PIXI_HOME=$PROJECT_HOME_PATH/pixi'
- 'export PIXI_HOME=$PROJECT_HOME_PATH/$USER/pixi'
- 'export PATH="$HOME/.pixi/bin:$PATH"'
- 'export XDG_DATA_HOME="PROJECT_HOME_PATH/data"'
- 'export XDG_CACHE_HOME="$PROJECT_HOME_PATH/cache"'
Expand All @@ -32,7 +32,7 @@ infrastructure:
- 'cd -'

cluster_switch:
train_path_c4: "/net/scratch/hscra/plgrid/plgmaciejpioro/c4/train"
eval_path_c4: "/net/scratch/hscra/plgrid/plgmaciejpioro/c4/validation"
train_path_fineweb: "/net/scratch/hscra/plgrid/plgmaciejpioro/fineweb-edu/train/train"
eval_path_fineweb: "/net/scratch/hscra/plgrid/plgmaciejpioro/fineweb-edu/train/train"
train_path_c4: "/net/storage/pr3/plgrid/plggllmeffi3/datasets/c4/train"
eval_path_c4: "/net/storage/pr3/plgrid/plggllmeffi3/datasets/c4/validation"
train_path_fineweb: "/net/storage/pr3/plgrid/plggllmeffi3/datasets/fineweb/train"
eval_path_fineweb: "/net/storage/pr3/plgrid/plggllmeffi3/datasets/fineweb/train"
9 changes: 5 additions & 4 deletions configs/_cluster/lem.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,15 @@ infrastructure:

script:
- '${export_env_variables_placeholders:}'
- 'export HF_HOME="/lustre/pd01/plgrid/plgllmefficont2/crewtool/hf"'
- 'export PROJECT_HOME_PATH=/lustre/pd03/plgrid/plgllmefficont3/nano'
- 'export HF_HOME=$PROJECT_HOME_PATH/hf_cache'

# hydra errors
- 'export HYDRA_FULL_ERROR=1'

# export pixi variables
- 'export PIXI_HOME=/lustre/pd01/plgrid/plgllmefficont2/pixi'
- 'export PATH="$PIXI_HOME/bin:$PATH"'
- 'export PIXI_HOME=$PROJECT_HOME_PATH/$USER/pixi'
- 'export PATH="$HOME/.pixi/bin:$PATH"'
- 'export XDG_DATA_HOME="$PIXI_HOME/data"'
- 'export XDG_CACHE_HOME="$PIXI_HOME/cache"'
- 'export XDG_STATE_HOME="$PIXI_HOME/state"'
Expand All @@ -34,4 +35,4 @@ cluster_switch:
train_path_c4: "/lustre/pd01/plgrid/plgllmefficont2/datasets/c4/train"
eval_path_c4: "/lustre/pd01/plgrid/plgllmefficont2/datasets/c4/validation"
train_path_fineweb: "/lustre/pd01/plgrid/plgllmefficont2/crewtool/proba2/fineweb-edu/train/train"
eval_path_fineweb: "/lustre/pd01/plgrid/plgllmefficont2/crewtool/proba2/fineweb-edu/train/train"
eval_path_fineweb: "/lustre/pd01/plgrid/plgllmefficont2/crewtool/proba2/fineweb-edu/train/train"
Loading