Request for a model training tutorial

[TonyZhang-sjtu]

Hi~
I wanna train an MLP router with reference to your scripts, where I found a file named "../llm_performance.npz". However, this file has not been uploaded to GitHub yet, and I'm not sure what is in it. So, would you please give me some instructions on how to construct this file?
Thanks very much!

[linggm3]

Hi~
Thanks for your interest! The llm_performance.npz file contains preprocessed data for training and evaluating the MLP router. Specifically, it includes six arrays:

• train_embed, val_embed, test_embed: embeddings for train/val/test splits.
• train_score, val_score, test_score: corresponding performance scores.

For reference, test_router.py outlines the pipeline to build the .npz dataset and perform router training and testing.

Here’s the relevant code snippet:

np.savez('data/tmp_perf.npz', 
         train_embed=router_dataset['embedding']['train_embed'],
         val_embed=router_dataset['embedding']['val_embed'],
         test_embed=router_dataset['embedding']['test_embed'],
         train_score=train_score, 
         val_score=val_score, 
         test_score=test_score)

You can simply change baseline = 'knn' to baseline = 'mlp' in test_router.py and run python test_router.py to train and test the MLP router.

Let me know if you need more details! 😊

[TonyZhang-sjtu]

Big thanks for your timely and comprehensive response. I've run the code successfully and get more questions that I'd like to discuss with you~

1. I noticed a line if difficulty == 'hard': break, which seems to exclude the experiments with 10, 100 and 1000 LLM candidates. How can I reproduce the experiment under "hard" setting?

2. The appendix of the paper said, "For specific candidates of these settings and other evaluation candidates, please refer directly to our code." But I didn't find the definition of these candidates and code of choosing the most proper candidates for "all-strong", "all-weak" and "strong-to-weak" settings. Would you please show me how I can get to know and access the specific LLM candidates?

3. I also wanna try using different LLM candidates like 3, 6, or 20 etc. But I find the definition seems to be imported directly from data/router_dataset/{to_handle}_router_dataset.pkl. Therefore, how can I modify or regenerate these pkl files?

4. Under the leaderboard_embed, leaderboard_prompt, and leaderboard_score folders, there are pkl files named _mmlu_pro, _new, and _old. What are the differences between them? When I try to train a router, which one should I refer to?

Thank u very much!!!

[linggm3]

Thanks for your questions! Here are quick answers:

1. Simply remove or comment out the line if difficulty == 'hard': break to run the "hard" setting experiments.

2. Check "Advanced Tutorial 2: Constructing Router Dataset" in the README and the get_router_dataset.py file for candidate definitions and selection methods.

3. Modify get_router_dataset.py to generate custom PKL files with your desired LLM candidates (3, 6, 20, etc.).

4. They denote different benchmark sets. The differences are:

• _old: 6 earlier benchmarks (generally higher LLM scores)
• _new/_mmlu_pro: Newer, more challenging benchmarks (lower scores)

We recommend training and testing your router across all of them for a comprehensive evaluation.

Hope this helps! Feel free to reach out if you have more questions.