[Question] How to benchmark a custom optimizer with its specific hyperparameters and search spaces?

[TrandeLik]

Hi! First off, thank you for creating and maintaining LightBench! It looks like a powerful tool for evaluating optimizers.

I want to evaluate a custom optimization algorithm using your library. However, after examination of both README and project code, I'm having trouble understanding the correct way to integrate my own optimizer, define its unique hyperparameters, and specify the search spaces for tuning them.

Thank you for your time and any guidance you can provide

[ClashLuke]

Hi Dmitrii, thank you for your interest!
You're right, that should be exposed a lot better, I appreciate the pointer and will fix that soon.

At the moment, the search space is defined here:

LightBench/lightbench/utils.py

Lines 829 to 837 in e91d19f

	sampler = AutoSampler(
	seed=0x123125,
	search_space={
	"lr": optuna.distributions.FloatDistribution(1e-7, 100, log=True),
	"1mbeta1": optuna.distributions.FloatDistribution(1e-5, 1, log=True),
	"1mbeta2": optuna.distributions.FloatDistribution(1e-7, 1, log=True),
	"1mshampoo_beta": optuna.distributions.FloatDistribution(1e-7, 1, log=True),
	},
	)

(+ multiple usages of those same variable names)

Optimizers are instantiated here:

LightBench/lightbench/utils.py

Lines 49 to 55 in e91d19f

	def get_optim(optim: str | C.BaseOpt, params, **kwargs) -> C.BaseOpt:
	args = {**base_args, **kwargs}
	if isinstance(optim, str):
	optim = getattr(heavyball, optim)
	signature = inspect.signature(optim)
	o = optim(params, **{k: v for k, v in args.items() if k in signature.parameters})
	return o

You can modify both functions to instantiate your own local optimizer and change the search space to better fit it.