Reward Function Templates

[fionnachan]

@johnnyL7 requested a feature to make the reuse of reward functions more convenient when the user wants to reuse a set of previous reward functions for a new model.

The following is a sketch of my idea.

User can save any existing reward function of a non-draft Experiment as a reward function template by clicking the "Save" button on the reward function panel

On the Project/Model view, there is an option to create multiple experiments.

Similarly, at the final step of the Model Upload process, the user can choose to create multiple experiments. Clicking that button would bring the user to a page which asks for the same information as demonstrated above.

Add "Saved" Page to Main Menu

User viewing their saved reward function templates. Clicking the trash button would show a confirmation dialog asking if the user wants to delete the reward function template. It should delete the reward function template from the DB.

User can edit their saved reward function templates

[slinlee]

I like this. There's a lot of good thinking, @fionnachan

Here are some questions in no particular order:

• What if a user wants to start with a template and then modify it (maybe to match a change in a reward var name?)
  • If starting multiple experiments, maybe they're each experiment is created as a draft until the user clicks 'start training'?
  • On the new experiment page, we could add a dropdown to select from the templates. This would populate the reward function editing area. This lets the user preview or edit the reward the function before starting training.
• When starting multiple experiments, how about we just show a list of all the templates and the user can check a checkbox to opt-in each reward function they want to start. This way they don't have to think of the number of experiments first
• Do we need a 'favorite' template function in addition to the ability to save templates? This would be to differentiate the current best choices.

[fionnachan]

1. Yea I was thinking that we create multiple draft experiments for them and not run them immediately so that should solve the problem?

   • Yes that's great :D
     => I'll add the new experiment view mockup in the comments below later

2. Oh ok. Do you think we should show the reward function code itself too or just the reward function template names?
   => I'll create a mockup for this too

3. That sounds good but I'd prefer it to be a separate "feature enhancement" ticket after this "reward function template" work

Btw would we want to give free & paid users a different total limit of reward function templates they can have?

[slinlee]

paid vs free

how about we make templates a paid only feature? Sound like a good idea @ejunprung ?

[ejunprung]

Yeah, definitely. I think we everything we do going forward (except fixing bugs, sales/marketing stuff, etc.) should be paid only.

[fionnachan]

New Experiment View

List box
The only caveat here is that the list could be long enough that the list box will need a scrollbar

[ejunprung]

Are reward templates specific to particular model versions? What happens if the user changes the reward syntax and breaks the template? I think 90%+ of the time, the rewards are not portable.

[slinlee]

This is meant exactly to port reward functions between model revisions. The way that we're trying to get around that is to create the new experiments as 'drafts' first. Then the user can edit them and also see which variables are invalid.

[slinlee]

Add "Saved" Page to Main Menu

User viewing their saved reward function templates. Clicking the trash button would show a confirmation dialog asking if the user wants to delete the reward function template. It should delete the reward function template from the DB.

@fionnachan I think I missed this section when I first read through. To me, the scope of the templates is per project, since that's what would have the same objectives and goals across models.

So a user wouldn't use a coffeeshop function on the moonlander, since the coffeeshop would be measuring things like wait time and customers served.

The different saved functions would be changes like, "double the importance of fuel savings" or "factor in customer satisfaction" for a single project.

[fionnachan]

I think it's easier to organise the templates on one dedicated page. If I am the user, I would want to know how many templates I have at any give time, and to be able to edit / remove them.

We can move the "Templates" button to the Project/Model view instead if a standalone section is not a good option and only show the templates for that project

Originally I thought it's up to the user to keep which templates they have and show all of them on the dropdown / list. I agree that showing only those for the project would be better.

[slinlee]

Yep the same list of templates but just on the project level sounds good. For a minimal first pass I think the feature would still work without the list of templates too.