Given that there has been desire from some of the scientists to be able to provide feedback on model performance (particularly, to highlight the failure cases, mostly false positives, of the model), there needs to be a structural process for quickly addressing feedback.
To this end, utilizing GitHub issues is good option for a pipeline to have people state, track, and resolve their own issue with the model as well as reference other issues as well to see if there have been prior similar cases.
This pipeline should consist of the following:
- A formal distinction between issues meant for the code base and issues meant for model retraining
- issue tags should be made here to tag these distinctions
- A GitHub issues template for people to use for easily and concretely providing feedback so that the maintainers don't need to do very much work
Given that there has been desire from some of the scientists to be able to provide feedback on model performance (particularly, to highlight the failure cases, mostly false positives, of the model), there needs to be a structural process for quickly addressing feedback.
To this end, utilizing GitHub issues is good option for a pipeline to have people state, track, and resolve their own issue with the model as well as reference other issues as well to see if there have been prior similar cases.
This pipeline should consist of the following: