Milestones
List view
Understand personas - who will use HumaneBench, and how?
Overdue by 1 year(s)•Due by October 2, 2025•1/2 issues closedWe want [HumaneBench.ai](http://HumaneBench.ai) to become an industry standard for benchmarking LLMs against humane tech principles. To this end, we want to create a *synthetic benchmarking data generation pipeline* that lets engineers and researchers easily create relevant test cases to benchmark their models for their specific application context. The base case is benchmarking chatbots such as ChatGPT, Claude, Mistral etc. using prompts from a variety of topics humans may interact with them about, such as: * Education and research * Relationship issues * Physical health and wellness * Mental and emotional health * Ethical dilemmas * … Beyond this use of the benchmark, we want to make it easy to adapt the pipeline for humane evaluation of LLMs in specific contexts. For example, if an LLM is being used for therapy, some categories to evaluate it in could be: * Giving compassionate advice * Showing understanding of human emotions * Referring to human experts/emergency services when appropriate * Encouraging healthy behavior * Discouraging harmful behavior * … The dataset should be a JSONL file with the following column headers: * `input`: the prompt entered by the user * `target`: an LLM response aligned with humane tech principles * `category`: the general situation/context of the prompt and response * `severity`: low, medium, high, critical (details below) * `principle_to_evaluate`: which one of the six humane tech principles is being evaluated For the `severity` column, here are definitions for each level: * low: general questions, everyday decisions, low-stakes situations * medium: important personal decisions, moderate consequences * high: significant life decisions, potential harm, vulnerable populations * critical: crisis situations, immediate danger, emergency response needed “Vulnerable populations” are demographics for which we want to pay closer attention to how LLM responses may affect them. Here are some examples: * children and teens * elderly people * people in immediate crisis or distress * people with mental health issues/disabilities * non-native language speakers * people with low tech literacy The pipeline should result in a high-quality, relevant dataset with low redundancy between individual elements. It should also include tools for manual oversight, review and quality assurance. Resources: * Our team’s collection of documents, recorded conversations and other resources on humane tech: [All Things Humane Tech](https://app.storytell.ai/app/invite/collection?token=pub\_sEZzSHnVwi0qvGEjfCKYwQMLm1ciaM-e3TvzSQIV6NqtjRrfi7FRtwgB3s3-) * [Constitutional AI paper from Anthropic](https://arxiv.org/pdf/2212.08073) for RLHF approaches to minimize harm and guide ethical data curation practices * We are not training models yet, but this may be inspiring * [Darkbench.ai paper](https://arxiv.org/pdf/2503.10728)
Overdue by 1 year(s)•Due by October 17, 2025•6/12 issues closedTransform humanebench from a basic demonstration tool into a robust LLM benchmarking framework by implementing multi-level scoring, error handling, and configuration capabilities inspired by [darkbench](https://github.com/apartresearch/DarkBench/tree/main/darkbench)'s architecture. - Reliability: Consistent, well-structured evaluation results even with flaky model responses - Granularity: Nuanced scoring that captures degrees of human-friendliness - Flexibility: Configurable evaluation parameters for different use cases - Debuggability: Rich metadata collection for troubleshooting and analysis - Scalability: Support for larger datasets and multiple judge models
Overdue by 1 year(s)•Due by October 17, 2025•1/6 issues closed