Skip to content

README.md

Latest commit

 

History

History
214 lines (140 loc) · 8.16 KB

README.md

File metadata and controls

214 lines (140 loc) · 8.16 KB
Raw

Open Source LLM Arena

Collect human preference datasets for less-resourced languages and specific sectors,
while raising awareness about model diversity, bias, and environmental impact.

Built by the French government, now growing into new languages and sectors.

🇫🇷 French platform  ·   🇩🇰 Danish platform

Supported by DINUM, Ministry of Culture, ALT-EDIC, Denmark, and recognised as a Digital Public Good

How does it work?

🇫🇷 The French use case

Launched in October 2024 by DINUM and the French Ministry of Culture to address the lack of French-language preference data for LLM training nd evaluation.

Since launch: 700,000+ prompts, 250,000+ preference votes, 450,000+ visitors. One of the largest non-English human preference datasets available. All data published openly on Hugging Face:comparIA/comparia-fr-arena

We published a pre-print to dive deep into the project's strategy in France.

 Compar:IA featured on France 2 news, being used in a classroom

Compar:IA on the France 2 evening news, used in the classroom to teach students about AI models, bias, and environmental impact.

For whom?

🌍 Languages

Most LLMs underperform outside English. Compar:IA collects the preference data needed to close this gap.

Already live in French and Danish, and planning launches in Sweden, Estonia and Lithuania.

🏛️ Sectors

Generic benchmarks miss domain-specific needs. A sector arena reveals which models handle specialised language best.

Healthcare, legal, education, public admin, agriculture...

🏢 Organisations

Run your own arena, evaluate models on your real-world tasks, and contribute data back to the commons.

Governments, universities, hospitals, companies, NGOs...

Benefits

💡 Raise awareness

Teach citizens and professionals about model diversity, bias, and environmental cost. Already used in schools and training sessions.

Blind comparison between two models

📊 Generate rare datasets

Produce instruction and preference data in less-ressourced languages.

Dataset analysis visualization

🔁 Downstream reuse

Data feeds into new model training, leaderboards, use case mappings, and other research topics.

Downstream data analysis

Interested in an arena for your language, sector, or organisation?

The platform is fully open source, self-hostable, and customizable: choose your models, translate the interface, adapt prompt suggestions, add your logo. We can host it for you or help you set it up yourself.

Whatever your situation, reach out first and we'll figure out the best path together.

📬 contact@comparia.beta.gouv.fr

Contribute, we need you 🤝

Compar:IA is a digital common. Whether you can offer funding, code, translations, or simply ideas, there is a place for you.

💰 Financially. Compar:IA has been funded by DINUM and the French Ministry of Culture, with European support from ALT-EDIC. We are actively looking for new partners and funders to sustain the infrastructure, expand to new languages, and keep the project independent. contact@comparia.beta.gouv.fr

💻 In code. The entire platform is open source and we welcome contributions of all sizes: bug fixes, new features, translations, documentation. Come build with us. GitHub repository

💬 In discussions. Share your ideas, flag issues, or just ask questions on GitHub Discussions. We want to hear from you. GitHub Discussions

Any other way. Partnerships, academic collaborations, media coverage, spreading the word: every contribution matters. Reach out and let's talk. Contact us

Roadmap

🟢 In Progress

🔮 Up Next

  • Live use-case mapping (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Message history (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Socio-demographic data collection (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Back-office management (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)

⛵ Shipped

  • New voting system (🇪🇺 ALT-EDIC, 🇫🇷 Ministry of Culture)
  • Web search (🇪🇺 ALT-EDIC)
  • Separation of all platforms into separate instances (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Ranking consolidation and internationalization (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Language/platform-specific model support (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Gradio → FastAPI migration (🇫🇷 Ministry of Culture, 🇫🇷 DINUM, 🇪🇺 ALT-EDIC)
  • EcoLogits update (🇪🇺 ALT-EDIC, 🇫🇷 DINUM)
  • Dataset publishing pipeline v1 (🇫🇷 DINUM, 🇫🇷 Ministry of Culture)
  • Leaderboard v1 (🇫🇷 DINUM, 🇫🇷 Ministry of Culture, in collaboration with 🇫🇷 PEReN)
  • Archived models (🇫🇷 DINUM, 🇫🇷 Ministry of Culture)
  • Blog section (🇫🇷 DINUM, 🇫🇷 Ministry of Culture)
  • Internationalization foundations (🇫🇷 DINUM, 🇫🇷 Ministry of Culture)
  • compar:IA v1 (🇫🇷 DINUM, 🇫🇷 Ministry of Culture)

👉 Full technical roadmap on GitHub

Getting started

The platform is fully open source and self-hostable.

Self-host with Docker (single server, automatic HTTPS via Caddy): see DOCKER_INSTALL.md

Local development (basic):

cp .env.example .env       # Configure environment variables
make install               # Install all dependencies
source .env
make dev-backend           # Backend on http://localhost:8008
make dev-frontend          # Frontend on http://localhost:5173

For the full setup guide (instances, KeePass, Docker, testing, models, i18n, architecture), see CONTRIBUTING.md.

Digital Public Goods Badge