Skip to content

End-to-end pipeline deployment on EEE infrastructure #1

Description

@arishofmann

The pipeline already generates BenchmarkCards end-to-end. The next step is getting it deployed on EEE infrastructure so it can run automatically when new benchmarks come in.

Steps:

  • Get access to EvalEval HF org and GitHub org
  • Test HF inference credits, check which models are available
  • Create HF dataset for generated cards: https://huggingface.co/datasets/evaleval/auto-benchmarkcards
  • Validate that card quality holds with the new LLM backend
  • Look at the existing EEE webhook/bot infrastructure and figure out how to plug in
  • Set up automation so new benchmarks without a card get one generated and submitted as a PR
  • Surface the confidence flags from validation in the PR so reviewers know which fields need the most attention

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions