[Epic] CUGA Evaluation Framework - Accuracy, Consistency & Policy Compliance

[adiasaf]

What You Want

Establish a comprehensive evaluation framework for CUGA to ensure the agent meets production-grade standards for accuracy, consistency, and policy compliance across diverse workloads and configurations.

Success Criteria

Primary KPIs:

• ≥90% pass rate on policy-controlled task executions
• 99% safety compliance (specific KPI to be defined)
• Validated on open-source/open-model configurations
• Tested on public benchmarks + real customer datasets

Initial Evaluation Targets

1. Public Benchmarks

   • AppWorld: Demonstrate CUGA outperforms ReAct baseline
   • Vakra: Demonstrate CUGA outperforms ReAct baseline

2. Real-World Validation

   • Evaluate on 2+ real customer datasets
   • Measure performance across different domains and use cases

3. Future Benchmark Considerations

   • Tau: May be added as additional evaluation benchmark

Benchmarks

• AppWorld - Benchmark for app-based agents
• Vakra - Microsoft's agent evaluation benchmark
• Tau - Future benchmark consideration
• ReAct paper: https://arxiv.org/abs/2210.03629

[merveunu]

@adiasaf I hope this will also capture the delta memory and RL teams have been observing w/their components' contribution to react vs cuga...Can you also have an explicit work item for those in this epic please?

cc: @visahak @serialorganizer