Hi! I maintain EvalPort, a small open spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets meant to move between different eval tools instead of each one inventing its own file format.
This SDK's client.chat.chat(messages=..., model="palmyra-x5") response already carries what's needed for a lightweight eval record — the input messages, model, choices[0].message.content, and id. Something like:
from evalport import TestResult
def to_evalport(messages, model, chat_completion) -> TestResult:
return TestResult(
test_case_id=chat_completion.id,
input={"messages": messages},
actual_output=chat_completion.choices[0].message.content,
metadata={"model": model},
)I know this SDK is Stainless-generated, so I'm not proposing a change to the generated client — more asking whether a short snippet like this would be worth having in examples/ alongside the streaming helpers, or whether it's not a good fit for a library this thin. Genuinely fine either way, just wanted to check before doing anything.
Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — feel free to close this if it's not useful.
Hi! I maintain EvalPort, a small open spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets meant to move between different eval tools instead of each one inventing its own file format.
This SDK's
client.chat.chat(messages=..., model="palmyra-x5")response already carries what's needed for a lightweight eval record — the inputmessages,model,choices[0].message.content, andid. Something like:I know this SDK is Stainless-generated, so I'm not proposing a change to the generated client — more asking whether a short snippet like this would be worth having in
examples/alongside the streaming helpers, or whether it's not a good fit for a library this thin. Genuinely fine either way, just wanted to check before doing anything.Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — feel free to close this if it's not useful.