Blazing fast structured decisions for System One open-models self-hosted with a TypeSafe AI compatible API, written in Rust.
tokio,axumandserde, the usual suspects- System One compatible API Spec
candlewithtokenizersrelease candidate!- Dynamic, token-based batching
- SDPA on CPU, Metal, and CUDA
- Flash Attention on Ampere, Ada Lovelace, and Hopper
- Blazing fast inference for Laya
Install it with support for CPU, Metal or CUDA.
cargo install sys1 --features cpu
# cargo install sys1 --no-default-features --features metal
# cargo install sys1 --no-default-features --features cuda
# cargo install sys1 --no-default-features --features cuda,flash-attn-2 # Ampere, Ada Lovelace, or Hopper
# cargo install sys1 --no-default-features --features cuda,flash-attn-3 # HopperThen run it with any of the supported models (more coming soon!).
convaiinnovations/layafor English text, guardrails, email triageconvaiinnovations/laya-multilingualfor 100+ languages, ~2.2x fasterconvaiinnovations/laya-typed-decisionsfor typed-decisions workflows
sys1 --model-id convaiinnovations/laya --dtype autoAnd, just query your Jev-compatible API at /v1/systemone (or /v1/decide).
curl http://localhost:3000/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"model": "convaiinnovations/laya",
"state": {
"message": "I was charged twice for invoice 4411. Please refund me today."
},
"questions": {
"route": {
"type": "choice",
"instructions": "Where should this ticket go?",
"criteria": {
"billing": "payments, refunds, invoices",
"bug": "the product is broken",
"account": "login or access"
}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this message?",
"criteria": [
"routine, no rush",
"today",
"urgent",
"critical, about to churn"
]
},
"escalate": {
"type": "noul",
"instructions": "Escalate to a human immediately?"
}
}
}'Or, install the TypeSafe AI SDK for Python, or the TypeSafe AI SDK for Javascript / Typescript.
from typesafe_sdk import Choice, TypeSafeClient
client = TypeSafeClient(base_url="http://localhost:3000", api_key="")
response = client.system_one(
model="convaiinnovations/laya",
state={"message": "I was charged twice for invoice 4411."},
questions={
"route": Choice(
instructions="Where should this ticket go?",
criteria={"billing": None, "bug": None, "account": None},
),
},
)
print(response.choices["route"].choice)