Proposal: libsy refactor

[grahamking]

A simpler core for libsy

Switchyard · crates/libsy

run_stream only. Three kinds of Step. No clients, no timeouts, no type erasure. About 1,000 fewer lines.

---

1. The idea

run_stream becomes the only way to use libsy. You get a stream of steps. You answer the ones that need answering. You don't have to respond.

Changes are mostly described in the example doc comments.

---

2. The three steps

/// One item on an algorithm's step stream.
pub enum Step {
    /// Please make this model call, then `respond` with the result.
    CallLlm(Box<CallLlm>),
    /// A routing choice, published as it is made.
    Decision(Decision),
    /// The final answer. libsy sends this when the algorithm returns.
    Done(Box<Response>),
}

A decision is a plain struct. There is no trait and nothing to downcast.

#[derive(Clone, Debug)]
pub struct Decision {
    /// The model to call.
    pub selected_model: String,
    /// Why, for logs and traces.
    pub reasoning: Option<String>,
    /// True for the call that answers the request. False for the
    /// classifier and judge calls made along the way.
    pub is_answer_call: bool,
}

A call the libsy user should make. The fields are public.

pub struct CallLlm {
    pub ctx:      Context,
    pub request:  Request,
    pub decision: Decision,   // decision.selected_model is the model_id this should go to.
    reply: oneshot::Sender<Result<Response>>,
}

impl CallLlm {
    /// Give the algorithm your result.
    pub fn respond(self, result: Result<Response>);

    /// Or take the contents and don't answer at all - aka "decision-only".
    pub fn into_parts(self) -> (Context, Request, Decision);
}

---

3. Writing an algorithm

An algorithm gets a Driver to work with. The driver has two methods.

impl Driver {
    /// Ask for a model call and wait for the answer.
    pub async fn call(&self, request: Request, decision: Decision) -> Result<Response>;

    /// Publish a routing choice.
    pub async fn decide(&self, decision: Decision) -> Result<()>;
}

Algorithm implementers must implement this trait which has two methods.

#[async_trait]
pub trait Algorithm: Send + Sync + 'static {
    fn name(&self) -> &str;

    /// The algorithm is all in here. Should return the final answer to users' request.
    async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<Response>;
}

An algorithm never publishes Step::Done itself. It returns, and libsy does it. Every run ends with Done or an error.

Example algorithm: It asks a judge model which target to use, publishes the choice, then calls the winner.

#[async_trait]
impl Algorithm for JudgeRouter {
    fn name(&self) -> &str { "judge_router" }

    async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<Response> {
        // 1. Ask the judge. This is not the answer, so is_answer_call is false.
        let verdict = driver.call(
            judge_prompt(&request),
            // This Decision explains what you're doing, stays inside CallLlm.
            Decision {
                selected_model: self.judge.clone(),
                reasoning: Some("picking a target".into()),
                is_answer_call: false,
            },
        ).await?;

        // 2. Publish the choice.
        let decision = Decision {
            selected_model: parse_target(&verdict)?,
            reasoning: Some("the judge picked this".into()),
            is_answer_call: true,
        };
        driver.decide(decision.clone()).await?;

        // 3. Call the winner and return its response. Returning ends the
        //    run. libsy publishes this as Step::Done.
        driver.call(request, decision).await
    }
}

---

4. Driving an algorithm

You call run_stream and loop like before.

let mut stream = algorithm.run_stream(ctx, request);
let mut trace = Vec::new();

while let Some(step) = stream.next().await {
    match step? {
        Step::CallLlm(call) => {
            let result = self.client_for(&call.decision.selected_model)
                .call(call.ctx.clone(), call.request.clone())
                .await;
            call.respond(result);
        }
        Step::Decision(decision) => trace.push(decision),
        Step::Done(response)   => return Ok((trace, *response)),
    }
}

Drop the stream at any time and the run stops. Nothing to clean up.

---

5. Bonus: decision-only, for free

A CallLlm does not need responding to. You can instead extract the decision and request, then exit the run_stream loop.

Step::CallLlm   is_answer_call == false   (judge / classifier — answer these)
Step::Decision  the routing choice
Step::CallLlm   is_answer_call == true    ← take it and stop
Step::Done

So take the answer call, don't respond, and return. Dropping the stream ends the run.

while let Some(step) = stream.next().await {
    match step? {
        // The call that would answer. Take it and leave.
        Step::CallLlm(call) if call.decision.is_answer_call => {
            let (_ctx, request, decision) = call.into_parts();
            return Ok((decision, request));
        }
        // A judge call. The algorithm still needs a real reply.
        Step::CallLlm(call) => call.respond(serve(&call).await),
        _ => {}
    }
}

You get the decision and the request as it would have been sent, after any prompt rewriting the algorithm did.

---

6. What goes away

Removed	Lines
The type-erased driver — Box<dyn Any> payloads, downcasts, the take-once receiver, its 15 tests	599
run and run_observed, and the call-serving code inside them	~120
LlmTarget, RoutedRequest, CallLlmRequest, count_tokens	~150
The two-stream select, the tail future, and the three Box::pin layers	~140
trait Decision, tier, as_any, and its 5 implementations	~120
Observer callbacks, the routing-overhead mutex, DriverError, 4 LibsyError variants	~160
Added back: the new Step, Driver, StepStream, Decision	+~155

Net: about 1,000 lines gone.

• No Box<dyn Any> anywhere. Nothing to downcast.
• No trait objects in the step protocol at all.
• No deep request clone per call.
• libsy no longer depends on the client traits in switchyard-protocol.

Two methods on the trait. Two methods on the driver. Three kinds of step. That is the whole API.

---

Switchyard crates/libsy · line counts measured against the tree at 5c25651b.