Skip to content

feat: production mode (budgets, cost, events, OpenTelemetry, parallel tools, replay) - #10

Merged
leofmarciano merged 2 commits into
mainfrom
devin/1793686716-production
Nov 3, 2026
Merged

leofmarciano merged 2 commits into
mainfrom
devin/1793686716-production

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Makes weightspace ready to run as a service against OpenAI / Anthropic, without changing the public API.

  • Results you can act on: Result.stop_reason, ok, tokens (usage normalized across providers, incl. prompt-caching reads/writes and reasoning), cost (with pricing=), run_id, timings, error.

  • Budgets: token_budget, cost_budget, time_budget. When one runs out the LLM gets a last turn to submit.

  • No lost work: Ctrl-C returns a partial Result; API failures raise SessionError with the partial result on .result.

  • Observability: run events + callbacks= (LoggingCallback, EventLog, TranscriptWriter, ConsoleLogger) and weightspace.otel.OpenTelemetryCallback with GenAI semantic-convention spans (invoke_agent / chat / execute_tool).

  • Parallel tools: read-only tool calls from one turn run on a thread pool (wsp.tool(..., parallel=True) for custom ones). Writes, Python and probes stay sequential.

  • Providers: max_retries / timeout on all backends, Anthropic streaming for long requests, OpenAI prompt-caching key set per run on api.openai.com, parallel_tool_calls / safety_identifier pass through.

  • Replay: ScriptedLLM.from_transcript(...) (the README promised this but it did not exist), CLI --replay.

  • Sandbox: timeout now works off the main thread and without SIGALRM; code runs under torch.no_grad().

  • Perf: diff() / patch() in 1M-element blocks, describe_tensor memoized and uses a Gram spectrum for tall/wide matrices.

    64M-weight embedding, CPU before after
    ws.diff() after a small edit 739 ms, 1017 MB peak 129 ms, 18 MB peak
    singular values, 16384 x 1024 819 ms 117 ms
  • CLI: --probe FILE.py:FN, --forward-budget, --price, --max-cost, --max-tokens, --max-time, --events, --json, --version.

  • agent.py split into tools.py, result.py, events.py, usage.py. New docs/production.md, py.typed, version 0.3.0 (main already shipped 0.2.0).

How it was tested

  • ruff check . && ruff format --check .
  • pytest (45 passed, 18 new in tests/test_production.py), mypy src/weightspace, python -m build + twine check --strict
  • Added a CHANGELOG.md entry (if user-facing)
  • Live on anthropic:claude-opus-5-5: toy fix task (broken entry in a 4x4 layer), fixed with a one-weight edit, tokens, prompt-caching and cost reported ($0.025), plus a streaming run with max_tokens=32000.

…llel tools and replay

Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
@leofmarciano
leofmarciano merged commit a987a9c into main Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant