Skip to content

feat: record accumulated LLM cost per automation run - #280

Open
hieptl wants to merge 3 commits into
mainfrom
hieptl/oss-5210
Open

feat: record accumulated LLM cost per automation run#280
hieptl wants to merge 3 commits into
mainfrom
hieptl/oss-5210

Conversation

@hieptl

@hieptl hieptl commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

HUMAN:

I have verified the changes.


AGENT:

Why

Automation runs don't record what they cost, so users can't see what their automations spend or spot the expensive ones. The value already exists inside the sandbox — both preset scripts compute conversation.conversation_stats.get_combined_metrics().accumulated_cost and print it — but it's a local variable that gets thrown away, and automation_runs has no column to hold it.

One correction to the issue as filed: it proposed making the preset scripts add cost to the callback payload, but the preset scripts never build that payload — the SDK does, in Workspace.__exit___send_completion_callback(). The transport half therefore lands in software-agent-sdk; this PR is the service half plus the preset-side call.

Summary

  • Add a nullable cost column to AutomationRun (generic sa.Float, cross-database per AGENTS.md) with migration 012; accept cost on RunCompleteRequest and expose it on AutomationRunResponse.
  • complete_run persists it on both COMPLETED and FAILED — failed runs still spend money.
  • Both preset scripts read the cost in the finally block and hand it to the workspace via register_cost(), so failed runs report their spend too. The read is wrapped in try/except: it falls back to the agent server when no state is cached, and on the failure path that server may already be gone, so it must never mask the original exception.

Issue Number

Resolves #242

How to Test

Automated:

uv run pytest tests/test_router.py -k CompleteRun -q
uv run pytest tests/test_db.py -k Migrations -q

Red/green was verified explicitly — with the source changes stashed and migration 012 moved aside, the new tests fail:

FAILED tests/test_router.py::TestCompleteRun::test_complete_run_saves_cost
FAILED tests/test_router.py::TestCompleteRun::test_complete_run_saves_cost_for_failed_runs
FAILED tests/test_router.py::TestCompleteRun::test_complete_run_without_cost_leaves_it_unset
================= 3 failed, 118 deselected, 1 warning in 2.98s =================

tests/test_db.py: assert "cost" in run_columns
E   AssertionError: assert 'cost' in {'automation_id', 'bash_command_id', 'completed_at',
                                      'conversation_id', 'created_at', 'error_detail', ...}

Full suite with the change applied:

$ uv run pytest tests/ -q
================= 1066 passed, 10 warnings in 70.76s (0:01:10) =================

Migration verified by hand on real SQLite, both directions:

$ AUTOMATION_DB_URL="sqlite+aiosqlite:///$D" uv run alembic upgrade head
--- columns after upgrade ---
[('cost', 'FLOAT')]
head: [('012',)]
--- downgrade -1 ---
cost present after downgrade: False
head: [('011',)]

End-to-end HTTP round trip against the real ASGI app (in-memory SQLite, get_session / authenticate_request overridden the same way tests/conftest.py wires them):

COMPLETED  -> response cost=0.4213  db cost=0.4213
FAILED     -> response cost=1.5
omitted    -> 200, cost stays NULL (back-compat with SDK 1.37.0)
list runs  -> costs=[0.4213, 1.5]

That third line is the compatibility case that matters: the currently pinned SDK (1.37.0) sends no cost field, and the callback must keep succeeding.

Preset scripts: python -m py_compile clean, both files byte-identical in the changed region (diff of the finally block), and every line under the file's de-facto 88-char limit.

Lint/format: uv run ruff check openhands/ migrations/ tests/ and ruff format --check clean.

Video/Screenshots

N/A — backend only, no UI in this PR. Evidence is the round-trip and migration transcripts above.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

@hieptl hieptl self-assigned this Jul 30, 2026
@github-actions github-actions Bot added the type: feat A new feature label Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Coverage

@hieptl
hieptl requested a review from malhotra5 July 30, 2026 16:39
Comment thread openhands/automation/presets/prompt/sdk_main.py Outdated

@malhotra5 malhotra5 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment regarding the workspace API for registering cost

@hieptl
hieptl requested a review from malhotra5 July 31, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record accumulated LLM cost per automation run

2 participants