Repository navigation
fix(cost): estimate gpu_seconds when exact think-bucket timing is missing (gpu_usd=$0 on holo3) - #959
Open
mercurialsolo wants to merge 1 commit into
Open
mercurialsolo wants to merge 1 commit into
mercurialsolo wants to merge 1 commit into
Conversation
…sing Holo3 /v1/predict runs report session_costs.gpu_usd = $0 despite running on an A100/H100. gpu_seconds is owned by the TimeMeter "think" bucket (exact inference wall-time), but that bucket reads ~0 on this path — and because a TimeMeter IS wired, record_step deliberately skips the synthetic steps × per-step fallback. Net: gpu_seconds stays 0 → gpu_cost = $0, so spend tracking silently under-reports GPU cost for the most-used tier. CostMeter.totals() now falls back to the same steps × gpu_seconds_per_step estimate (config-tunable via MANTIS_COST_GPU_SECONDS_PER_STEP) when gpu_seconds == 0 but gpu_steps > 0. The exact path is untouched: when the think bucket recorded real wall-time, that value wins — the estimate never overrides real timing. This is an estimate, not exact billing. The root fix — wiring the think bucket to capture in-container llama.cpp inference on the holo3 path — is a separate, deeper change; this stops the silent $0 in the meantime. Tests: three cases in test_cost_meter.py — estimate fires when timing is missing, exact seconds win over the estimate, and $0 stays $0 with no GPU work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Holo3
/v1/predictruns reportsession_costs.gpu_usd = $0despite running on an A100/H100 — spend tracking silently under-reports GPU cost for the most-used tier (visible across the E1 pilot runs).Root cause:
gpu_secondsis owned by the TimeMeterthinkbucket (exact inference wall-time), but that bucket reads ~0 on this path — and because a TimeMeter is wired,record_stepdeliberately skips the syntheticsteps × per-stepfallback. Net:gpu_secondsstays0→gpu_cost = $0.What
CostMeter.totals()falls back togpu_steps × gpu_seconds_per_step(config-tunable viaMANTIS_COST_GPU_SECONDS_PER_STEP) only whengpu_seconds == 0butgpu_steps > 0. The exact path is untouched — when the think bucket recorded real wall-time, that wins; the estimate never overrides real timing.Caveat
This is an estimate, not exact billing. The exact fix — wiring the
thinkbucket to capture in-container llama.cpp inference on the holo3 path — is a separate, deeper change. This stops the silent$0in the meantime and is config-tunable.Tests
test_cost_meter.py— estimate fires when timing missing; exact seconds win;$0stays$0with no GPU work.40 passed.🤖 Generated with Claude Code