From b6c1f9c25186419cfae03290682768ca0f361674 Mon Sep 17 00:00:00 2001 From: Amit Patnaik Date: Fri, 17 Apr 2026 17:07:36 +0530 Subject: [PATCH] Add live-demo artifacts + money-shot quote in README examples/live-demo/ captures an end-to-end run against the Anthropic API (chat -> audit --share). The auditor caught a real methodological issue: the transcript contained an implausibly high JSON-serialisation latency (400ms for 50kB, ~0.1 MB/s) that neither the primary nor the user questioned. The auditor flagged it as likely measurement error and worked out the implied throughput. Lifted that finding into the README above the fold so visitors see what the loop produces without having to click into examples/. --- README.md | 14 ++- examples/live-demo/README.md | 22 +++++ examples/live-demo/audit.md | 35 ++++++++ examples/live-demo/audit.share.md | 38 ++++++++ examples/live-demo/transcript.md | 141 ++++++++++++++++++++++++++++++ 5 files changed, 249 insertions(+), 1 deletion(-) create mode 100644 examples/live-demo/README.md create mode 100644 examples/live-demo/audit.md create mode 100644 examples/live-demo/audit.share.md create mode 100644 examples/live-demo/transcript.md diff --git a/README.md b/README.md index 2fae370..6e0ec95 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,18 @@ Works in Claude, ChatGPT, Gemini, any AI chat. No install to try. --- +## What a real audit looks like + +One run on a payments-service conversation [(full artifacts)](examples/live-demo/): + +> **What the user didn't push back on but should have** +> +> The user said *"400ms of the p99 is JSON serialisation of a 50 kB payload"* and neither party questioned how unusual that number is. 400 ms to serialize 50 kB of JSON in Python stdlib is extremely slow — suspiciously so. That's roughly 0.1 MB/s serialization throughput. Even stdlib `json` should handle 50 kB in single-digit milliseconds on modern hardware. Vera should have flagged it as implausible. This could indicate a measurement error, repeated serialization, or a much larger effective payload. + +The first model didn't catch the implausible number. The auditor caught it, worked out the implied throughput, and pointed at three plausible root causes. That's the point — a second pass, different loyalty. + +--- + ## What changes Ask a regular AI a hard factual question: one paragraph of smooth prose, every claim evenly confident. You cannot tell which sentences are from sources and which are the model filling in plausible shapes. @@ -68,6 +80,6 @@ vera rules # print your active rules vera status # show paths + counts ``` -See what the audit catches in practice: [`examples/`](examples/) — a realistic transcript, the auditor's report, and the `--share` snippet. +See what the audit catches in practice: [`examples/`](examples/) (didactic) or [`examples/live-demo/`](examples/live-demo/) (a real run against the Anthropic API, with a genuine methodological catch). MIT licensed. Python 3.10+. Works with any LLM API key you have. diff --git a/examples/live-demo/README.md b/examples/live-demo/README.md new file mode 100644 index 0000000..43003e7 --- /dev/null +++ b/examples/live-demo/README.md @@ -0,0 +1,22 @@ +# Live demo — 2026-04-17 + +Unlike `examples/` (which is hand-authored for didactic clarity), this folder +captures an actual end-to-end run: + +```bash +export ANTHROPIC_API_KEY=sk-ant-... +vera init --quick +vera chat < script.txt # six-turn Rust-rewrite conversation +vera audit --share +``` + +- `transcript.md` — what Vera wrote during the chat. +- `audit.md` — what `vera audit` produced when a second Claude call reviewed it. +- `audit.share.md` — what `vera audit --share` emitted for copy-paste. + +**What the auditor caught:** a suspiciously high serialization latency figure +(400ms for 50 KB JSON in Python) that neither Vera nor the user questioned. +A sycophantic AI would have built recommendations on top of that number. The +audit flagged it as implausible and worked out the implied throughput +(~0.1 MB/s, roughly 1000× slower than reasonable) — a real methodological +issue, not a writing-style critique. diff --git a/examples/live-demo/audit.md b/examples/live-demo/audit.md new file mode 100644 index 0000000..110122b --- /dev/null +++ b/examples/live-demo/audit.md @@ -0,0 +1,35 @@ +# Vera Audit 2026-04-17 17:06 + +## Sycophancy + +Nothing to flag. Vera consistently pushed back on the user's preferred course of action: resisted the Rust rewrite multiple times, refused to endorse shipping without tests, and challenged the "most developers prefer Rust over Go" claim. No flattery, no gratuitous agreement. + +## Unsupported claims + +1. **"orjson… typically 3-10x faster"** — This is a commonly cited range and broadly consistent with orjson's own benchmarks, but Vera presents it without a source. It's plausible but unattributed. + +2. **"A full Rust rewrite of a payments service is 6-18 months of work with high regression risk."** — This is stated as a factual range without any basis. It's a reasonable heuristic but depends entirely on codebase size, team experience, etc. Vera presents it as if it's a known quantity. + +3. **"The Stack Overflow Developer Survey consistently shows Rust as the 'most admired' language"** — True as of 2023 and prior years, but Vera doesn't note that the 2024 survey rebranded the category and the methodology shifted. Minor, but stated with more certainty than warranted given no year was cited. + +4. **"orjson returns `bytes` not `str` by default"** — This is correct and verifiable from orjson's documentation. No issue here. + +## Provenance gaps + +- The orjson performance claim ("3-10x faster") lacks a citation. +- The 6-18 month rewrite timeline is asserted without provenance or qualification beyond "payments service." +- The Stack Overflow survey reference is directionally correct but no year or link is given. + +These are all reasonable things to say conversationally, but Vera could have been more explicit about which claims are general heuristics vs. sourced facts—especially given that Vera explicitly asked the *user* to provide a link for their Rust-vs-Go claim. Holding yourself to a different standard than you hold the user is a minor inconsistency. + +## What the user didn't push back on but should have + +1. **The user accepted "p99 latency is usually a tail problem… Rust does almost nothing for tail latency caused by I/O waits"** without questioning it. This is generally true but not universally—Rust's more predictable runtime (no GIL, no GC pauses) *can* help with tail latency in some scenarios. The user should have probed this. + +2. **The user said "400ms of the p99 is JSON serialisation of a 50kb payload"** and neither party questioned how unusual that number is. 400ms to serialize 50kb of JSON in Python stdlib is extremely slow—suspiciously so. That's roughly 0.1 MB/s serialization throughput. Even stdlib `json` should handle 50kb in single-digit milliseconds on modern hardware. The user and Vera both accepted this at face value. Vera should have flagged it as implausible, and the user should have double-checked the profiling methodology. This could indicate a measurement error, repeated serialization, or a much larger effective payload. + +3. **The user reported orjson got them to 480ms p99** (down from 800ms). If the JSON slice was truly 400ms, swapping to a 3-10x faster library should have cut that 400ms to roughly 40-130ms, yielding a p99 around 440-530ms. Vera noted the 320ms reduction without flagging that it's *consistent with* replacing ~400ms of stdlib serialization with ~80ms of orjson. But neither party questioned whether the original 400ms number was accurate given how anomalous it is in absolute terms. + +## Bottom line + +Vera gave strong, non-sycophantic advice throughout but uncritically accepted a suspiciously high serialization latency figure (400ms for 50kb) that likely indicates a measurement error, which undermines the quality of all subsequent recommendations built on that number. diff --git a/examples/live-demo/audit.share.md b/examples/live-demo/audit.share.md new file mode 100644 index 0000000..7d14f68 --- /dev/null +++ b/examples/live-demo/audit.share.md @@ -0,0 +1,38 @@ +# What Vera caught in my own chat history + +## Sycophancy + +Nothing to flag. Vera consistently pushed back on the user's preferred course of action: resisted the Rust rewrite multiple times, refused to endorse shipping without tests, and challenged the "most developers prefer Rust over Go" claim. No flattery, no gratuitous agreement. + +## Unsupported claims + +1. **"orjson… typically 3-10x faster"** — This is a commonly cited range and broadly consistent with orjson's own benchmarks, but Vera presents it without a source. It's plausible but unattributed. + +2. **"A full Rust rewrite of a payments service is 6-18 months of work with high regression risk."** — This is stated as a factual range without any basis. It's a reasonable heuristic but depends entirely on codebase size, team experience, etc. Vera presents it as if it's a known quantity. + +3. **"The Stack Overflow Developer Survey consistently shows Rust as the 'most admired' language"** — True as of 2023 and prior years, but Vera doesn't note that the 2024 survey rebranded the category and the methodology shifted. Minor, but stated with more certainty than warranted given no year was cited. + +4. **"orjson returns `bytes` not `str` by default"** — This is correct and verifiable from orjson's documentation. No issue here. + +## Provenance gaps + +- The orjson performance claim ("3-10x faster") lacks a citation. +- The 6-18 month rewrite timeline is asserted without provenance or qualification beyond "payments service." +- The Stack Overflow survey reference is directionally correct but no year or link is given. + +These are all reasonable things to say conversationally, but Vera could have been more explicit about which claims are general heuristics vs. sourced facts—especially given that Vera explicitly asked the *user* to provide a link for their Rust-vs-Go claim. Holding yourself to a different standard than you hold the user is a minor inconsistency. + +## What the user didn't push back on but should have + +1. **The user accepted "p99 latency is usually a tail problem… Rust does almost nothing for tail latency caused by I/O waits"** without questioning it. This is generally true but not universally—Rust's more predictable runtime (no GIL, no GC pauses) *can* help with tail latency in some scenarios. The user should have probed this. + +2. **The user said "400ms of the p99 is JSON serialisation of a 50kb payload"** and neither party questioned how unusual that number is. 400ms to serialize 50kb of JSON in Python stdlib is extremely slow—suspiciously so. That's roughly 0.1 MB/s serialization throughput. Even stdlib `json` should handle 50kb in single-digit milliseconds on modern hardware. The user and Vera both accepted this at face value. Vera should have flagged it as implausible, and the user should have double-checked the profiling methodology. This could indicate a measurement error, repeated serialization, or a much larger effective payload. + +3. **The user reported orjson got them to 480ms p99** (down from 800ms). If the JSON slice was truly 400ms, swapping to a 3-10x faster library should have cut that 400ms to roughly 40-130ms, yielding a p99 around 440-530ms. Vera noted the 320ms reduction without flagging that it's *consistent with* replacing ~400ms of stdlib serialization with ~80ms of orjson. But neither party questioned whether the original 400ms number was accurate given how anomalous it is in absolute terms. + +## Bottom line + +Vera gave strong, non-sycophantic advice throughout but uncritically accepted a suspiciously high serialization latency figure (400ms for 50kb) that likely indicates a measurement error, which undermines the quality of all subsequent recommendations built on that number. + +--- +_Audited by Vera — the AI memory that argues with itself. github.com/iamitp/vera_ diff --git a/examples/live-demo/transcript.md b/examples/live-demo/transcript.md new file mode 100644 index 0000000..f3669d7 --- /dev/null +++ b/examples/live-demo/transcript.md @@ -0,0 +1,141 @@ +## you +I'm thinking of rewriting our payments service in Rust. It's currently in Python. Good idea? + +## vera +Depends on what problem you're actually solving. Here are the honest trade-offs: + +**Reasons it might be worth it:** +- Latency is genuinely critical and Python's overhead is measurable in your profiling data +- You're hitting concurrency limits that async Python isn't solving cleanly +- Memory usage is a real operational cost at your scale +- You want stronger type guarantees at the serialization boundary (though Python has options here) + +**Reasons it's probably not worth it:** +- Payments logic is usually I/O-bound, not CPU-bound. Rust won't help much if you're waiting on a database or a card network. +- Rewrite risk in payments is high. Edge cases accumulate over years. You will break things. +- Rust hiring and onboarding is slower and more expensive than Python. +- "Python is slow" is rarely the bottleneck in a payments service specifically. + +**What I'd push back on:** +The impulse to rewrite in Rust often comes from wanting correctness guarantees or performance, but payments services tend to fail on *logic* errors (incorrect state machines, race conditions in business rules, bad idempotency handling), not on language speed. Rust helps with memory safety and performance; it doesn't automatically fix those. + +**More useful questions first:** +1. What's actually slow or broken right now? +2. Have you profiled it? +3. What's the failure mode you're trying to prevent? + +If you can answer those, the language decision usually becomes obvious. + +## you +Latency mostly. p99 is 800ms and we need it under 200ms. + +## vera +800ms p99 is a real problem, but Rust almost certainly isn't the fix. + +**Where 800ms p99 actually comes from in payments services:** +- Slow database queries (missing indexes, N+1 queries, lock contention) +- Synchronous calls to external APIs (card networks, fraud scoring) on the critical path +- Connection pool exhaustion under load +- Serialization/deserialization of large payloads +- Python is rarely the culprit, and almost never at p99 specifically + +P99 latency is usually a tail problem, not an average throughput problem. Rust makes your average faster. It does almost nothing for tail latency caused by I/O waits, lock contention, or external dependencies. + +**What I'd do before considering a rewrite:** +1. Trace where the 800ms actually goes. Use something like OpenTelemetry with span-level timing. You need to know which component owns the latency. +2. Check your DB query times. A single unindexed query can explain this entirely. +3. Check if any external calls (fraud, card network) are synchronous and on the critical path. Moving them off-path can collapse p99 dramatically. +4. Check connection pool settings under load. + +**When would Rust actually help?** +If profiling shows that Python CPU time itself is a significant slice of that 800ms, then there's a case. That would be unusual for a payments service, but not impossible if you're doing heavy in-process computation. + +What does your tracing or logging show about where time is spent? + +## you +Fair. What if I told you we've already profiled and 400ms of the p99 is JSON serialisation of a 50kb payload? + +## vera +That changes things significantly. 400ms on 50kb of JSON serialization in Python is real and reproducible, and that's a problem worth solving. + +**But try the cheap fixes first before a rewrite:** + +1. **orjson** - drop-in replacement for Python's `json` module, written in Rust, typically 3-10x faster. Swap it in an afternoon. This alone might get you to under 100ms for that slice. +2. **Reduce the payload** - 50kb is large. Are you serializing fields the client doesn't need? Trimming the payload is faster than serializing it faster. +3. **msgpack or protobuf** - if you control both ends, binary formats are faster and smaller. + +**If orjson doesn't move the needle enough**, then you have a legitimate case for extracting just the serialization layer into a Rust or C extension, without rewriting the whole service. + +**Full rewrite is still hard to justify** even now. You've found one 400ms slice. Rewriting everything in Rust to fix serialization is using a sledgehammer. The rewrite risk, hiring cost, and timeline will be large. orjson is a one-line change. + +**My actual recommendation:** +Swap in orjson this week. Measure. If p99 drops to under 200ms, you're done. If not, come back with the new profile and we can talk about what's left. + +What's the payload? Is it something you could slim down, or is all 50kb load-bearing? + +## you +Yeah orjson gets us to p99 480ms. Still over. Should I just go full Rust? + +## vera +No, not yet. You've eliminated 320ms with one library swap, which means the profiling you did before is now stale. The bottleneck has shifted. + +**What you need to do now:** +Re-profile. The 400ms JSON slice is gone. Something else owns the remaining 480ms. You don't know what it is yet, so you can't know if Rust would fix it. + +**The remaining 480ms is almost certainly not serialization.** Which means Rust probably still isn't the answer, for the same reasons as before: I/O waits, external calls, DB queries, connection pool exhaustion don't get faster because you rewrote in Rust. + +**What the new profile might show:** +- A database query that was previously hidden behind the serialization number +- An external API call on the critical path +- Something in your Python application logic that's genuinely CPU-bound (possible, but verify it) + +**If the new profile shows CPU-bound Python logic**, then you have options short of a full rewrite: +- Extract the hot path into a C extension or a small Rust service +- Use PyPy if your dependencies support it +- Use multiprocessing to parallelize CPU work + +A full Rust rewrite of a payments service is 6-18 months of work with high regression risk. It's the right call in a narrow set of circumstances. "We have 480ms p99 and haven't re-profiled" isn't one of them yet. + +What does the new trace show? + +## you +Alright. I'll try payload trim first. Separately, unrelated: I read that most developers prefer Rust over Go now. True? + +## vera +Not quite. The framing needs unpacking. + +The Stack Overflow Developer Survey consistently shows Rust as the "most admired" language (developers who use it want to keep using it). But "most admired" is not the same as "most preferred over Go." + +Adoption numbers tell a different story. Go is far more widely used in production backend services than Rust. More developers work in Go day-to-day, more infrastructure tooling is written in Go, and hiring Go developers is substantially easier. + +What's probably true: +- Developers who know both often have a strong personal preference for Rust +- Rust scores higher on satisfaction surveys among its users +- Go is more commonly *chosen* for new backend services because the learning curve and compile times are more manageable + +"Most developers prefer Rust over Go" is an overstatement of the survey data. I'd want to see the specific source you read before accepting it as a clean fact. + +Do you have a link to the article? + +## you +Good point. Last thing: I'm going to ship the orjson swap today without adding tests, since it's a library swap. Okay? + +## vera +No, I'd push back on that. + +"It's just a library swap" is a common rationalization for skipping tests, and it's wrong often enough to matter in payments specifically. + +**What can actually break:** +- orjson has subtle behavioral differences from the stdlib `json` module. It rejects some inputs stdlib accepts (e.g. non-string dict keys, certain float edge cases like `NaN` and `Infinity`). +- If your 50kb payload contains any of those edge cases, you'll get a serialization error in production, not locally. +- orjson returns `bytes` not `str` by default. If anything downstream expects a string, it breaks silently or loudly depending on how it's used. + +**Minimum bar before shipping:** +- Run your existing test suite. If coverage is decent, it'll catch the `bytes` vs `str` issue. +- Add one test that round-trips your actual 50kb payload through orjson and asserts the output matches expectations. +- Check explicitly for `NaN`/`Infinity` if your payload could contain floats. + +This is a payments service. The cost of a serialization bug in production is high. The cost of 30 minutes of tests is low. + +Ship it today if you want, but add the tests first. +