fix(simulation): make batch-interview result collection fault-tolerant + single-platform report display - #63
Open
MA1503 wants to merge 1 commit into
Open
Conversation
…t + single-platform report display
Interviewing a focus group in a single-platform (Reddit-only) simulation
surfaced a chain of related defects that silently dropped agents or showed a
dead placeholder in the report.
- Batch-interview collection is now per-agent fault-tolerant: one agent
without a DB row no longer aborts the loop and drops every following agent.
On failure the agent is recorded as {"response": "", "error": ...} instead.
Applied to all collection loops in run_reddit_simulation,
run_twitter_simulation and run_parallel_simulation (twitter + reddit copies).
- graph_tools: only emit platform blocks for platforms that actually returned
results — no more dead "(No response from this platform)" block for the
absent platform in single-platform sims (format unchanged when both present).
- Step4Report.vue: default the platform tab to the first platform carrying a
real answer (not hard 'twitter'); symmetric platform fallback in
getAnswerForQuestion; Markdown-tolerant question-split regex (**Question 1:**).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug chain
Running a Reddit-only simulation and then interviewing a focus group of
9 agents for the report surfaced a chain of related defects. Only a subset of
agents ever showed a real answer; the rest were silently dropped or displayed a
dead placeholder.
(a) One missing DB entry drops all following agents
In the batch-interview handlers, results are collected in a plain loop:
If a single agent has no interview row in the DB (e.g. its INTERVIEW action
never landed — context overflow, empty generation),
_get_interview_resultraises. In
run_reddit_simulation.py/run_twitter_simulation.pythe wholehandler falls through to its outer
except, so every agent after the failingone is lost.
run_parallel_simulation.pyhas the same shape in its Twitter andReddit collection loops (separate copies). Observed: 3 of 9 interview results
returned.
Fix: wrap each
_get_interview_resultcall in a per-agenttry/except. Onfailure, record
{"agent_id": ..., "response": "", "error": str(e)}and keepgoing instead of aborting. Applied to all collection loops in all three run
scripts.
(b) Dead platform block in single-platform sims
graph_tools.pyalways builds both platform blocks into the interviewresponse text:
In a Reddit-only sim there is no
twitter_*result at all, so every response ispadded with a bogus
[Twitter Platform Response] (No response from this platform)block that confuses downstream display.
Fix: scan the result keys for
twitter_/reddit_prefixes first and onlyemit blocks for platforms that actually returned data. When both are present the
output format is unchanged.
(c) Report UI defaults to the absent platform
frontend/src/components/Step4Report.vue:getPlatformTabhard-defaults to'twitter', so a Reddit-only interview openson the empty Twitter tab and shows the placeholder.
getAnswerForQuestiononly falls back Reddit→Twitter, never the reverse — so aTwitter tab in a Reddit-only sim stays empty.
splitAnswerByQuestionsmatchesQuestion N:but not Markdown-wrapped headerslike
**Question 1:**, which the backend model frequently emits — multi-questionanswers then collapse into a single blob.
Fixes:
to
'twitter'only when both are real.getAnswerForQuestion(works both directions).**Question 1:**,__Question 1:__,> Question 1 :) plus trimming of leftover emphasis markers around extractedanswers.
Repro
enable_twitter=false).ones shown open on the empty Twitter tab with
(No response from this platform).error),the tab defaults to Reddit, and multi-question answers split correctly even
when the model wraps the headers in Markdown.
Tests
ast.parseclean on all three modified run scripts andgraph_tools.py.Question 1: …,**Question 1:** …, and marker-free prose — producing theexpected
["Alpha","Beta"],["Alpha","Beta"], and whole-text resultsrespectively (the Markdown case previously collapsed to a single blob).
🤖 Generated with Claude Code