Skip to content

fix(simulation): make batch-interview result collection fault-tolerant + single-platform report display - #63

Open
MA1503 wants to merge 1 commit into
nikmcfly:mainfrom
MA1503:fix/interview-result-collection
Open

fix(simulation): make batch-interview result collection fault-tolerant + single-platform report display#63
MA1503 wants to merge 1 commit into
nikmcfly:mainfrom
MA1503:fix/interview-result-collection

Conversation

@MA1503

@MA1503 MA1503 commented Jul 24, 2026

Copy link
Copy Markdown

The bug chain

Running a Reddit-only simulation and then interviewing a focus group of
9 agents for the report surfaced a chain of related defects. Only a subset of
agents ever showed a real answer; the rest were silently dropped or displayed a
dead placeholder.

(a) One missing DB entry drops all following agents

In the batch-interview handlers, results are collected in a plain loop:

for agent_id in agent_prompts.keys():
    result = self._get_interview_result(agent_id)   # raises if no DB row
    results[agent_id] = result

If a single agent has no interview row in the DB (e.g. its INTERVIEW action
never landed — context overflow, empty generation), _get_interview_result
raises. In run_reddit_simulation.py / run_twitter_simulation.py the whole
handler falls through to its outer except, so every agent after the failing
one is lost
. run_parallel_simulation.py has the same shape in its Twitter and
Reddit collection loops (separate copies). Observed: 3 of 9 interview results
returned.

Fix: wrap each _get_interview_result call in a per-agent try/except. On
failure, record {"agent_id": ..., "response": "", "error": str(e)} and keep
going instead of aborting. Applied to all collection loops in all three run
scripts.

(b) Dead platform block in single-platform sims

graph_tools.py always builds both platform blocks into the interview
response text:

twitter_text = twitter_response if twitter_response else "(No response from this platform)"
reddit_text  = reddit_response  if reddit_response  else "(No response from this platform)"
response_text = f"[Twitter Platform Response]\n{twitter_text}\n\n[Reddit Platform Response]\n{reddit_text}"

In a Reddit-only sim there is no twitter_* result at all, so every response is
padded with a bogus [Twitter Platform Response] (No response from this platform)
block that confuses downstream display.

Fix: scan the result keys for twitter_ / reddit_ prefixes first and only
emit blocks for platforms that actually returned data. When both are present the
output format is unchanged.

(c) Report UI defaults to the absent platform

frontend/src/components/Step4Report.vue:

  • getPlatformTab hard-defaults to 'twitter', so a Reddit-only interview opens
    on the empty Twitter tab and shows the placeholder.
  • getAnswerForQuestion only falls back Reddit→Twitter, never the reverse — so a
    Twitter tab in a Reddit-only sim stays empty.
  • splitAnswerByQuestions matches Question N: but not Markdown-wrapped headers
    like **Question 1:**, which the backend model frequently emits — multi-question
    answers then collapse into a single blob.

Fixes:

  • Default tab = first platform carrying a real (non-placeholder) answer; fall back
    to 'twitter' only when both are real.
  • Symmetric platform fallback in getAnswerForQuestion (works both directions).
  • Markdown-tolerant question-split regex (**Question 1:**, __Question 1:__,
    > Question 1 :) plus trimming of leftover emphasis markers around extracted
    answers.

Repro

  1. Run a Reddit-only simulation (enable_twitter=false).
  2. Generate the report with a focus-group interview of many agents.
  3. Before this change: several agents silently missing from the results, and the
    ones shown open on the empty Twitter tab with (No response from this platform).
  4. After: every interviewed agent appears (failed ones carry an explicit error),
    the tab defaults to Reddit, and multi-question answers split correctly even
    when the model wraps the headers in Markdown.

Tests

  • ast.parse clean on all three modified run scripts and graph_tools.py.
  • The split function was exercised in isolation against three inputs —
    Question 1: …, **Question 1:** …, and marker-free prose — producing the
    expected ["Alpha","Beta"], ["Alpha","Beta"], and whole-text results
    respectively (the Markdown case previously collapsed to a single blob).

🤖 Generated with Claude Code

…t + single-platform report display

Interviewing a focus group in a single-platform (Reddit-only) simulation
surfaced a chain of related defects that silently dropped agents or showed a
dead placeholder in the report.

- Batch-interview collection is now per-agent fault-tolerant: one agent
  without a DB row no longer aborts the loop and drops every following agent.
  On failure the agent is recorded as {"response": "", "error": ...} instead.
  Applied to all collection loops in run_reddit_simulation,
  run_twitter_simulation and run_parallel_simulation (twitter + reddit copies).
- graph_tools: only emit platform blocks for platforms that actually returned
  results — no more dead "(No response from this platform)" block for the
  absent platform in single-platform sims (format unchanged when both present).
- Step4Report.vue: default the platform tab to the first platform carrying a
  real answer (not hard 'twitter'); symmetric platform fallback in
  getAnswerForQuestion; Markdown-tolerant question-split regex (**Question 1:**).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant