Skip to content

observability: window traps that read as results, and a per-game watch list #176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mstan merged 1 commit into from
Aug 23, 2026
Merged

observability: window traps that read as results, and a per-game watch list #176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 50 additions & 28 deletions TCP_COMMANDS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,30 +92,30 @@ Columns: **N** = native, **D** = DuckStation oracle.
| `pc_hit_clear` | | ✓² | — | Clear the last-hit record |
| `quit` | ✓ | | — | Shutdown native runtime |

¹ Native `vram_peek` is the legacy name; DS calls it `read_vram`. Same semantics.
² The `pc_*` family is specific to the DS oracle: DuckStation's CPU core honours `CPU::AddBreakpointWithCallback`, while our native runtime dispatches whole recompiled functions (no mid-function PC breaks).

### Boot-time write ranges

Set `PSX_WTRACE_BOOT=lo,hi[;lo,hi...]` before launching a debug-tools build to
retain the first writes to one or more half-open RAM ranges from guest
instruction zero. Addresses may be hexadecimal or decimal; KSEG addresses are
normalized to physical addresses. For example, the Crash Bash investigation
that motivated this option can be reproduced without title-specific code:

```powershell
$env:PSX_WTRACE_BOOT='0x000B3A80,0x000B3B00'
.\CrashBashRecomp.exe
```

Connect at any later point and query `wtrace_boot_stats`,
`wtrace_boot_summary`, or `wtrace_boot_dump`. Each retained entry includes the
write address/value/width, guest PC and return address, register context, frame,
and DMA channel. The option is ignored in builds made with debug tools disabled.

---

## Divergence-hunt workflow
¹ Native `vram_peek` is the legacy name; DS calls it `read_vram`. Same semantics.
² The `pc_*` family is specific to the DS oracle: DuckStation's CPU core honours `CPU::AddBreakpointWithCallback`, while our native runtime dispatches whole recompiled functions (no mid-function PC breaks).
### Boot-time write ranges
Set `PSX_WTRACE_BOOT=lo,hi[;lo,hi...]` before launching a debug-tools build to
retain the first writes to one or more half-open RAM ranges from guest
instruction zero. Addresses may be hexadecimal or decimal; KSEG addresses are
normalized to physical addresses. For example, the Crash Bash investigation
that motivated this option can be reproduced without title-specific code:
```powershell
$env:PSX_WTRACE_BOOT='0x000B3A80,0x000B3B00'
.\CrashBashRecomp.exe
```
Connect at any later point and query `wtrace_boot_stats`,
`wtrace_boot_summary`, or `wtrace_boot_dump`. Each retained entry includes the
write address/value/width, guest PC and return address, register context, frame,
and DMA channel. The option is ignored in builds made with debug tools disabled.
---
## Divergence-hunt workflow

When a recompiled-BIOS bug is suspected, the two servers let you find the **first** divergence instead of chasing symptoms. Standard procedure (inherited from v3's `DEBUG.md`):

Expand Down Expand Up @@ -214,7 +214,16 @@ ring names the return path that let it come back.
`sp_b`/`ra_b`/`s0_b`/`s3_b`, post-call `pc_a`/`ra_a`/`sp_a`/`s0_a`/
`s3_a`/`v0_a`, `bail`/`rfe`/`esc`/`in_exc` flags, `dstatic`/`dblocks`/
`dexc` engine-attribution deltas across the call, `last_func`.
- `{"cmd":"callret_watch","lo":"0"}` — disarm.
- `{"cmd":"callret_watch","disarm":true}` — disarm. The legacy
`{"lo":"0"}` spelling (no `hi`) still disarms and says so in the reply.

The ring is armed iff the window is NON-EMPTY (`hi > lo`), so `lo` may be
`0` — `{"lo":"0","hi":"0x200000"}` really does watch the whole address
space. Every reply carries `armed`, and the dump carries `armed` plus the
active window, because an unarmed ring reports `total: 0` — which otherwise
reads exactly like a genuine "those calls never happened" answer. Passing
`lo` with no `hi` is refused rather than silently inheriting the previous
ceiling (usually `0`, i.e. an empty window).

## `hle_dump` — BIOS-HLE tier call ring (native only)

Expand Down Expand Up @@ -263,9 +272,9 @@ The TCP server is the canonical instrumentation surface. Rule 3 in `CLAUDE.md` i

## Complete command index (generated)

**292 commands registered** — 279 on the native server (`runtime/src/debug_server.c`), 61 on the Beetle server (`runtime/src/beetle_debug_server.c`).

47 of 292 have prose above; **245 are index-only**. An index-only command still works — it just has no description here yet. Send it `{"cmd":"<name>"}` and read the reply, or find its `handle_*` function in the server source.
**305 commands registered** — 292 on the native server (`runtime/src/debug_server.c`), 61 on the Beetle server (`runtime/src/beetle_debug_server.c`).
49 of 305 have prose above; **256 are index-only**. An index-only command still works — it just has no description here yet. Send it `{"cmd":"<name>"}` and read the reply, or find its `handle_*` function in the server source.

Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this block has drifted from the code.

Expand All @@ -289,6 +298,7 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `card_buffer_dump` | ✓ | | |
| `card_data_writes` | ✓ | | |
| `card_data_writes_reset` | ✓ | | |
| `card_handoff` | ✓ | | |
| `card_mgr_clear` | ✓ | | |
| `card_mgr_trace` | ✓ | | |
| `card_read_summary` | ✓ | | |
Expand Down Expand Up @@ -379,6 +389,7 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `frame_timeseries` | ✓ | ✓ | ✓ |
| `freeze_check` | ✓ | | |
| `game_options` | ✓ | | |
| `geom_correction` | ✓ | | ✓ |
| `get_frame` | ✓ | ✓ | ✓ |
| `get_quads` | ✓ | | |
| `get_registers` | ✓ | ✓ | ✓ |
Expand All @@ -404,6 +415,11 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `hle_dump` | ✓ | | ✓ |
| `idle_skip` | ✓ | | |
| `imask_trace` | ✓ | | |
| `input_route_append` | ✓ | | |
| `input_route_clear` | ✓ | | |
| `input_route_start` | ✓ | | |
| `input_route_status` | ✓ | | |
| `input_route_stop` | ✓ | | |
| `insn_freeze` | ✓ | | |
| `insn_freeze_snapshot` | ✓ | | |
| `insn_freeze_status` | ✓ | | |
Expand Down Expand Up @@ -447,6 +463,10 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `parity_ctl` | ✓ | ✓ | |
| `parity_dump` | ✓ | ✓ | |
| `pause` | ✓ | | ✓ |
| `pc_probe_arm` | ✓ | | |
| `pc_probe_clear` | ✓ | | |
| `pc_probe_dump` | ✓ | | |
| `pgxp` | ✓ | | |
| `phase_hot` | ✓ | | |
| `phase_profile` | ✓ | | |
| `ping` | ✓ | ✓ | ✓ |
Expand Down Expand Up @@ -477,6 +497,7 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `savestate` | ✓ | | |
| `screenshot` | ✓ | ✓ | ✓ |
| `screenshot_file` | ✓ | ✓ | ✓ |
| `screenshot_hires` | ✓ | | ✓ |
| `set_input` | ✓ | ✓ | ✓ |
| `set_snapshot` | ✓ | ✓ | ✓ |
| `sio_arm_audit` | ✓ | | |
Expand Down Expand Up @@ -563,5 +584,6 @@ Regenerate with `python tools/gen_tcp_commands.py`; `--check` fails if this bloc
| `xlate` | ✓ | | |
| `xprobe` | ✓ | | |
| `xprobe_arm` | ✓ | | |
| `xprobe_watch` | ✓ | | |

<!-- END AUTOGENERATED COMMAND INDEX -->
105 changes: 96 additions & 9 deletions runtime/src/debug_server.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12111,12 +12111,16 @@ static void handle_s3_smear_watch(int id, const char *json)
? hex_to_u32(buf) : 0u;
g_s3_smear_valid = 0;
}
send_fmt("{\"id\":%d,\"ok\":true,\"lo\":\"0x%08X\",\"hi\":\"0x%08X\","
/* `armed` so a never-tripped watch (valid:0) can be told apart from one
* that was never recording — the same ambiguity the callret ring had. */
send_fmt("{\"id\":%d,\"ok\":true,\"armed\":%s,"
"\"lo\":\"0x%08X\",\"hi\":\"0x%08X\","
"\"excl\":\"0x%08X\","
"\"valid\":%d,\"pc\":\"0x%08X\",\"insn\":\"0x%08X\","
"\"s3_old\":\"0x%08X\",\"s3_new\":\"0x%08X\","
"\"call_target\":\"0x%08X\",\"frame\":%u}\n",
id, g_s3_smear_lo, g_s3_smear_hi, g_s3_smear_excl,
id, (g_s3_smear_hi > g_s3_smear_lo) ? "true" : "false",
g_s3_smear_lo, g_s3_smear_hi, g_s3_smear_excl,
g_s3_smear_valid,
g_s3_smear_pc, g_s3_smear_insn, g_s3_smear_old, g_s3_smear_new,
g_s3_smear_tgt, g_s3_smear_frame);
Expand All @@ -12141,25 +12145,74 @@ static void handle_callret_watch(int id, const char *json)
uint32_t last_func_a;
} E;
extern uint32_t g_callret_lo, g_callret_hi;
extern int callret_armed(void);
extern E g_callret_ring[]; extern uint64_t g_callret_seq;
const uint32_t cap = 64u; /* MUST match CALLRET_CAP (dirty_ram_interp.c) */
char buf[32];
/* Explicit disarm, and the documented legacy spelling for it. `lo` with no
* `hi` used to mean "disarm" only because the gate tested lo != 0; keep
* that contract for lo=0 (scripts rely on it) but answer it explicitly
* instead of silently. */
const int want_disarm = json_get_int(json, "disarm", 0) != 0;
if (want_disarm) {
g_callret_lo = g_callret_hi = 0;
g_callret_seq = 0;
send_fmt("{\"id\":%d,\"ok\":true,\"armed\":false,\"lo\":\"0x00000000\","
"\"hi\":\"0x00000000\"}\n", id);
return;
}
if (json_get_str(json, "lo", buf, sizeof(buf))) {
g_callret_lo = hex_to_u32(buf);
if (json_get_str(json, "hi", buf, sizeof(buf)))
g_callret_hi = hex_to_u32(buf);
const uint32_t new_lo = hex_to_u32(buf);
char hibuf[32];
const int have_hi = json_get_str(json, "hi", hibuf, sizeof(hibuf)) != NULL;
if (!have_hi) {
/* Legacy disarm: {"lo":"0"} with no hi. Honour it, name it. */
if (new_lo == 0) {
g_callret_lo = g_callret_hi = 0;
g_callret_seq = 0;
send_fmt("{\"id\":%d,\"ok\":true,\"armed\":false,"
"\"lo\":\"0x00000000\",\"hi\":\"0x00000000\","
"\"note\":\"disarmed (legacy lo=0 spelling; prefer"
" disarm=true)\"}\n", id);
return;
}
/* A floor with no ceiling silently inherited the previous hi —
* usually 0, i.e. an empty window that recorded nothing while
* replying ok. Refuse instead of guessing. */
send_fmt("{\"id\":%d,\"ok\":false,\"error\":\"lo without hi\","
"\"armed\":%s,\"lo\":\"0x%08X\",\"hi\":\"0x%08X\","
"\"note\":\"pass BOTH lo and hi (hi > lo) to arm, or"
" disarm=true to stop; window left unchanged\"}\n",
id, callret_armed() ? "true" : "false",
g_callret_lo, g_callret_hi);
return;
}
g_callret_lo = new_lo;
g_callret_hi = hex_to_u32(hibuf);
g_callret_seq = 0;
send_fmt("{\"id\":%d,\"ok\":true,\"lo\":\"0x%08X\",\"hi\":\"0x%08X\"}\n",
id, g_callret_lo, g_callret_hi);
/* Report `armed` rather than leaving the caller to infer it: an empty
* window records nothing, and "total: 0" from an unarmed ring reads
* exactly like a real "these events never happened" answer. */
const int armed = callret_armed();
send_fmt("{\"id\":%d,\"ok\":true,\"lo\":\"0x%08X\",\"hi\":\"0x%08X\","
"\"armed\":%s%s}\n",
id, g_callret_lo, g_callret_hi, armed ? "true" : "false",
armed ? ""
: ",\"note\":\"empty window (hi <= lo) - NOT recording\"");
return;
}
uint64_t total = g_callret_seq;
uint32_t avail = total < cap ? (uint32_t)total : cap;
size_t BUF_SZ = 512u + (size_t)avail * 512u;
char *out = (char *)malloc(BUF_SZ); if (!out) { send_err(id, "oom"); return; }
/* Carry the window and armed state on the READ too: "total": 0 is
* otherwise indistinguishable between "armed, nothing matched" and "never
* armed", and the second reads as evidence that the events did not happen. */
size_t pos = (size_t)snprintf(out, BUF_SZ,
"{\"id\":%d,\"ok\":true,\"total\":%llu,\"entries\":[",
id, (unsigned long long)total);
"{\"id\":%d,\"ok\":true,\"armed\":%s,\"lo\":\"0x%08X\",\"hi\":\"0x%08X\","
"\"total\":%llu,\"entries\":[",
id, callret_armed() ? "true" : "false", g_callret_lo, g_callret_hi,
(unsigned long long)total);
for (uint32_t i = 0; i < avail && pos < BUF_SZ - 600; i++) {
E *e = &g_callret_ring[(total - avail + i) & (cap - 1u)];
pos += (size_t)snprintf(out + pos, BUF_SZ - pos,
Expand Down Expand Up @@ -12806,6 +12859,39 @@ static void handle_ce_profile(int id, const char *json)

/* Arm the §18 boundary trip at runtime once the idle baseline is known:
* xprobe_arm {"frame_trip":N,"stk_kb":K,"warmup":F}. Any 0 disables that arm. */
/* "xprobe_watch": get or set the JAL/JALR watched-target list that filters the
* xprobe `watched` dump. Previously a hardcoded list of one title's addresses,
* so every other game silently watched nothing. Pass `targets` (comma /
* semicolon / space separated, 0x or decimal) to replace the list; pass an
* empty string to clear it; pass nothing to read it back. Also settable before
* the process starts via PSX_XPROBE_WATCH, so a target can be watched from
* instruction zero. Always reports the active list, so an empty `watched` dump
* is never ambiguous between "not called" and "not watched". */
static void handle_xprobe_watch(int id, const char *json)
{
extern void dirty_ram_xprobe_watch_set(const char *spec);
extern int dirty_ram_xprobe_watch_get(int index, uint32_t *phys_out);
extern int dirty_ram_xprobe_watch_count(void);
char spec[512];
if (json_get_str(json, "targets", spec, sizeof(spec)))
dirty_ram_xprobe_watch_set(spec);
char list[768];
size_t pos = 0;
const int n = dirty_ram_xprobe_watch_count();
for (int i = 0; i < n && pos < sizeof(list) - 24; i++) {
uint32_t phys = 0;
if (!dirty_ram_xprobe_watch_get(i, &phys)) break;
pos += (size_t)snprintf(list + pos, sizeof(list) - pos, "%s\"0x%08X\"",
i ? "," : "", phys);
}
list[pos] = '\0';
send_fmt("{\"id\":%d,\"ok\":true,\"count\":%d,\"watching\":[%s]%s}",
id, n, list,
n ? "" : ",\"note\":\"no targets watched - the xprobe watched dump"
" will be empty regardless of what the game calls; pass"
" targets=0x... to watch\"");
}

static void handle_xprobe_arm(int id, const char *json)
{
extern void dirty_ram_xprobe_arm(int frame_trip, int stk_kb, int warmup);
Expand Down Expand Up @@ -13241,6 +13327,7 @@ static const CmdEntry s_commands[] = {
{ "stack_profile", handle_stack_profile },
{ "xprobe", handle_xprobe },
{ "xprobe_arm", handle_xprobe_arm },
{ "xprobe_watch", handle_xprobe_watch },
{ "ce_profile", handle_ce_profile },
{ "frame", handle_frame },
{ "frame_fingerprint", handle_frame_fingerprint },
Expand Down
Loading