docs(base44-troubleshooter): realtime logs --follow + a route to workflows runs

skills/base44-troubleshooter/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: base44-troubleshooter
-description: Troubleshoot production issues using backend function logs. Use when investigating app errors, debugging function calls, or diagnosing production problems in Base44 apps.
+description: Troubleshoot production issues using backend function logs and workflow run history. Use when investigating app errors, debugging function calls, diagnosing why a scheduled job or automation failed, or diagnosing production problems in Base44 apps.
 ---
 
 # Troubleshoot Production Issues
@@ -30,18 +30,93 @@ npx base44 logs --app-id app_123
 | Command | Description | Reference |
 |---------|-------------|-----------|
 | `base44 logs` | Fetch function logs for this app | [project-logs.md](references/project-logs.md) |
+| `base44 workflows runs` | List workflow runs, newest first; failed runs carry the failing task and the underlying error | [workflows-runs.md](../base44-cli/references/workflows-runs.md) |
+| `base44 workflows list` | List this app's workflows with status and run summary | [workflows-list.md](../base44-cli/references/workflows-list.md) |
+
+## Logs are not read-after-write
+
+**A single fetch that misses your run proves nothing.** One-shot `base44 logs` reads
+an index that lags behind the invocation, so an empty result right after triggering a
+function is the expected result, not evidence of a problem.
+
+- **Live debugging: use `--follow`.** Where the realtime stream is available it
+  delivers lines in **under a second**. Where it is not, the CLI says so and polls
+  instead (`Realtime logs are not available for this app — falling back to polling
+  (lines may lag ~20-30s).`). Either way `--follow` is the right tool — you never
+  have to pick.
+- **One-shot fetches lag ~20-30s.** That is ingestion time, not a filter problem.
+- **When output is empty, the variable to change is TIME, never a flag.** Wait and
+  re-run the same command. Widening `--limit`, dropping `--level`, or switching
+  `--order` changes nothing about a line that has not been ingested yet, and
+  re-rolling flags is how agents talk themselves into a wrong diagnosis.
 
 ## Troubleshooting Flow
 
-### 1. Check Recent Errors
+### 1. Watch it happen — `--follow`
+
+If you can trigger the failure (or it is happening now), start here rather than
+fetching after the fact:
+
+```bash
+npx base44 logs --follow
+npx base44 logs --follow --function <function_name>
+```
+
+Then invoke the function and read what arrives. Rules that keep you from misreading
+a healthy stream:
+
+- **Never decide on a timer.** Open the stream, trigger the function, and read until
+  you see the lines — do not read a fixed window, print, and conclude "broken".
+  A quiet stream is quiet because nothing has been invoked.
+- **Delivery is per-invocation, not per-line.** A long-running function's lines all
+  arrive when the invocation ends. Silence mid-invocation is normal.
+- **Redeploying mid-follow is safe.** A deploy rotates the script in seconds and the
+  same open stream delivers the new code's lines on the next invoke. Do not tear the
+  stream down and rebuild it after every deploy.
+- **`--since` is rejected with `--follow`** (the stream starts from now), as are
+  `--until` and `--order`. For anything historical, use a one-shot fetch.
+- **The mode is decided once, at startup.** If the first connection is refused or
+  unreachable, the run polls for its whole life. If the stream opens, there is no
+  polling fallback left.
+- **A stream lost mid-run ends the command** with
+  `The realtime log stream stopped and could not be re-established`. That non-zero
+  exit is the stream giving up, not proof that logging is broken — re-run the same
+  command rather than changing flags.
+
+### 2. Ask whether it was a scheduled run, not a request
+
+Workflows are the automation system — cron schedules, entity triggers, connector
+events, in-app agent actions. **When the complaint is "my scheduled job didn't run" or
+"the automation stopped working", function logs are the wrong tool.** They show what a
+function printed; they cannot tell you whether a run was dispatched at all, which task
+inside it failed, or why the workflow stopped firing.
+
+```bash
+npx base44 workflows runs --status failed
+npx base44 workflows list
+```
+
+A failed run carries the failing task and the underlying error, so start there and drop
+into `base44 logs` only once you know which function a failing task called.
+`workflows list` reports `consecutiveFailures` — anything above zero is a workflow that
+needs attention.
+
+Two things that mislead here:
+
+- **`manual` in the trigger column does not mean a person clicked something.** It is
+  what a run is stamped with when it was dispatched with no trigger type at all.
+- **Test runs are included**, tagged next to the trigger type as `(scheduled, test)`.
+  A run you fired yourself to check something will show up in the list.
+
+### 3. Check Recent Errors
 
 Start by pulling the latest errors across all functions:
 
 ```bash
 npx base44 logs --level error
 ```
 
-### 2. Drill Into a Specific Function
+### 4. Drill Into a Specific Function
 
 If you know which function is failing:
 
@@ -55,16 +130,31 @@ If you are outside the project directory, pass the app explicitly:
 npx base44 logs --app-id app_123 --function <function_name> --level error
 ```
 
-### 3. Inspect a Time Range
+A `--function` filter is a filter on *stamped* rows. Apps still on the legacy
+per-function deployment emit unstamped rows, so a filtered view can hide them; this
+self-heals on the app's next deploy. If a filtered run comes back empty, re-run
+without `--function` before concluding there are no logs.
+
+### 5. Inspect a Time Range
 
 Correlate with user-reported issue timestamps:
 
 ```bash
 npx base44 logs --function <function_name> --since <start_time> --until <end_time>
 ```
 
-### 4. Analyze the Logs
+### 6. Analyze the Logs
 
 - Look for stack traces and error messages in the output
 - Check timestamps to correlate with user-reported issues
-- Use `--limit` to fetch more entries if the default 50 isn't enough
+- Pass `--limit` explicitly to reach further back — there is no default page size, and a value above 500 is clamped down to 500
+
+## Reading an empty result
+
+`No logs found matching the filters.` is ambiguous — never read it as "healthy". It
+means one of:
+
+- the run has not been ingested yet (most common — wait and re-run, see above)
+- no function by that name, or the filter dropped unstamped rows (see step 3)
+- the app has not been published, when reading `--env prod`
+  (`No production logs found.` — try `--env preview` for draft logs)

skills/base44-troubleshooter/references/project-logs.md

@@ -15,18 +15,18 @@ This command can run from a linked project, or outside a project when you pass `
 | Option | Description | Required |
 |--------|-------------|----------|
 | `--function <names>` | Filter by function name(s), comma-separated. If omitted, fetches logs for all functions in the current app | No |
-| `--since <datetime>` | Show logs from this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`) | No |
+| `--since <datetime>` | Show logs from this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`). Cannot be combined with `--follow` | No |
 | `--until <datetime>` | Show logs until this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`). Cannot be combined with `--follow` | No |
 | `--level <level>` | Filter by log level: `info`, `warning`, `error`, `debug` | No |
-| `-n, --limit <n>` | Number of results to return (1-1000, default: 50) | No |
-| `--order <order>` | Sort order: `asc` or `desc` (default: `desc`). Cannot be combined with `--follow` | No |
+| `-n, --limit <n>` | Number of results to return. **No default** — the CLI sends a limit only when you pass one, and a value above 500 is clamped down to 500 | No |
+| `--order <order>` | Sort order: `asc` or `desc`. Only affects a **multi-function** fetch (it orders the client-side merge); ignored when reading a single function. Cannot be combined with `--follow` | No |
 | `--env <env>` | Which deployment to read logs from: `preview` (current draft) or `prod` (published). Default: `preview` | No |
-| `-f, --follow` | Stream new logs as they arrive instead of a one-shot fetch | No |
+| `-f, --follow` | Stream new logs as they arrive instead of a one-shot fetch. Realtime (sub-second) where the stream is available; where it cannot be opened the CLI polls instead (~20-30s lag). Cannot be combined with `--since`, `--until` or `--order` | No |
 
 ## Examples
 
 ```bash
-# Fetch logs for all project functions (last 50 entries)
+# Fetch logs for all project functions
 npx base44 logs
 
 # Fetch logs for a specific app without a local checkout
@@ -50,17 +50,20 @@ npx base44 logs --since 1h
 # Fetch logs within a time range
 npx base44 logs --since 2024-01-15T10:00:00 --until 2024-01-15T12:00:00
 
-# Fetch last 100 log entries in ascending order
-npx base44 logs -n 100 --order asc
+# Merge several functions' logs oldest-first (--order applies to the merge)
+npx base44 logs --function send-email,process-payment -n 100 --order asc
 
 # Last 10 errors for a specific function
 npx base44 logs --function myFunction --level error --limit 10
 
 # Fetch logs from the published (prod) deployment instead of preview
 npx base44 logs --env prod
 
-# Stream new logs live as they arrive
+# Stream new logs live as they arrive (all functions)
 npx base44 logs --follow
+
+# Stream one function's logs live
+npx base44 logs --follow --function my-function
 ```
 
 ## Notes
@@ -70,7 +73,97 @@ npx base44 logs --follow
 - When multiple functions are specified, logs are merged and sorted by timestamp.
 - If `--function` is omitted, logs are fetched for **all functions** in the current app.
 - The `--limit` applies after merging logs from all specified functions.
+- There is **no default page size**. The CLI sends a limit only when you pass one, and a `--limit` above 500 is clamped down to 500 — so do not plan on paging further back by raising the number. What you get when you omit it is not a fixed number: the runtime decides, and it may cap you at 500 anyway. Pass `--limit` when the count matters to you.
+- `--order` is only honored for the client-side merge of several functions. The server does not read it, so it is inert on a single-function fetch — the entries come back newest-first regardless.
 - The `--since` and `--until` values accept an ISO datetime, or a relative shorthand (e.g. `1h`, `30m`, `2d`) measured back from now. ISO values without a timezone are normalized to UTC (appends `Z`).
 - `--env` defaults to `preview`. If `prod` returns no logs, the app may not have been published yet — try `--env preview` to see draft logs.
-- `--follow` streams logs indefinitely (oldest to newest) instead of a single fetch; it's incompatible with `--until` and `--order`.
+- **`No logs found matching the filters.` is ambiguous.** It means one of: the run has not been ingested yet (~20-30s; wait and re-run — *do not* change flags), there is no function by that name, or a `--function` filter dropped unstamped rows from a legacy per-function deployment. It never means "the app is healthy".
+- `--follow` streams logs indefinitely (oldest to newest) instead of a single fetch; it's incompatible with `--since`, `--until` and `--order`. A stream that is lost and cannot be re-established ends the command with an error rather than dropping to polling. See [Following logs live](#following-logs-live).
 - Pass the global `--json` flag to emit each log entry (or, with `--follow`, each new line) as JSON instead of the human-readable format.
+
+## Following logs live
+
+`--follow` is the tool to reach for when you can reproduce the problem, because it
+does not wait on log ingestion.
+
+### Streaming or polling is decided once, at startup
+
+`--follow` opens a realtime stream before it prints anything, and **that first attempt
+decides the mode for the whole run.**
+
+**The stream opens** — lines arrive **in under a second** of the invocation ending, and
+you stay in realtime for the rest of the run.
+
+**The stream is refused** — the app is still on a legacy per-function deployment, or
+the feature is not enabled for it. The CLI says so and polls instead, for the life of
+the process:
+
+```
+Realtime logs are not available for this app — falling back to polling (lines may lag ~20-30s).
+```
+
+**The stream cannot be reached** — a transient failure that survived the retries. Same
+outcome, different message:
+
+```
+Could not reach the realtime log stream — falling back to polling (lines may lag ~20-30s).
+```
+
+Either way the command keeps working; the only difference is latency. On a legacy
+per-function app, `--follow --function <name>` is refused (404) and polls — that
+self-heals on the app's next deploy, there is nothing to fix.
+
+### A stream that dies mid-run ends the command — it does not quietly start polling
+
+Once the stream has opened there is **no polling fallback left**. If it is lost and
+cannot be re-established — repeated dead connections, or a typed end frame saying the
+tail is gone — `--follow` exits with an error rather than degrading:
+
+```
+The realtime log stream stopped and could not be re-established
+```
+
+It suggests starting a new tail (`base44 logs --follow`) or reading recent logs without
+streaming (`base44 logs`).
+
+**This matters if you are driving the CLI from a script or an agent loop.** A non-zero
+exit from `--follow` partway through is the stream giving up, not proof that logging is
+broken and not a reason to change flags. Re-run the same command. Ordinary reconnects
+are invisible: the CLI reconnects on its own and only errors once it has run out of
+attempts.
+
+### Reading the stream without fooling yourself
+
+- **Silence is not a verdict.** Never open the stream, read for a fixed window, and
+  conclude the pipeline is broken — a stream with nothing invoked against it is
+  correctly silent. Trigger the function, then read until the lines arrive.
+- **Delivery is per-invocation.** A function's lines are delivered as a batch when
+  the invocation ends, so a long-running call is silent while it runs.
+- **A deploy does not break an open stream.** The script rotates in seconds and the
+  same stream carries the new code's lines on the next invoke. Restarting the stream
+  after each deploy adds noise and loses nothing.
+- **`--function` filters on stamped rows.** Failure records are function-stamped, so
+  filtered streams keep them; rows from a legacy per-function script carry no stamp
+  and are dropped by the filter until the app's next deploy. When a filtered stream
+  looks empty, drop the filter before concluding anything.
+
+### Driving the SSE endpoint directly
+
+The CLI handles all of this for you — this section matters only if you are consuming
+`/api/apps/<id>/functions-mgmt/logs/stream` yourself.
+
+- Log frames are unnamed `data:` events (`{time, level, function, message}`).
+- Comment lines (`: ping`) are keepalives. They carry no logs but they **do** prove
+  the connection is alive — treat a ping as liveness, not as an empty read.
+- The typed `event: end` frame is what disambiguates silence, and its `retriable`
+  flag is the whole contract: `true` — every reason the backend currently sends,
+  including a tail that went unavailable — means **reconnect immediately**; `false`
+  means stop streaming. Do not branch on the reason string. What you do after a
+  `false` is your choice as a raw consumer — the bounded polling route is the obvious
+  one — but note the CLI itself does **not** do that: it ends the run with an error
+  (see above). (And a fallback to polling in the CLI is driven by the *first*
+  connection being refused, never by an end frame.)
+- Reconnect on a drop rather than giving up on the first one. Carry the last seen
+  timestamp across the reconnect — but for **dedupe**, and to resume a polling
+  fallback where the stream left off. It does not make the handover gapless: a tail
+  has no replay, so lines emitted during the gap are simply gone.