diff --git a/skills/base44-troubleshooter/SKILL.md b/skills/base44-troubleshooter/SKILL.md index 60a346f..65fd269 100644 --- a/skills/base44-troubleshooter/SKILL.md +++ b/skills/base44-troubleshooter/SKILL.md @@ -1,6 +1,6 @@ --- name: base44-troubleshooter -description: Troubleshoot production issues using backend function logs. Use when investigating app errors, debugging function calls, or diagnosing production problems in Base44 apps. +description: Troubleshoot production issues using backend function logs and workflow run history. Use when investigating app errors, debugging function calls, diagnosing why a scheduled job or automation failed, or diagnosing production problems in Base44 apps. --- # Troubleshoot Production Issues @@ -30,10 +30,85 @@ npx base44 logs --app-id app_123 | Command | Description | Reference | |---------|-------------|-----------| | `base44 logs` | Fetch function logs for this app | [project-logs.md](references/project-logs.md) | +| `base44 workflows runs` | List workflow runs, newest first; failed runs carry the failing task and the underlying error | [workflows-runs.md](../base44-cli/references/workflows-runs.md) | +| `base44 workflows list` | List this app's workflows with status and run summary | [workflows-list.md](../base44-cli/references/workflows-list.md) | + +## Logs are not read-after-write + +**A single fetch that misses your run proves nothing.** One-shot `base44 logs` reads +an index that lags behind the invocation, so an empty result right after triggering a +function is the expected result, not evidence of a problem. + +- **Live debugging: use `--follow`.** Where the realtime stream is available it + delivers lines in **under a second**. Where it is not, the CLI says so and polls + instead (`Realtime logs are not available for this app — falling back to polling + (lines may lag ~20-30s).`). Either way `--follow` is the right tool — you never + have to pick. +- **One-shot fetches lag ~20-30s.** That is ingestion time, not a filter problem. +- **When output is empty, the variable to change is TIME, never a flag.** Wait and + re-run the same command. Widening `--limit`, dropping `--level`, or switching + `--order` changes nothing about a line that has not been ingested yet, and + re-rolling flags is how agents talk themselves into a wrong diagnosis. ## Troubleshooting Flow -### 1. Check Recent Errors +### 1. Watch it happen — `--follow` + +If you can trigger the failure (or it is happening now), start here rather than +fetching after the fact: + +```bash +npx base44 logs --follow +npx base44 logs --follow --function +``` + +Then invoke the function and read what arrives. Rules that keep you from misreading +a healthy stream: + +- **Never decide on a timer.** Open the stream, trigger the function, and read until + you see the lines — do not read a fixed window, print, and conclude "broken". + A quiet stream is quiet because nothing has been invoked. +- **Delivery is per-invocation, not per-line.** A long-running function's lines all + arrive when the invocation ends. Silence mid-invocation is normal. +- **Redeploying mid-follow is safe.** A deploy rotates the script in seconds and the + same open stream delivers the new code's lines on the next invoke. Do not tear the + stream down and rebuild it after every deploy. +- **`--since` is rejected with `--follow`** (the stream starts from now), as are + `--until` and `--order`. For anything historical, use a one-shot fetch. +- **The mode is decided once, at startup.** If the first connection is refused or + unreachable, the run polls for its whole life. If the stream opens, there is no + polling fallback left. +- **A stream lost mid-run ends the command** with + `The realtime log stream stopped and could not be re-established`. That non-zero + exit is the stream giving up, not proof that logging is broken — re-run the same + command rather than changing flags. + +### 2. Ask whether it was a scheduled run, not a request + +Workflows are the automation system — cron schedules, entity triggers, connector +events, in-app agent actions. **When the complaint is "my scheduled job didn't run" or +"the automation stopped working", function logs are the wrong tool.** They show what a +function printed; they cannot tell you whether a run was dispatched at all, which task +inside it failed, or why the workflow stopped firing. + +```bash +npx base44 workflows runs --status failed +npx base44 workflows list +``` + +A failed run carries the failing task and the underlying error, so start there and drop +into `base44 logs` only once you know which function a failing task called. +`workflows list` reports `consecutiveFailures` — anything above zero is a workflow that +needs attention. + +Two things that mislead here: + +- **`manual` in the trigger column does not mean a person clicked something.** It is + what a run is stamped with when it was dispatched with no trigger type at all. +- **Test runs are included**, tagged next to the trigger type as `(scheduled, test)`. + A run you fired yourself to check something will show up in the list. + +### 3. Check Recent Errors Start by pulling the latest errors across all functions: @@ -41,7 +116,7 @@ Start by pulling the latest errors across all functions: npx base44 logs --level error ``` -### 2. Drill Into a Specific Function +### 4. Drill Into a Specific Function If you know which function is failing: @@ -55,7 +130,12 @@ If you are outside the project directory, pass the app explicitly: npx base44 logs --app-id app_123 --function --level error ``` -### 3. Inspect a Time Range +A `--function` filter is a filter on *stamped* rows. Apps still on the legacy +per-function deployment emit unstamped rows, so a filtered view can hide them; this +self-heals on the app's next deploy. If a filtered run comes back empty, re-run +without `--function` before concluding there are no logs. + +### 5. Inspect a Time Range Correlate with user-reported issue timestamps: @@ -63,8 +143,18 @@ Correlate with user-reported issue timestamps: npx base44 logs --function --since --until ``` -### 4. Analyze the Logs +### 6. Analyze the Logs - Look for stack traces and error messages in the output - Check timestamps to correlate with user-reported issues -- Use `--limit` to fetch more entries if the default 50 isn't enough +- Pass `--limit` explicitly to reach further back — there is no default page size, and a value above 500 is clamped down to 500 + +## Reading an empty result + +`No logs found matching the filters.` is ambiguous — never read it as "healthy". It +means one of: + +- the run has not been ingested yet (most common — wait and re-run, see above) +- no function by that name, or the filter dropped unstamped rows (see step 3) +- the app has not been published, when reading `--env prod` + (`No production logs found.` — try `--env preview` for draft logs) diff --git a/skills/base44-troubleshooter/references/project-logs.md b/skills/base44-troubleshooter/references/project-logs.md index 8bb2daf..aa5b42c 100644 --- a/skills/base44-troubleshooter/references/project-logs.md +++ b/skills/base44-troubleshooter/references/project-logs.md @@ -15,18 +15,18 @@ This command can run from a linked project, or outside a project when you pass ` | Option | Description | Required | |--------|-------------|----------| | `--function ` | Filter by function name(s), comma-separated. If omitted, fetches logs for all functions in the current app | No | -| `--since ` | Show logs from this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`) | No | +| `--since ` | Show logs from this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`). Cannot be combined with `--follow` | No | | `--until ` | Show logs until this time. ISO datetime or relative shorthand (e.g. `1h`, `30m`, `2d`). Cannot be combined with `--follow` | No | | `--level ` | Filter by log level: `info`, `warning`, `error`, `debug` | No | -| `-n, --limit ` | Number of results to return (1-1000, default: 50) | No | -| `--order ` | Sort order: `asc` or `desc` (default: `desc`). Cannot be combined with `--follow` | No | +| `-n, --limit ` | Number of results to return. **No default** — the CLI sends a limit only when you pass one, and a value above 500 is clamped down to 500 | No | +| `--order ` | Sort order: `asc` or `desc`. Only affects a **multi-function** fetch (it orders the client-side merge); ignored when reading a single function. Cannot be combined with `--follow` | No | | `--env ` | Which deployment to read logs from: `preview` (current draft) or `prod` (published). Default: `preview` | No | -| `-f, --follow` | Stream new logs as they arrive instead of a one-shot fetch | No | +| `-f, --follow` | Stream new logs as they arrive instead of a one-shot fetch. Realtime (sub-second) where the stream is available; where it cannot be opened the CLI polls instead (~20-30s lag). Cannot be combined with `--since`, `--until` or `--order` | No | ## Examples ```bash -# Fetch logs for all project functions (last 50 entries) +# Fetch logs for all project functions npx base44 logs # Fetch logs for a specific app without a local checkout @@ -50,8 +50,8 @@ npx base44 logs --since 1h # Fetch logs within a time range npx base44 logs --since 2024-01-15T10:00:00 --until 2024-01-15T12:00:00 -# Fetch last 100 log entries in ascending order -npx base44 logs -n 100 --order asc +# Merge several functions' logs oldest-first (--order applies to the merge) +npx base44 logs --function send-email,process-payment -n 100 --order asc # Last 10 errors for a specific function npx base44 logs --function myFunction --level error --limit 10 @@ -59,8 +59,11 @@ npx base44 logs --function myFunction --level error --limit 10 # Fetch logs from the published (prod) deployment instead of preview npx base44 logs --env prod -# Stream new logs live as they arrive +# Stream new logs live as they arrive (all functions) npx base44 logs --follow + +# Stream one function's logs live +npx base44 logs --follow --function my-function ``` ## Notes @@ -70,7 +73,97 @@ npx base44 logs --follow - When multiple functions are specified, logs are merged and sorted by timestamp. - If `--function` is omitted, logs are fetched for **all functions** in the current app. - The `--limit` applies after merging logs from all specified functions. +- There is **no default page size**. The CLI sends a limit only when you pass one, and a `--limit` above 500 is clamped down to 500 — so do not plan on paging further back by raising the number. What you get when you omit it is not a fixed number: the runtime decides, and it may cap you at 500 anyway. Pass `--limit` when the count matters to you. +- `--order` is only honored for the client-side merge of several functions. The server does not read it, so it is inert on a single-function fetch — the entries come back newest-first regardless. - The `--since` and `--until` values accept an ISO datetime, or a relative shorthand (e.g. `1h`, `30m`, `2d`) measured back from now. ISO values without a timezone are normalized to UTC (appends `Z`). - `--env` defaults to `preview`. If `prod` returns no logs, the app may not have been published yet — try `--env preview` to see draft logs. -- `--follow` streams logs indefinitely (oldest to newest) instead of a single fetch; it's incompatible with `--until` and `--order`. +- **`No logs found matching the filters.` is ambiguous.** It means one of: the run has not been ingested yet (~20-30s; wait and re-run — *do not* change flags), there is no function by that name, or a `--function` filter dropped unstamped rows from a legacy per-function deployment. It never means "the app is healthy". +- `--follow` streams logs indefinitely (oldest to newest) instead of a single fetch; it's incompatible with `--since`, `--until` and `--order`. A stream that is lost and cannot be re-established ends the command with an error rather than dropping to polling. See [Following logs live](#following-logs-live). - Pass the global `--json` flag to emit each log entry (or, with `--follow`, each new line) as JSON instead of the human-readable format. + +## Following logs live + +`--follow` is the tool to reach for when you can reproduce the problem, because it +does not wait on log ingestion. + +### Streaming or polling is decided once, at startup + +`--follow` opens a realtime stream before it prints anything, and **that first attempt +decides the mode for the whole run.** + +**The stream opens** — lines arrive **in under a second** of the invocation ending, and +you stay in realtime for the rest of the run. + +**The stream is refused** — the app is still on a legacy per-function deployment, or +the feature is not enabled for it. The CLI says so and polls instead, for the life of +the process: + +``` +Realtime logs are not available for this app — falling back to polling (lines may lag ~20-30s). +``` + +**The stream cannot be reached** — a transient failure that survived the retries. Same +outcome, different message: + +``` +Could not reach the realtime log stream — falling back to polling (lines may lag ~20-30s). +``` + +Either way the command keeps working; the only difference is latency. On a legacy +per-function app, `--follow --function ` is refused (404) and polls — that +self-heals on the app's next deploy, there is nothing to fix. + +### A stream that dies mid-run ends the command — it does not quietly start polling + +Once the stream has opened there is **no polling fallback left**. If it is lost and +cannot be re-established — repeated dead connections, or a typed end frame saying the +tail is gone — `--follow` exits with an error rather than degrading: + +``` +The realtime log stream stopped and could not be re-established +``` + +It suggests starting a new tail (`base44 logs --follow`) or reading recent logs without +streaming (`base44 logs`). + +**This matters if you are driving the CLI from a script or an agent loop.** A non-zero +exit from `--follow` partway through is the stream giving up, not proof that logging is +broken and not a reason to change flags. Re-run the same command. Ordinary reconnects +are invisible: the CLI reconnects on its own and only errors once it has run out of +attempts. + +### Reading the stream without fooling yourself + +- **Silence is not a verdict.** Never open the stream, read for a fixed window, and + conclude the pipeline is broken — a stream with nothing invoked against it is + correctly silent. Trigger the function, then read until the lines arrive. +- **Delivery is per-invocation.** A function's lines are delivered as a batch when + the invocation ends, so a long-running call is silent while it runs. +- **A deploy does not break an open stream.** The script rotates in seconds and the + same stream carries the new code's lines on the next invoke. Restarting the stream + after each deploy adds noise and loses nothing. +- **`--function` filters on stamped rows.** Failure records are function-stamped, so + filtered streams keep them; rows from a legacy per-function script carry no stamp + and are dropped by the filter until the app's next deploy. When a filtered stream + looks empty, drop the filter before concluding anything. + +### Driving the SSE endpoint directly + +The CLI handles all of this for you — this section matters only if you are consuming +`/api/apps//functions-mgmt/logs/stream` yourself. + +- Log frames are unnamed `data:` events (`{time, level, function, message}`). +- Comment lines (`: ping`) are keepalives. They carry no logs but they **do** prove + the connection is alive — treat a ping as liveness, not as an empty read. +- The typed `event: end` frame is what disambiguates silence, and its `retriable` + flag is the whole contract: `true` — every reason the backend currently sends, + including a tail that went unavailable — means **reconnect immediately**; `false` + means stop streaming. Do not branch on the reason string. What you do after a + `false` is your choice as a raw consumer — the bounded polling route is the obvious + one — but note the CLI itself does **not** do that: it ends the run with an error + (see above). (And a fallback to polling in the CLI is driven by the *first* + connection being refused, never by an end frame.) +- Reconnect on a drop rather than giving up on the first one. Carry the last seen + timestamp across the reconnect — but for **dedupe**, and to resume a polling + fallback where the stream left off. It does not make the handover gapless: a tail + has no replay, so lines emitted during the gap are simply gone.