feat(mcp): leadbay_getting_started — guided first-run walkthrough - #175
Draft
ArtyETH06 wants to merge 12 commits into
Draft
feat(mcp): leadbay_getting_started — guided first-run walkthrough#175ArtyETH06 wants to merge 12 commits into
ArtyETH06 wants to merge 12 commits into
Conversation
A brand-new user learns Leadbay by DOING, not by reading. Three gates, each
presenting exactly ONE option, each click running a real call:
1. "Pull today's leads" -> leadbay_pull_leads (no args)
2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free)
3. "Run this every morning" -> no Leadbay tool; the host's scheduler
Ships as both a prompt (auto-emits a Claude Code skill) and a read-only
composite tool returning the step manifest, so natural phrasing ("I'm new")
reaches the tour on hosts that don't surface MCP prompts.
Two hard rules encoded in the manifest, prompt, and tests:
- Gate 2 spends NOTHING. Omitting `titles` returns mode:"discover", the free
preview. Passing titles/confirm/email/phone launches a PAID reveal, and
spending a 90-second-old account's quota to demo a feature is the worst
outcome available here.
- Gate 3 has calls:null because Leadbay exposes no scheduling API. The gate's
literal recurring wording ("every morning") is what lets the host's existing
SCHEDULED_TASK_PARAGRAPH flow fire on its own terms — server.ts is untouched,
so the daily-check-in evals are unaffected.
A warming lens (computing_wishlist/computing_scores) is explained and re-pulled
rather than reported as "no leads" — pull_leads already returns a server-built
two-option warm-up payload for that state, rendered verbatim.
The only edits to existing test files are two one-line registration appends
(the prompts/list toEqual array and TOOLS_WITH_ROUTING), matching the precedent
in 6ddbcf8 and the instruction in CLAUDE.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Leadbay has NO CRM integration — no push, export, sync, or third-party OAuth exists, and report_friction's canonical missing_capability example is literally "Why can't I export to HubSpot?". But the AGENT usually does have one: users run a HubSpot/Salesforce/Pipedrive/Attio connector in the same host. So the new gate delegates rather than pretending. The walkthrough is now four gates: 1. "Pull today's leads" -> leadbay_pull_leads 2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 3. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 4. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 3 reuses the detection mechanism the connected-outreach-tool table in leadbay_prospecting_overview already established: read the host's installed-connector / installed-MCP inventory, else infer from conversation, else ask. It names the CAPABILITY, never a third-party tool name — a backticked `hubspot_create_company` would be the first such reference in the repo and would rot silently when the connector renames its tools. Four honesty guards, each with a test: - Never claim a CRM record was created unless the connector confirmed it. - Never write an email or phone into the CRM. Gate 2 is the FREE title preview, so no contact detail was ever revealed; writing one is fabricated PII. - With no connector: say so in one honest line, name the user's CRM, and offer leadbay_report_friction with category:'missing_capability' — the real route. - Don't hunt for a leadbay_* CRM tool; none exists. Modelled on the gate-4 scheduler delegation shipped in the previous commit — same shape, same "only the host can create one" discipline. Co-Authored-By: Claude <noreply@anthropic.com>
The tutorial now starts by proving the connection works, instead of opening with prose and jumping straight to leads. Five gates: 1. "Check my account" -> leadbay_account_status 2. "Pull today's leads" -> leadbay_pull_leads 3. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 4. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 5. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 1 sits on top of two PINNED regressions, so its branches encode both and the tests assert them: - WORKFLOWS #30 — a brand-new org has no billing plan, so quota_status 401s. leadbay_account_status swallows that into `quota_error`. The gate must say NOTHING about quota and must NEVER suggest logging in again: the token is valid, the same response just read the account. This is the product#3761 401-hallucination bug, and a first-run tutorial is the worst possible place to reintroduce it. - WORKFLOWS #31 — account-status.ts:166 withholds the lens unless the trigger text mentions it. "Walk me through Leadbay" doesn't, so there is nothing to report; the gate must not volunteer it nor call another tool to find it. The lens appears naturally at gate 2. The underdeliver eval fixtures quota_status as a 401 on purpose, so the run proves the silence rather than assuming it. Co-Authored-By: Claude <noreply@anthropic.com>
…it fires
Live test showed the agent running the tools straight through without ever
firing a choice widget — the user watched a demo instead of taking a tutorial.
Root cause: the gates only DESCRIBED the widget in prose (gate_label /
gate_description as loose strings), leaving the agent to assemble the widget
call itself. Meanwhile leadbay_pull_leads ships a ready-made `next_steps`
{question, options[]} object, and the shared routing snippet says a next_steps
payload is the source of truth and must be mapped VERBATIM. The tour was on
the weak path.
Each step now carries that same payload shape:
next_steps: { question, options: [{label, description, kind}] } // exactly 1
explain: "what to TELL the user before firing the widget"
So the agent renders data instead of interpreting prose, and each gate is two
beats — explain, then ask, then WAIT for the click. The explain beat is the
tutorial half: gate 2 teaches what a lens is, gate 3 teaches what enrichment
is and that the preview costs nothing.
The prompt gains an explicit "NEVER run a step's tool without firing its
widget first and receiving the click" rule (with the one sane exception: the
user's own message already said to run everything), plus three failure modes
for exactly what went wrong.
The new prompt↔manifest widget-text audit caught a real bug on its first run:
markdown line-wrapping had split the widget strings across newlines, so the
shipped prompt and the manifest disagreed on the question text.
Co-Authored-By: Claude <noreply@anthropic.com>
…e clicks The buttons disappear when the tour ends. A user who was only ever shown buttons learned to click a tutorial and nothing about using Leadbay tomorrow. The manifest now carries `keep_going`: a what-you-want -> what-you-say cheat-sheet the agent renders as a small table at the finish. | Today's fresh leads | "Show me today's leads" | | Who to follow up with | "What should I follow up on" | | The story on one company | "Research <Company>" | | An email to a contact | "Draft outreach for <Contact>" | | Change who you target | "Narrow the audience to <sector>"| | Switch target audience | "Show me my lenses" | Every phrase is lifted VERBATIM from that tool's own routing.triggers, and a test enforces it: each `say` is checked against the trigger blocks in the generated tool descriptions, so a phrase that doesn't actually route fails the build. Teaching a phrase that silently does nothing is worse than teaching none, and "sounds about right" is exactly how that happens. Verified the guard bites: swapping in a plausible "Fetch my newest prospects please" fails with the offending phrase named. Skipped when the user abandons the tour early — they're already off doing what they wanted, and a tutorial summary would just interrupt. Co-Authored-By: Claude <noreply@anthropic.com>
…yllabus
The tour front-loaded text: STEP 0 asked for 2-3 sentences PLUS a preview of
all five upcoming clicks, and then gate 1 added its own explain beat on top.
The first button ended up buried under paragraphs nobody reads.
The opening is now, in one message:
1. one sentence on what Leadbay is
2. one short line naming the step — "Let's start with your account status."
3. gate 1's widget, fired immediately
and then it stops. No five-step preview, no lens explanation yet — each gate
already explains itself when its own turn comes, so saying it up front is
duplication that costs the user the thing they actually wanted: seeing it work.
Gate 1's widget text is shortened to match ("Check my Leadbay account status.")
and the gate no longer stacks a second explanation on top of the opening lines.
Two new failure modes cover the regression: opening with a wall of text, and
ending the first message without firing the widget at all.
The prompt<->manifest drift audit paid for itself again — markdown line-wrap
had split the new widget description across a newline, so prompt and manifest
disagreed on the exact string.
Co-Authored-By: Claude <noreply@anthropic.com>
…mprovised tour Live failure in Claude Desktop chat: "Walk me through Leadbay please" produced the agent's OWN product overview — a mental-model essay plus a lens table and a four-option "Where do you want to start?" widget. Nothing from leadbay_getting_started ran. The prompt was not missing. It was listed in PROMPT_CATALOG_BULLETS at ~char 10,786 of a 24.5k-char instruction block, one bullet among fifteen. A listing tells the agent the prompt EXISTS; nothing told it to CHOOSE it, so it wrote something itself. Adds a FIRST RUN routing line to the server instructions, placed before the generic start-here flow (now ~char 6,887, ahead of the catalog). It names the phrasings verbatim, says invoke `leadbay_getting_started` via `prompts/get`, and explicitly forbids improvising an overview — with the reason attached, so the prohibition doesn't read as arbitrary and get ignored the moment the agent thinks its own summary would be nicer. Deliberately NOT fixed by narrowing leadbay_prospecting_overview's very broad short_description: that regenerates its shipped snapshot and widens the blast radius. The routing line is the smaller, more direct change. New audit test pins the phrasings, the invoke verb, the anti-improvisation rule, and — load-bearing — that the line lands BEFORE the catalog listing. Co-Authored-By: Claude <noreply@anthropic.com>
Live in Claude Desktop, gate 1 came out as prose — "Let's start with your
account status — say the word and I'll check it" with a bold "-> Check my
account" line. No widget, no button.
Root cause was a contradiction the walkthrough shipped with: the prompt said
"exactly ONE option", while the shared next-steps routing snippet it INCLUDES
says "2-4 mutually-exclusive options". The model followed the shared rule and
fell back to prose. pull-leads.ts:123-127 already documented this exact failure
("a single option would make the model emit an invalid widget call, or silently
drop to prose"); the walkthrough was written against the other belief.
Every gate now carries two options: the forward action, then `I'm done for
now`. That satisfies the host contract so the widget renders, while keeping
exactly one way FORWARD — the tutorial still never asks a first-run user to
choose between paths. The exit must end the tour, never route elsewhere, or it
reintroduces the choice the rule exists to remove.
The rule is renamed ONE-FORWARD-OPTION and states the reason inline, so the
next person doesn't "fix" it back to one option.
Opening lines rewritten to lead with what the user gets ("a fresh batch of
companies worth selling to every day") and to promise something concrete
("five quick steps, and you'll have real leads by the end").
Tests now assert the two-option shape, exactly one exit, exit-ends-the-tour,
and forward-option-first.
Co-Authored-By: Claude <noreply@anthropic.com>
The user clicks a button labelled "check my account status" and got back "you're connected as X, an admin on Y". That under-delivers on the button they pressed. It was also wrong against the tool's own contract: leadbay_account_status's rendering_hint says "Report user + org, AND quota whenever readable — include quota even on a plain 'what account am I on?'", silent ONLY when quota is null, quota_error is set, or the org is unlimited. My gate told the agent to report user/org/plan and never mentioned quota at all, so it under-reported on every account where quota reads fine. Gate 1 now includes the canonical `rendering/quota-windows` snippet — the same one leadbay_account_status uses — so the tour renders what the web app renders: Daily / Weekly / Monthly, each with a ▰▱ gauge, % used, $ spent against cap, resets countdown, and the per-resource breakdown. Never raw "credits". The silence gate is preserved and widened to all three cases it actually covers (null / quota_error / unlimited_credits), not just the 401 — so WORKFLOWS #30 still holds and a plan-less org sees nothing about quota rather than an error. Two failure modes added: answering with a bare greeting when quota IS readable, and rendering credits or raw resource_type strings instead of the gauges. Co-Authored-By: Claude <noreply@anthropic.com>
The tour explained mechanics but never made the case. "Leadbay keeps a lens —
a description of who you sell to" tells a new user what the feature is; it does
not tell them why they should care, which is the actual question a first-run
user is asking.
Opening is now a short paragraph instead of two lines: what Leadbay is, how the
lens works, and what the five steps will leave them holding. Still bounded — it
must NOT walk the five steps one at a time, which is the wall-of-text version
that buried the first button two commits ago.
Every gate gains a concrete payoff, in working-life terms rather than feature
terms:
leads -> replaces the hour spent digging through directories; and the lens
sharpens from what you like, contact or skip
enrich -> ask for the operations director by title instead of pitching
whoever answers the switchboard
CRM -> no copy-pasting between tabs; a lead found here doesn't quietly
die in a chat window
daily -> prospecting is the first thing that slips on a busy week; this
removes the part that depends on remembering
Tests assert every gate carries a WHY IT'S USEFUL beat and that the opening
still teaches the lens without regressing to a syllabus, so a later trim can't
quietly strip the reasoning back out.
Co-Authored-By: Claude <noreply@anthropic.com>
Gate 1 rendered the quota gauges but left a first-run user staring at percentages and dollar figures they've never seen, with no way to tell whether they're good, bad, or something to worry about. It now follows the render with one or two plain lines: what the numbers count (the AI work Leadbay does for them — researching companies, qualifying leads — not something they spend by clicking around) and why it matters to them (it paces how many fresh leads arrive; heavy use now means a bigger batch queued next time, and it's where a smaller-than-expected batch would show its reason). Bounded deliberately: a sentence or two, no walking through every resource row, and no pricing pitch — the tool's own guidance already covers wait-vs-top-up when a window is actually exhausted. The explanation is skipped whenever the silence gate fires (quota null, quota_error, or unlimited_credits). Describing a gauge that isn't on screen is worse than saying nothing, and a test pins that. Co-Authored-By: Claude <noreply@anthropic.com>
The tour stopped at the free title preview and never revealed a contact, so a
first-run user finished the walkthrough without ever seeing the thing that
makes Leadbay useful: a name and an email to actually reach.
Gate 3 now runs in two beats:
Beat 1 — free. leadbay_enrich_titles with no titles/confirm/email/phone
returns mode:"discover", the available job titles. "Nothing spent
yet."
Beat 2 — paid, on consent. Ask them to pick 2-3 leads, state the cost BEFORE
they choose, then call again with the chosen titles, confirm:true
and email:true. Poll leadbay_bulk_enrich_status to completion and
report only the contacts that actually resolved.
The consent guarantee is the ordering, and it is explicit: the gate click
bought the free look, not the reveal. Silence is not consent, and neither is
"they clicked the gate earlier". Declining is a normal outcome — keep the
preview and move on.
It then says what it cost in one line (one credit per contact revealed), which
is the moment gate 1's quota numbers stop being abstract: they just watched
them move.
Gate 4 updated to match — it may now pass through real emails/phones, but ONLY
the ones the enrichment returned; if the user declined there are none, and
inventing one is fabrication.
The underdeliver eval keeps NO launch fixture on purpose: in that scenario the
user is never asked and never confirms, so a launch would hit an undeclared
endpoint and fail the run — the consent guarantee, enforced.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A brand-new user learns Leadbay by doing, not by reading. Five gates, each presenting exactly one option.
Check my accountleadbay_account_statusPull today's leadsleadbay_pull_leadsEnrich top leadsleadbay_enrich_titles(notitles→ freemode:"discover")Add these to my CRMRun this every morningShips as both a prompt (auto-emits a Claude Code skill) and a read-only composite tool returning the step manifest, so natural phrasing ("I'm new") reaches the tour on hosts that don't surface MCP prompts as slash commands.
Gate 1 sits on two pinned regressions
The tutorial opens by proving the connection works. That makes it the worst possible place to reintroduce either locked
account_statusbehaviour, so the manifest encodes both and the tests assert them:quota_status401s and lands inquota_error. The gate says nothing about quota and never suggests logging in again: the token is valid, the same response just read the account. (product#3761's 401-hallucination bug.) The underdeliver eval fixtures that 401 on purpose, so the run proves the silence instead of assuming it.account-status.ts:166withholds the lens unless the trigger text mentions it. "Walk me through Leadbay" doesn't, so there's nothing to report and no other tool to reach for. The lens appears naturally at gate 2.Leverage the host, don't fake the feature
Gates 4 and 5 delegate capabilities Leadbay does not have and the host usually does, returning
calls: nullso no agent can infer aleadbay_*tool that would do it.CRM (gate 4). Leadbay has no CRM integration — no push, export, sync, or third-party OAuth. (
report_friction's canonicalmissing_capabilityexample is literally "Why can't I export to HubSpot?".) But the agent often has a HubSpot/Salesforce/Pipedrive/Attio connector in the same host, so the gate tells it to check its own tool set and use that. Detection reuses the mechanism the connected-outreach-tool table inleadbay_prospecting_overviewalready established: host's installed-connector inventory → the conversation → ask.It names the capability, never a third-party tool name. A backticked
hubspot_create_companywould be the first such reference in this repo and would rot silently when that connector renames its tools.Scheduling (gate 5). Same shape. The gate's literal recurring wording ("every morning") is what lets the existing
SCHEDULED_TASK_PARAGRAPHflow fire on its own terms —server.tsis untouched, so the daily-check-in evals are unaffected.Honesty guards
Each has a test:
leadbay_report_frictionwithcategory: "missing_capability". Not instructions for a connector they don't have.titles/confirm/email/phonelaunches a paid reveal; spending a 90-second-old account's quota to demo a feature is the worst outcome this feature could produce.pull_leadsalready returns a server-built two-option warm-up payload forcomputing_wishlist/computing_scores, rendered verbatim.Gates
pnpm prompts:build·pnpm -r build·pnpm -r typecheck·pnpm -r test— all green: 1,318 tests, 0 failures (core 632, mcp 623, components 47, promptforge 16).pnpm prompts:checkclean (no template drift). Description well under the 17,000-char cap.Verified end-to-end over real MCP stdio JSON-RPC:
prompts/listreturns 13 (was 12) and includes the new prompt;prompts/getserves the full body with no unsubstituted placeholders. That confirms the two-place registration is complete —leadbay_extend_my_lensandleadbay_followup_check_inare both missing theirCATALOGentry today and so never appear inprompts/list.Eval coverage
The two scenarios are authored to the README shape and, like
pull-leads-orderandscan-portfolio-signals, go live when the scenario-execution glue lands — so there is no live judge run to report yet. No MM/IA/NF/TSF numbers; I won't paste a score I didn't observe. The deterministic red/green proof is in the unit mirrors:getting-started.test.ts(14 tests) — one option per gate, gate 1's two regressions, the four forbidden spend args,calls:null+ honesty guards on both delegated gates, the warming-lens branch, zero HTTP requestsgetting-started-walkthrough.test.ts(15 tests) — prompt↔manifest gate-label agreement, plus "names no third-party tool name", which is the drift a future edit would introduce silentlyI confirmed these bite: dropping
confirmfromforbidden_argsfails the spend-gate assertion.WORKFLOWS.md rows #50 (walkthrough) and #51 (over-claim guard) carry the contracts.
Not covered
Single-option widget rendering is unverified on a live host.
pull-leads.ts:123-127carries a code comment asserting a lone option makes the model emit an invalid widget call or silently drop to prose, and the shared routing snippet documents a 2–4 option contract. Built as specified with two mitigations — the prose fallback is stated in the tour, and the escape hatch is typing rather than a button — butAskUserQuestion(Claude Code) andask_user_input_v0(Claude chat) have different schemas and fail silently, so this needs eyes on both.The CRM push is untested against a real connector. No CRM MCP server was installed in this environment, so only the no-connector path was exercised. The with-connector branch is prompt guidance, verified by assertion, not by a live HubSpot write.
Also out of scope: storing a deep-link back to the CRM record (
crm-record-link.mdalready does this for imports) and makingleadbay_prospecting_overviewcross-route back to the tour — that regenerates its shipped snapshot, so it's a follow-up.Note for review
The only changes to existing test files are two one-line registration appends — the
prompts/listtoEqualarray andTOOLS_WITH_ROUTING. Both are the established way to register a prompt/routed tool here (precedent:6ddbcf85,dedb95e2; CLAUDE.md instructs the latter). No existing test logic was modified.Closes https://github.com/leadbay/product/issues/3952