Skip to content

fix(doctor): judge the LLM provider the way the daemon does - #1575

Merged
rohitg00 merged 7 commits into
rohitg00:mainfrom
dmazhukov:fix/1572-doctor-llm-provider
Oct 26, 2026
Merged

rohitg00 merged 7 commits into
rohitg00:mainfrom
dmazhukov:fix/1572-doctor-llm-provider

Conversation

@dmazhukov

@dmazhukov dmazhukov commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

The no-llm-provider-key doctor check now asks detectLlmProviderKind, which the daemon uses, instead of its own list of six key names, and lists only the keys it would use for the LLM. Doctor also reads .env with the daemon's parser. Placeholder keys still count as missing.

With the built CLI, OPENAI_API_KEY plus the OPENAI_API_KEY_FOR_LLM=false line from .env.example passed as "found: OPENAI_API_KEY" while the daemon ran noop, and AGENTMEMORY_ALLOW_AGENT_SDK=true without a key failed and opened the editor while the daemon ran agent-sdk.

Provider PRs #558, #1111, #1355 and #1554 extend detectLlmProviderKind, so doctor follows them; a key-name list cannot express Azure's key-plus-endpoint rule. #1111, #1355 and #1554 will conflict textually here.

A missing .env also counts: the daemon treats an absent file as empty, so a key exported in the shell resolves a provider without one. env-missing stays the check that reports the absent file.

Where a named key comes from the environment, the verdict says so in its own line rather than in more info, which a passing check never prints.

Embedding checks and the cli.ts hint are unchanged.

The check also merges the shell environment over .env, as getMergedEnv does, so its verdict describes a daemon started from the same environment, and keys found only there are marked (environment). It takes the environment from before hydrateProcessEnvFromFile(), so a key pasted in through the editor fix passes the re-check.

Fixes #1572

Summary by CodeRabbit

  • Bug Fixes
    • Provider-key diagnostics recognize keys available for language-model use, including Google keys, and exclude OpenAI keys reserved for embeddings.
    • Diagnostics check values from both the environment and .env files, with environment values taking precedence. Placeholder values are treated as missing.
    • The agent-SDK fallback is accepted without a provider key. Diagnostics clarify when environment-sourced keys may not be visible to a separately started daemon.
    • Missing-file and missing-key messages clarify fallback behavior and key requirements.

Devin Review

The no-llm-provider-key check counted six key names on its own, while the
daemon decides through detectLlmProviderKind. With OPENAI_API_KEY and
OPENAI_API_KEY_FOR_LLM=false doctor reported the key as found although the
daemon runs in noop mode, and with AGENTMEMORY_ALLOW_AGENT_SDK=true and no
key it failed and opened the editor while the daemon had an LLM.

Drop placeholder keys from the env file, pass the rest to
detectLlmProviderKind and list only the keys it would use for the LLM.
Doctor now parses the env file with the daemon's parser, so a line
uncommented from .env.example keeps its trailing comment out of the value
in both. On the same .env doctor reaches the daemon's verdict, except that
placeholder keys still count as missing.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
@vercel

vercel Bot commented Oct 16, 2026

Copy link
Copy Markdown

@dmazhukov is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
Walkthrough

The doctor diagnostic now uses shared environment parsing and provider detection. It evaluates .env and process environment values, with defined process values taking precedence. It reports usable provider keys or the agent-SDK fallback.

Changes: Doctor provider detection

Layer / File(s) Summary
Environment parsing and capture
src/config.ts, src/cli.ts, test/env-loader.test.ts
parseEnvContent parses supplied text. Environment hydration returns a snapshot of process.env before .env values are copied. The CLI passes that snapshot to doctor effects.
Provider diagnostic and validation
src/cli/doctor-diagnostics.ts, test/cli-doctor-fixes.test.ts
The diagnostic uses the shared provider detector and overlays defined process values on file values. It reports usable keys or the agent-SDK fallback. Tests cover key filtering, environment overrides, environment-only keys, placeholders, and the fallback.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: rohitg00

Merge Risk: Low · up to 76985

Users with a custom env file may see doctor report a missing provider key even though the daemon has loaded it. Align doctor’s file path before merging, or accept this bounded diagnostic gap.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 5 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "fix(doctor): judge the LLM provider the way the daemon does" matches the main change. The PR makes the no-llm-provider-key doctor check use detectLlmProviderKind, read .env with the daemon'…
Linked Issues check Passed [#1572] requires agentmemory doctor to match the daemon's LLM-provider verdict. The reviewed changes make no-llm-provider-key use detectLlmProviderKind, parse .env with parseEnvContent, and …
Out of Scope Changes check Passed The changes stay within [#1572]. parseEnvContent, the pre-hydration environment snapshot, CLI wiring, diagnostic details, and related tests support consistent doctor and daemon provider detection. N…
Full details: Title check

Explanation

The title "fix(doctor): judge the LLM provider the way the daemon does" matches the main change. The PR makes the no-llm-provider-key doctor check use detectLlmProviderKind, read .env with the daemon's parser, and merge the shell environment over .env, so the doctor verdict matches the daemon's provider detection. The title is specific and concise, and it follows a conventional commit style. It does not list every detail, such as the GOOGLE_API_KEY recognition or the environment-key warning, which is acceptable for a title.

Full details: Linked Issues check

Explanation

[#1572] requires agentmemory doctor to match the daemon's LLM-provider verdict. The reviewed changes make no-llm-provider-key use detectLlmProviderKind, parse .env with parseEnvContent, and merge the pre-hydration process environment over file values. They preserve placeholder handling and cover the OpenAI toggle, agent-SDK fallback, environment precedence, missing files, and environment-only keys in tests. This addresses the required provider-detection parity.

Full details: Out of Scope Changes check

Explanation

The changes stay within [#1572]. parseEnvContent, the pre-hydration environment snapshot, CLI wiring, diagnostic details, and related tests support consistent doctor and daemon provider detection. No unrelated product behavior is shown in the supplied change summary.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

The daemon resolves provider keys through getMergedEnv(), where
process.env wins over ~/.agentmemory/.env. The provider check read the
file only, so an exported key, or OPENAI_API_KEY_FOR_LLM=false in the
shell, made doctor report the opposite of what the daemon does.

The check now merges the environment over the file. It takes the
environment from before hydrateProcessEnvFromFile() copied the file into
process.env: the copied values go stale once the file is edited, so
merging live process.env would keep a pasted key failing on re-check
after the editor fix. Keys found only in the environment are marked in
the detail, since a daemon started elsewhere does not see them.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
coderabbitai[bot]

This comment was marked as resolved.

getMergedEnv() merges the file under process.env and treats a missing
file as empty, so a daemon started with ANTHROPIC_API_KEY exported and
no ~/.agentmemory/.env resolves an LLM provider. The provider check
returned early on the missing file and reported the opposite.

The check now reads the environment regardless; env-missing stays the
diagnostic that reports an absent file. With neither a file nor a key
the detail still points at the env-missing fix, since the editor fix
has nothing to open.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
The caveat about a daemon started from another environment sat in
moreInfo, which runDoctor prints only through the interactive "more"
action on a failing check. On a passing one it printed just the detail,
so the one configuration the caveat was written for never showed it.

The detail now carries it whenever a named key comes from the
environment, and a key the environment overrides is marked too, not
only one the file lacks. An undefined environment value no longer
blanks the file's key. env-missing no longer claims BM25-only when a
key is exported, and the key list names GOOGLE_API_KEY, which both
detectLlmProviderKind and realProviderKeys already accept.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
dmazhukov added a commit to dmazhukov/agentmemory that referenced this pull request Oct 17, 2026
Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 3 potential issues.

Devin Review

Comment thread src/cli.ts Resolved
ok: real.length > 0,
detail: real.length > 0 ? `found: ${real.join(", ")}` : "no provider key set",
};
for (const k of placeholderProviderKeys(env)) delete env[k];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Placeholder Gemini key masks valid alias

With placeholder GEMINI_API_KEY and real GOOGLE_API_KEY, no-llm-provider-key passes after dropping the placeholder. The daemon's detectProvider selects GEMINI_API_KEY first, so Gemini calls use the placeholder instead.

Learn more

The provider check removes placeholder values from its copy of the merged environment. The actual detectProvider does not remove them: it treats any nonempty value as a real key and prefers GEMINI_API_KEY over its GOOGLE_API_KEY alias. As a result the doctor's positive result does not correspond to the key the daemon will send to Gemini. The same discrepancy can affect which provider wins when a higher-priority provider has a placeholder.

Example: Put GEMINI_API_KEY=your-key-here and GOOGLE_API_KEY=real-google-key in the file. Doctor says found: GOOGLE_API_KEY; the worker builds its Gemini provider with your-key-here and requests fail.

Recommended fix: Check the actual selected provider and its selected credential against placeholder values, rather than deleting placeholder keys before provider selection. Account for the Gemini alias precedence and the priority order in detectProvider.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main gives the same verdict for this env (found: GOOGLE_API_KEY). The placeholder is picked on the daemon side, src/providers/index.ts:98 (GEMINI_API_KEY || GOOGLE_API_KEY), which this PR does not touch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The daemon side is now #1588: a placeholder GEMINI_API_KEY no longer shadows a real GOOGLE_API_KEY, and doctor and the daemon share one placeholder rule.

Comment on lines +206 to +212
const real = realProviderKeys(env).filter(
(k) =>
detectLlmProviderKind({
[k]: env[k],
OPENAI_API_KEY_FOR_LLM: env.OPENAI_API_KEY_FOR_LLM,
}) === "llm",
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Unused provider keys reported as active

When several keys exist, no-llm-provider-key reports each as usable despite the daemon selecting only one. detectProvider stops at the first provider, so the verdict misidentifies the active key.

Learn more

The provider-kind helper answers whether a given configuration has any LLM route; it does not say which route wins. Running it once per key treats all valid keys as used, even though detectProvider selects the first provider in its priority order. The result is misleading when debugging which credential a running worker uses, especially when one key exists only in the shell.

Example: With OPENAI_API_KEY=sk-openai-real and ANTHROPIC_API_KEY=sk-ant-real, doctor reports both as found. The worker picks OpenAI; it never uses the Anthropic key for its primary provider.

Recommended fix: Derive the displayed active key from the same provider-selection priority and alias logic as detectProvider, while retaining a separate check for whether any working route exists.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main prints the same list for this env (found: ANTHROPIC_API_KEY, OPENAI_API_KEY); this PR does not change what the detail lists. Naming only the key detectProvider picks would be a separate change.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
coderabbitai[bot]

This comment was marked as resolved.

The daemon loads AGENTMEMORY_ENV_FILE when it is set, so doctor judged the provider key from a file the daemon does not read.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
@dmazhukov

Copy link
Copy Markdown
Contributor Author

Merged main to clear the conflict with #1592. Since #1592 the daemon loads AGENTMEMORY_ENV_FILE when it is set, so doctor now reads the same file (0d70da3, test against dist/cli.mjs).

Left alone: the env-missing fix (agentmemory init) still writes ~/.agentmemory/.env, the diagnostic messages still name that path, and the editor fix path is not pinned by a test.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Devin Review

Comment thread src/cli.ts
function buildDoctorEffects(): DoctorEffects {
return {
envFileExists: () => existsSync(join(homedir(), ".agentmemory", ".env")),
envFileExists: () => existsSync(doctorEnvPath()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing custom env file remains unfixed

When AGENTMEMORY_ENV_FILE points to a missing file, env-missing fails but its fix creates the default file. runInit ignores the custom path, so doctor reports a successful fix while the custom file remains missing.

Learn more

The env-missing check now inspects the path selected by AGENTMEMORY_ENV_FILE. Its fix invokes runInit, but runInit always targets the default file. As a result, the fix can create an unrelated file, or do nothing when that default file already exists, and the re-check still fails.

Example: Set AGENTMEMORY_ENV_FILE=/tmp/agent.env with neither /tmp/agent.env nor ~/.agentmemory/.env present. Choose Fix for env-missing. Doctor creates ~/.agentmemory/.env, then reports that /tmp/agent.env is still missing.

Recommended fix: Pass ctx.envPath to the env-missing fix and copy the template to that path with the same exclusive-create behavior as runInit; keep standalone init's default-path behavior unchanged.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/cli.ts
baseUrl: getBaseUrl(),
viewerUrl: getViewerUrl(),
envPath: join(homedir(), ".agentmemory", ".env"),
envPath: doctorEnvPath(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Custom env paths retain default-file guidance

With AGENTMEMORY_ENV_FILE set, checks read the custom file, but diagnostic messages and fix previews still name ~/.agentmemory/.env. Users can follow instructions for a different file.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@rohitg00
rohitg00 merged commit c5b8ebf into rohitg00:main Oct 26, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

doctor: no-llm-provider-key disagrees with the daemon's provider detection

2 participants