Skip to content

feat: per-purpose inference config (LLM / embedder / graph) #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ttupper92618 merged 4 commits into from
Mar 14, 2026
Merged

feat: per-purpose inference config (LLM / embedder / graph) #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c2ceb90
feat: per-purpose inference config for main LLM, embedder, and graph LLM
Mar 14, 2026
c6b3832
fix: derive auth diagnostics from per-purpose API keys
Mar 14, 2026
987d3c9
fix: preserve legacy health diagnostics fields for backward compat
Mar 14, 2026
bd45c22
fix: validate extraction strategy env var, gate graph auth on GRAPH_E…
Mar 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
# ── Inference provider (OpenAI-compatible) ────────────────────────────────────
# Point at hosted OpenAI, a local Ollama proxy, or foxmemory-infer.
# Shared defaults — used as fallback for any purpose-specific config below.
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_API_KEY=sk-...

# LLM used for memory extraction (fact extraction + ADD/UPDATE/DELETE decisions)
# ── Per-purpose inference config ──────────────────────────────────────────────
# Each purpose (main LLM, embedder, graph LLM) can target a different endpoint.
# Falls back to the shared OPENAI_BASE_URL / OPENAI_API_KEY when not set.
# This allows e.g. local inference for the main LLM with cloud for embeddings.

# Main LLM (fact extraction + ADD/UPDATE/DELETE decisions)
MEM0_LLM_MODEL=gpt-4.1-nano
# MEM0_LLM_BASE_URL= # falls back to OPENAI_BASE_URL
# MEM0_LLM_API_KEY= # falls back to OPENAI_API_KEY

# Embedding model — CANNOT be changed after the collection is created
MEM0_EMBED_MODEL=text-embedding-3-small
# MEM0_EMBED_BASE_URL= # falls back to OPENAI_BASE_URL
# MEM0_EMBED_API_KEY= # falls back to OPENAI_API_KEY

# ── Vector store (Qdrant) ─────────────────────────────────────────────────────
# When using the embedded image, Qdrant runs inside the container at 127.0.0.1.
Expand All @@ -27,6 +36,9 @@ FOXMEMORY_ANALYTICS_DB_PATH=/data/foxmemory-analytics.db
# NEO4J_USERNAME=neo4j
# NEO4J_PASSWORD=changeme
# MEM0_GRAPH_LLM_MODEL=gpt-4.1-mini
# MEM0_GRAPH_LLM_BASE_URL= # falls back to OPENAI_BASE_URL
# MEM0_GRAPH_LLM_API_KEY= # falls back to OPENAI_API_KEY
# MEM0_GRAPH_EXTRACTION_STRATEGY=tool_calling # or "json_prompting" for models without tool support
# MEM0_GRAPH_CUSTOM_PROMPT=
# MEM0_GRAPH_SEARCH_THRESHOLD=0.7
# MEM0_GRAPH_NODE_DEDUP_THRESHOLD=0.9
Expand Down
71 changes: 53 additions & 18 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,41 @@ const BUILD_COMMIT = process.env.GIT_SHA || process.env.BUILD_COMMIT || "unknown
const BUILD_IMAGE_DIGEST = process.env.IMAGE_DIGEST || "unknown";
const BUILD_TIME = process.env.BUILD_TIME || "unknown";

// --- Shared defaults (used as fallback for per-purpose config) ---
const OPENAI_BASE_URL = process.env.OPENAI_BASE_URL; // e.g. http://foxmemory-infer:8081/v1
const OPENAI_API_KEY = process.env.OPENAI_API_KEY || "local-infer-no-key";
const HAS_OPENAI_API_KEY = Boolean(process.env.OPENAI_API_KEY);

// --- Per-purpose inference config ---
// Each purpose (main LLM, embedder, graph LLM) can have its own provider, base URL,
// API key, and model. Falls back to the shared OPENAI_* values when not set.
// This allows e.g. local inference for the main LLM with cloud for embeddings.

// Main LLM (fact extraction, memory decisions)
const LLM_BASE_URL = process.env.MEM0_LLM_BASE_URL || OPENAI_BASE_URL;
const LLM_API_KEY = process.env.MEM0_LLM_API_KEY || OPENAI_API_KEY;

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Derive auth diagnostics from per-purpose API keys

By introducing MEM0_LLM_API_KEY/MEM0_EMBED_API_KEY/MEM0_GRAPH_LLM_API_KEY, deployments can now authenticate without setting OPENAI_API_KEY, but authMode and openaiApiKeyConfigured are still computed from process.env.OPENAI_API_KEY only. In that valid configuration, /health and startup diagnostics will incorrectly report local-default and false, which can mislead operators and monitoring when triaging production incidents. Please compute diagnostics from the effective keys (or expose per-purpose auth diagnostics) instead of the legacy shared variable.

Useful? React with 👍 / 👎.

const LLM_MODEL = process.env.MEM0_LLM_MODEL || "gpt-4.1-nano";

// Embedder
const EMBED_BASE_URL = process.env.MEM0_EMBED_BASE_URL || OPENAI_BASE_URL;
const EMBED_API_KEY = process.env.MEM0_EMBED_API_KEY || OPENAI_API_KEY;
const EMBED_MODEL = process.env.MEM0_EMBED_MODEL || "text-embedding-3-small";

// Graph memory (Neo4j). Enabled when NEO4J_URL + NEO4J_PASSWORD are set.
// MEM0_GRAPH_LLM_MODEL lets you use a separate, more capable model for entity/relation extraction.
// MEM0_GRAPH_LLM_* lets you use a separate, more capable model/endpoint for entity/relation extraction.
// Recommended hosted: gpt-4o-mini. Recommended local: Qwen2.5-14B-Instruct or Mistral-Small-3.1-24B.
const NEO4J_URL = process.env.NEO4J_URL || null;
const NEO4J_USERNAME = process.env.NEO4J_USERNAME || "neo4j";
const NEO4J_PASSWORD = process.env.NEO4J_PASSWORD || null;
const GRAPH_LLM_BASE_URL = process.env.MEM0_GRAPH_LLM_BASE_URL || OPENAI_BASE_URL;
const GRAPH_LLM_API_KEY = process.env.MEM0_GRAPH_LLM_API_KEY || OPENAI_API_KEY;
const GRAPH_LLM_MODEL = process.env.MEM0_GRAPH_LLM_MODEL || LLM_MODEL;
const GRAPH_EXTRACTION_STRATEGY = (() => {
const raw = process.env.MEM0_GRAPH_EXTRACTION_STRATEGY;
if (!raw) return undefined;
if (raw === "tool_calling" || raw === "json_prompting") return raw;
console.warn(`[config] invalid MEM0_GRAPH_EXTRACTION_STRATEGY "${raw}" — must be "tool_calling" or "json_prompting", falling back to "tool_calling"`);
return "tool_calling" as const;
})();

// Mutable effective models — start from env vars, overridden by DB on startup.
// Use these everywhere instead of the const env vars so hot-reload works.
Expand All @@ -61,8 +83,10 @@ function sanitizeBaseUrl(url?: string) {
}
}

const AUTH_MODE = HAS_OPENAI_API_KEY ? "api_key" : "local-default";
const OPENAI_BASE_URL_SANITIZED = sanitizeBaseUrl(OPENAI_BASE_URL);
const HAS_LLM_API_KEY = Boolean(process.env.MEM0_LLM_API_KEY || process.env.OPENAI_API_KEY);
const HAS_EMBED_API_KEY = Boolean(process.env.MEM0_EMBED_API_KEY || process.env.OPENAI_API_KEY);
const HAS_GRAPH_LLM_API_KEY = Boolean(process.env.MEM0_GRAPH_LLM_API_KEY || process.env.OPENAI_API_KEY);
const AUTH_MODE = (HAS_LLM_API_KEY || HAS_EMBED_API_KEY || (GRAPH_ENABLED && HAS_GRAPH_LLM_API_KEY)) ? "api_key" : "local-default";

// mem0 default prompts (copied from @foxlight-foundation/mem0ai prompts/index.ts).
// Used to show the effective prompt even when no custom prompt is set.
Expand Down Expand Up @@ -217,17 +241,17 @@ function createMemory(customPrompt?: string | null, customUpdatePrompt?: string
llm: {
provider: "openai",
config: {
apiKey: OPENAI_API_KEY,
apiKey: LLM_API_KEY,
model: effectiveLlmModel,
...(OPENAI_BASE_URL ? { baseURL: OPENAI_BASE_URL } : {})
...(LLM_BASE_URL ? { baseURL: LLM_BASE_URL } : {})
}
},
embedder: {
provider: "openai",
config: {
apiKey: OPENAI_API_KEY,
apiKey: EMBED_API_KEY,
model: EMBED_MODEL,
...(OPENAI_BASE_URL ? { baseURL: OPENAI_BASE_URL } : {})
...(EMBED_BASE_URL ? { baseURL: EMBED_BASE_URL } : {})
}
},
...(process.env.QDRANT_HOST
Expand Down Expand Up @@ -265,11 +289,12 @@ function createMemory(customPrompt?: string | null, customUpdatePrompt?: string
llm: {
provider: "openai",
config: {
apiKey: OPENAI_API_KEY,
apiKey: GRAPH_LLM_API_KEY,
model: effectiveGraphLlmModel,
...(OPENAI_BASE_URL ? { baseURL: OPENAI_BASE_URL } : {}),
...(GRAPH_LLM_BASE_URL ? { baseURL: GRAPH_LLM_BASE_URL } : {}),
},
},
...(GRAPH_EXTRACTION_STRATEGY ? { extractionStrategy: GRAPH_EXTRACTION_STRATEGY } : {}),
},
}
: {}),
Expand Down Expand Up @@ -497,8 +522,12 @@ app.get("/health", (_req, res) => {
embedModel: EMBED_MODEL,
diagnostics: {
authMode: AUTH_MODE,
openaiApiKeyConfigured: HAS_OPENAI_API_KEY,
openaiBaseUrl: OPENAI_BASE_URL_SANITIZED,
// Legacy fields (kept for backward compat with existing monitors)
openaiApiKeyConfigured: HAS_LLM_API_KEY,
openaiBaseUrl: sanitizeBaseUrl(LLM_BASE_URL),
// Per-purpose fields
apiKeyConfigured: { llm: HAS_LLM_API_KEY, embed: HAS_EMBED_API_KEY, graphLlm: HAS_GRAPH_LLM_API_KEY },
baseUrl: { llm: sanitizeBaseUrl(LLM_BASE_URL), embed: sanitizeBaseUrl(EMBED_BASE_URL), graphLlm: sanitizeBaseUrl(GRAPH_LLM_BASE_URL) },
Comment on lines +529 to +530

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve legacy health diagnostics fields

This change swaps diagnostics.openaiApiKeyConfigured/diagnostics.openaiBaseUrl for diagnostics.apiKeyConfigured/diagnostics.baseUrl in the health payloads, which is a breaking response-shape change for existing monitors and clients that read the old keys (the same replacement is also present in /v2/health). Because this commit is framed as per-purpose config support rather than a versioned API contract change, dropping the legacy fields will cause production integrations to see missing values immediately after deploy; keep the old fields as aliases (or explicitly version the schema) to avoid regressions.

Useful? React with 👍 / 👎.

graphEnabled: GRAPH_ENABLED,
neo4jUrl: NEO4J_URL,
graphLlmModel: GRAPH_ENABLED ? effectiveGraphLlmModel : null,
Expand Down Expand Up @@ -1536,8 +1565,12 @@ app.get("/v2/health", async (_req, res) => {
embedModel: EMBED_MODEL,
diagnostics: {
authMode: AUTH_MODE,
openaiApiKeyConfigured: HAS_OPENAI_API_KEY,
openaiBaseUrl: OPENAI_BASE_URL_SANITIZED,
// Legacy fields (kept for backward compat with existing monitors)
openaiApiKeyConfigured: HAS_LLM_API_KEY,
openaiBaseUrl: sanitizeBaseUrl(LLM_BASE_URL),
// Per-purpose fields
apiKeyConfigured: { llm: HAS_LLM_API_KEY, embed: HAS_EMBED_API_KEY, graphLlm: HAS_GRAPH_LLM_API_KEY },
baseUrl: { llm: sanitizeBaseUrl(LLM_BASE_URL), embed: sanitizeBaseUrl(EMBED_BASE_URL), graphLlm: sanitizeBaseUrl(GRAPH_LLM_BASE_URL) },
graphEnabled: GRAPH_ENABLED,
neo4jUrl: NEO4J_URL,
graphLlmModel: GRAPH_ENABLED ? effectiveGraphLlmModel : null,
Expand Down Expand Up @@ -2897,8 +2930,10 @@ const V2_OPENAPI_SPEC = {
type: "object",
properties: {
authMode: { type: "string" },
openaiApiKeyConfigured: { type: "boolean" },
openaiBaseUrl: { type: "string", nullable: true },
openaiApiKeyConfigured: { type: "boolean", description: "Legacy alias — reflects main LLM API key status." },
openaiBaseUrl: { type: "string", nullable: true, description: "Legacy alias — reflects main LLM base URL." },
apiKeyConfigured: { type: "object", properties: { llm: { type: "boolean" }, embed: { type: "boolean" }, graphLlm: { type: "boolean" } } },
baseUrl: { type: "object", properties: { llm: { type: "string", nullable: true }, embed: { type: "string", nullable: true }, graphLlm: { type: "string", nullable: true } } },
graphEnabled: { type: "boolean", description: "True when NEO4J_URL + NEO4J_PASSWORD are set." },
neo4jUrl: { type: "string", nullable: true },
graphLlmModel: { type: "string", nullable: true },
Expand Down Expand Up @@ -3363,8 +3398,8 @@ app.listen(PORT, () => {
JSON.stringify(
{
authMode: AUTH_MODE,
openaiApiKeyConfigured: HAS_OPENAI_API_KEY,
openaiBaseUrl: OPENAI_BASE_URL_SANITIZED,
apiKeyConfigured: { llm: HAS_LLM_API_KEY, embed: HAS_EMBED_API_KEY, graphLlm: HAS_GRAPH_LLM_API_KEY },
baseUrl: { llm: sanitizeBaseUrl(LLM_BASE_URL), embed: sanitizeBaseUrl(EMBED_BASE_URL), graphLlm: sanitizeBaseUrl(GRAPH_LLM_BASE_URL) },
llmModel: effectiveLlmModel,
embedModel: EMBED_MODEL,
qdrantHost: process.env.QDRANT_HOST || null,
Expand Down
Loading