Skip to content

Support any OpenAI-compatible LLM endpoint for summarization - #59

Merged
jonocodes merged 8 commits into
devfrom
claude/ai-summary-model-abstraction-gmfxtd
Jul 7, 2026
Merged

Support any OpenAI-compatible LLM endpoint for summarization#59
jonocodes merged 8 commits into
devfrom
claude/ai-summary-model-abstraction-gmfxtd

Conversation

@jonocodes

@jonocodes jonocodes commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

Refactored the summarization system to work with any OpenAI-compatible chat-completions endpoint, eliminating provider-specific code paths. This enables support for cloud services (Groq, OpenAI, Gemini), local model servers (llama.cpp, Ollama, LM Studio, vLLM), and custom endpoints through a single unified implementation.

Key Changes

  • Unified API layer: Replaced separate callGroqApi() and callOpenAiApi() functions with a single callChatApi() that works with any OpenAI-compatible endpoint
  • Provider configuration as data: Converted providers from a closed union type to a data-driven model where each provider is defined in the PROVIDERS array with endpoint URL, API key requirements, and available models
  • Added new providers:
    • Gemini (via Google's OpenAI-compatible endpoint)
    • Local / Custom provider allowing users to specify any base URL and model name manually
  • Optional API key support: Authorization header is now conditionally included, allowing local servers that don't require authentication to work out of the box
  • Custom endpoint handling: Added baseUrl parameter to summarizeText() and testApiConnection() to support user-supplied endpoints for custom providers
  • UI improvements:
    • Server URL input field appears only for the "Local / Custom" provider
    • Model selection switches between dropdown (built-in providers) and text input (custom provider)
    • API key hints are now provider-specific and stored in configuration
    • Test connection validation checks both API key requirements and custom URL requirements
  • Cookie management: Added getSummaryCustomBaseUrlFromCookie() and setSummaryCustomBaseUrlInCookie() to persist custom server URLs
  • Updated defaults: Changed default Groq model from Llama 3.3 70B to Qwen3 32B (reflecting Groq's model deprecation)

Implementation Details

  • The SummaryProvider type is now a plain string alias rather than a closed union, allowing extensibility
  • resolveEndpoint() helper determines the correct chat-completions URL based on provider type and user input
  • getProviderConfig() utility provides type-safe access to provider configuration
  • All provider-specific logic is eliminated from the API call path—the endpoint URL and model name are the only variables

https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc

@claude
Support any OpenAI-compatible LLM for AI summarization
4c0efee 
Collapse the per-provider summarization calls into a single config-driven
call path. Groq, OpenAI, Gemini, and local servers (llama.cpp, Ollama, LM
Studio, vLLM) all speak the same /chat/completions shape, so a provider is
now a row in the PROVIDERS table with a base URL rather than a bespoke
function.

- Add Gemini as a built-in provider via its OpenAI-compatible endpoint.
- Add a "Local / Custom" provider: user supplies a base URL and free-text
  model name; API key optional (Authorization header omitted when absent).
- Update the default Groq model to Qwen3 32B (Groq is deprecating Llama 3.3
  70B) and refresh the stored-model default to match.
- Preferences UI renders a URL field and free-text model input for custom
  providers, and uses per-provider API-key hints.
- Document the CORS requirement for browser-initiated requests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@netlify

netlify Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for savrdev ready!

Name Link
🔨 Latest commit cf0bb4b
🔍 Latest deploy log https://app.netlify.com/projects/savrdev/deploys/6a4c9457280dae0008486a65
😎 Deploy Preview https://deploy-preview-59--savrdev.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

claude added 7 commits July 6, 2026 23:05
@claude
Fix custom provider on Submit screen; surface real summarizer errors
b99a9b7 
The Submit screen's "Test Summary" path was not updated for the provider
abstraction: it never passed the custom base URL (so the Local/Custom
provider threw "No API endpoint configured") and hard-required an API key
(blocking keyless local servers). Mirror ArticleScreen's provider handling
there.

Also include the underlying error message in the failure toast in both
screens, so a deprecated model or a provider outage is visible to the user
instead of a generic "Failed to generate summary".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Scale summary output-token limit by detail level
4f04822 
The output cap was a fixed max_tokens: 1000, which truncated longer
summaries — especially with reasoning models like Gemini 2.5, where part of
the token budget is consumed by internal thinking before the answer is
emitted. Derive the limit from the detail level instead (512–6144 tokens),
and give the connection test some headroom too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Fetch live model lists from the provider's /v1/models endpoint
8da098a 
Hardcoded model lists drift as providers add and retire models. Add
fetchAvailableModels(), which derives the OpenAI-compatible /v1/models URL
from the chat-completions endpoint and returns the provider's current model
ids (stripping Gemini's "models/" prefix). A refresh button beside the model
selector in Preferences populates the dropdown from that live list, falling
back to the curated defaults when nothing has been fetched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Remove hardcoded model lists; model field is fetch-or-type
8f622c3 
Drop the curated per-provider model arrays entirely — they drift out of date
as providers add and retire models. The model field is now free text by
default (works offline and for local servers with no key), and the ↻ button
upgrades it to a dropdown populated from the provider's live /v1/models
endpoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Steer summary length via prompt, not max_tokens
102b88b 
max_tokens was being used to shape summary length per detail level, but a
token cap doesn't shorten output — it truncates it mid-sentence, which is a
worse failure mode than running a bit long. Move length control into the
prompt instead (a flat "keep it under 800 words" instruction alongside the
existing per-level qualitative guidance), and replace the tiered token caps
with a single flat MAX_OUTPUT_TOKENS = 4096 that exists only as a safety net
against a model ignoring the prompt and generating without bound.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Strip <think> chain-of-thought from summaries
d8ec747 
Reasoning models such as Qwen3 (on Groq) and DeepSeek-R1 emit their
chain-of-thought inline in the message content wrapped in <think>...</think>
tags. We were passing that through verbatim, so the reasoning leaked into the
summary. Strip those blocks (and any stray unpaired tags) at the single
content-extraction point so it applies to every provider and model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@claude
Save summary provenance metadata (model, provider, date)
cf0bb4b 
Record which model/provider produced a summary and when, stored as a
summaryMeta object on the article alongside the summary text. Cleared with
the summary, and surfaced as a "Generated by <model> (<provider>) · <date>"
caption in the summary drawer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeYafqjhHzqH1GCXwo1grc
@jonocodes
jonocodes merged commit c723a30 into dev Jul 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jonocodes @claude