Skip to content

Add mandatory inference prefixes based on regions for AWS bedrock models#224

Open
puffitos wants to merge 1 commit intocharmbracelet:mainfrom
puffitos:main
Open

Add mandatory inference prefixes based on regions for AWS bedrock models#224
puffitos wants to merge 1 commit intocharmbracelet:mainfrom
puffitos:main

Conversation

@puffitos
Copy link
Copy Markdown

  • I have read CONTRIBUTING.md.
  • I have created a discussion that was approved by a maintainer (for new features).

Summary

Resolves #223

  • Bedrock models now get their inference profile prefix resolved at load time based on AWS_REGION / AWS_DEFAULT_REGION, with per-model region filtering: use the regional prefix if available, fall back to global, or exclude the model if neither applies (e.g. Opus 4/4.1 are US-only).
  • Added TestBedrockConfigRegions to assert every model in bedrock.json has at least one region configured, and updated TestBedrockProvider to verify correct model count and prefix per region.

Why this approach

Simpler to maintain: one loop, one switch, no extra helpers. The tests validate the config itself, so adding or updating models in bedrock.json will fail fast if regions are missing.

Tradeoff

The Regions field on the Model struct is currently only used by Bedrock. This feels like feature creep on a shared type, but creating a separate Bedrock-specific config struct seemed like over-engineering for a single field. Happy to hear suggestions from maintainers on a better home for this.

Testing done

  • All unit tests pass.
  • Tested with a personal AWS account in us-east-1 — confirmed that inference profiles are mandatory for all current Anthropic models on Bedrock (direct model ID invocation returns ValidationException). See comment on #223 for details.
  • Used this branch with crush for a couple of days against a work Bedrock account (EU region) without issues.
  • Used this branch to fire a couple of prompts with Opus 4.6 using a us-east-1 personal account.

@puffitos puffitos requested a review from a team as a code owner March 22, 2026 10:01
@puffitos puffitos requested review from meowgorithm and raphamorim and removed request for a team March 22, 2026 10:01
@meowgorithm
Copy link
Copy Markdown
Member

meowgorithm commented Mar 26, 2026

Thanks for this one!

Just to confirm, to greatly summarize the changes, this PR essentially adds support for AWS_REGION and AWS_DEFAULT_REGION, correct?

@meowgorithm
Copy link
Copy Markdown
Member

meowgorithm commented Mar 26, 2026

@puffitos I pushed a few changes on top of yours; can you verify they make sense and that everything's working as expected?

I'm also handling the environment variable in Fantasy (our provider and model abstraction layer) to assure that it doesn't fall through the cracks (charmbracelet/fantasy#189).

@meowgorithm meowgorithm requested review from andreynering and removed request for raphamorim March 26, 2026 18:53
@puffitos
Copy link
Copy Markdown
Author

@meowgorithm thanks for the review and addressing the Regions struct! The changes make sense, yeah. I'm not sure about the removal of the apac support in general and the fallback to global, as there's still one model (sonnet 4 from the bedrock config), which isn't available globally from most regions (see https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html).

According to the docs one shouldn't be able to do cross-region inference from an unsupported region (for example when AWS_REGION=ap-south-1. Or am I understanding that false? I'll try to test that once I get home.

@puffitos
Copy link
Copy Markdown
Author

puffitos commented Mar 28, 2026

I'm either not understanding how bedrock works, or the removal of the apac region will break the usage of Claude 4 for some users eventually.

The global inference profiles for Claude 4 don't exist in some regions, like Mumbai for example (ap-south-1):

❯ aws bedrock list-inference-profiles --region ap-south-1 \
    --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'global')].[inferenceProfileId,inferenceProfileName]" \
    --output text | rg "Sonnet 4$"

They exist in other regions, like us-east-1:

❯ aws bedrock list-inference-profiles --region us-east-1 \
    --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'global')].[inferenceProfileId,inferenceProfileName]" \
    --output text | rg "Sonnet 4$"
global.anthropic.claude-sonnet-4-20250514-v1:0	Global Claude Sonnet 4

A call to bedrock (just saying hi to Claude), fails as well in ap-south-1 (when using global), but works when using the apac inference profile. I've removed my account ID from the logs:

# us-east-1 has a global inference profile, call works
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:us-east-1:xxxxxxxxxxx:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region us-east-1
❯ echo $?
0

# using the fallback global won't work in a region that doesn't have a global profile available
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:ap-south-1:xxxxxxxxxxx:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region ap-south-1

aws: [ERROR]: An error occurred (ValidationException) when calling the Converse operation: The provided model identifier is invalid.

Additional error details:
message: The provided model identifier is invalid.

# Using the "correct" apac profile works in the affected region.
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:ap-south-1:xxxxxxxxxxx:inference-profile/apac.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region ap-south-1
❯ echo $?
0

I think we need to keep the apac region.

@puffitos puffitos force-pushed the main branch 2 times, most recently from 63c3c8b to b5abb70 Compare April 7, 2026 08:19
@meowgorithm
Copy link
Copy Markdown
Member

Apologies for dropping off here, @puffitos. I think that makes sense.

Do you mind making the change? After that, let's get this and charmbracelet/fantasy#189 merged (if you can take a look at that it would be appreciated too).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bedrock models don't work because of inference profiles are required

2 participants