Repository navigation
Conversation
added 15 commits
September 19, 2026 22:12
Wire the deployed buyer crew to a deployed seller by ARN, and close the remaining gaps between the ECS and AgentCore runtimes for the buyer. Buyer -> seller integration (AgentCoreSellerProxy): - New OpenDirect-shaped proxy backed by bedrock-agentcore:InvokeAgentRuntime against the seller MCP runtime. Issues MCP tools/call over the Streamable-HTTP transport with the required Accept/contentType/protocol headers and the initialize -> notifications/initialized handshake, threads Mcp-Session-Id for microVM stickiness, and parses SSE-framed results. Maps structured seller products to OpenDirect via the shared wire mapper. - crew_tools: use the proxy for the ARN seller case (was a localhost stub). - New CFN stack infra/aws/agentcore/auth-agentcore.yaml: CFN-owned buyer runtime execution role with a scoped runtime/a4a_aamp_seller_* invoke wildcard, restricted to same-account / trusted cross-account (cross-org uses the gateway + CUSTOM_JWT path). deploy.sh deploys it and passes the role via `agentcore configure --execution-role`. AgentCore deploy parity fixes (deploy.sh / requirements.txt): - Add PYTHONPATH=/app/src so ad_buyer is importable in the runtime image. - Default the LLM + memory models to Bedrock Claude Sonnet (was nova-pro / nova-lite), matching settings.py and the Bedrock migration. - Pin crewai>=1.14.4,<2.0.0 (was ==1.10.1) to match pyproject.toml; the Bedrock fix patch and DealBookingFlow rely on CrewAI >=1.14. Tests: - Unit: tests/unit/agentcore/test_agentcore_seller_proxy.py (11) covering MCP result parsing/mapping, required MCP headers, the initialize handshake ordering + initialize-once, and SSE-framed response parsing. - Integration: tests/integration/agentcore/test_runtime.py adds TestBuyerSellerIntegration (deployed buyer->seller loop asserts real seller inv-* inventory via CloudWatch); raise invoke timeouts for the real crew runtime on Claude. - Fix test_agent_hierarchy temperature assertions to be model-aware: build_llm correctly omits temperature for Claude families that reject it (e.g. claude-sonnet-5), so assert the tuned value only when the configured model accepts temperature. All three buyer runtime integration tests + the buyer->seller loop verified live (real inv-* inventory reaches the buyer through the proxy).
…point
Route CrewAI's LLM calls through Amazon Bedrock's Anthropic-compatible
(Messages) endpoint instead of the Converse provider, and retire Amazon
Nova as the default model.
Motivation
----------
The Converse path required a monkey-patch (patches/crewai_bedrock_fix.py)
to strip orphaned toolUse/toolResult blocks. Bedrock's Anthropic Messages
endpoint (bedrock-runtime .../anthropic) assembles tool turns itself, so
that class of ValidationException does not arise. Nova Pro is being
deprecated; current-generation Claude tiers (Sonnet 5, Haiku 4.5) are
served on the Messages API.
Changes
-------
- llm.build_llm: add an Anthropic-Messages-compatible base_url branch,
selected via ANTHROPIC_COMPATIBLE_LLM_API_BASE_URL. Takes precedence
over the OpenAI-compatible branch so Claude is never misrouted to the
OpenAI client (Claude is not served on Bedrock's Chat Completions).
- settings: add anthropic_compatible_llm_api_{base_url,key}; default
model anthropic/claude-sonnet-4-5 -> anthropic/claude-sonnet-5.
- http_main: apply the Converse sanitizer only on the legacy Converse
path (bedrock/ model + no Messages base URL); on the Messages path
apply the new strict-strip patch instead.
- patches/crewai_bedrock_anthropic_fix.py (new): strip the tool "strict"
field that CrewAI (>=1.15) emits, which Bedrock's Anthropic endpoint
rejects (tools.0.custom.strict: Extra inputs are not permitted). Scoped
to Bedrock base URLs only; real Anthropic is unchanged.
- agentcore memory patch + crew_tools fallback: Nova -> Claude Haiku 4.5.
- infra/aws/agentcore/deploy.sh: source .env; default to Sonnet 5 /
Haiku 4.5 via the Messages endpoint; set PYTHONPATH=/app/src so the
package resolves under the toolkit's module entrypoint.
- infra/aws/agentcore/requirements.txt: align crewai with pyproject
(>=1.14.4) and add iab-agentic-primitives via a git-free archive URL
(the build image has no git binary).
- .env.example: document the Messages-endpoint configuration.
Compatibility
-------------
Backward-compatible: the Converse provider and its patch remain available
for anyone using a bedrock/ Converse model. Removing the Converse patch
entirely is deferred to a follow-up change.
Testing
-------
- Unit + mocked integration: 3462 passed.
- Live AgentCore integration (deployed dev runtime): 4/4 passed
(chat + three crew-plan invocations).
OpenAI's newer models (GPT-5.x, o1/o3/o4 reasoning models) reject the legacy max_tokens parameter on Bedrock's OpenAI-compatible endpoint with "400 invalid_request_error: Unsupported parameter: 'max_tokens' is not supported with this model." They take max_completion_tokens instead. build_llm now routes the token cap to max_completion_tokens for those model families (matched as substrings so Bedrock inference-profile ids like us.openai.gpt-5.6-sol and raw ids like gpt-5/o1 are covered) and keeps max_tokens for everything else (gpt-4o, Claude, Nova, ...). Surfaced while running the buyer crew against a Bedrock OpenAI (GPT-5.6) model to validate the "any model on Bedrock" path; with the fix the GPT crew invoke completes cleanly. Adds regression tests for the helper and the build_llm parameter routing.
Mirrors the seller fix. The crew path runs Claude on Bedrock's Anthropic Messages endpoint (x-api-key bearer token, not SigV4), so a token baked into --env expires within hours and 403s every crew call. - ad_buyer.llm.bedrock_token: mint a fresh bearer token from the runtime's own execution role at startup (authoritative on a Bedrock base URL so a stale baked key never shadows it), with a reactive refresh path; a non-Bedrock base URL still respects an operator-supplied key. Never crashes startup. - Wired into the http entrypoint at module load. - auth-agentcore.yaml: the CFN-owned buyer runtime role now grants bedrock:CallWithBearerToken (the bearer-token Messages path) alongside InvokeModel; deploy.sh stops baking an empty/stale key. - Adds aws-bedrock-token-generator to requirements. Live-verified: buyer crew campaign-planning integration tests pass under real Bedrock calls with no expired-token or permission errors.
…/refresh) For the AWS Bedrock AgentCore deployment path: when the buyer discovers a seller runtime fronted by a CUSTOM_JWT authorizer that advertises an OAuth2 token endpoint + scope in its AAMP registry 'authentication' object, it must present Authorization: Bearer <JWT>. OAuthTokenProvider mints that JWT via the client_credentials grant. - Reads token endpoint + scope from the DISCOVERED record; holds the buyer's own client_id/secret from settings (never from the registry, never logged). - POSTs grant_type=client_credentials (HTTP Basic), caches keyed by (token_endpoint, scope), refreshes proactively before expiry and reactively via invalidate() on a 401. Same helper serves a BYO-IdP token endpoint. - Additive: the legacy per-seller API-key path (AuthMiddleware) and the arn: AgentCore SigV4 proxy path are unchanged — OAuth applies only to a discovered seller that advertises an issuer (the cross-org case those paths can't cover). 10 unit tests (mint, cache, proactive/reactive refresh, per-scope keys, unconfigured/error handling, secret never logged).
…_JWT) When the seller runtime enforces a CUSTOM_JWT authorizer (AWS Bedrock AgentCore deployment path), SigV4/boto3 InvokeAgentRuntime is rejected — per the AgentCore docs, an OAuth runtime must be reached via a raw HTTPS request to the invocations URL with a Bearer token. This wires that path for the buyer's catalog/planning calls (MCP surface). - transport_selector.select_seller_client(): arn without OAuth -> SigV4 proxy; arn/https WITH advertised OAuth -> JWT path; https without OAuth -> plain MCP. agentcore_invocations_url() builds the verified HTTPS URL from an ARN (:->%3A, /->%2F, /invocations?qualifier=DEFAULT). - AgentCoreSellerProxy gains a JWT transport mode: same OpenDirect surface DealBookingFlow needs, but frames MCP over an httpx POST with a Bearer JWT (from OAuthTokenProvider), re-minting once on a 401. - IABMCPClient: optional OAuth bearer + no longer forces /mcp/sse on a full AgentCore invocations URL. - crew_tools routes through the selector when the seller advertises OAuth (SELLER_TOKEN_ENDPOINT); deploy.sh forwards the seller token endpoint/scope, the buyer's own client creds, and SELLER_MCP_RUNTIME_ARN (each only when set). Live-verified: the buyer crew reaches the seller's real inv-* catalog through the JWT/HTTPS MCP call to the CUSTOM_JWT seller MCP runtime (no 401/403). 14 unit tests for the selector; buyer unit suite green.
Add tests/integration/test_registry_roundtrip.py (task 7.3): seed the library registry double, discover the seller via AampRegistryClient over an httpx ASGITransport (in-process, no hosted registry), and assert the transport selector routes an OAuth-advertising https endpoint to the JWT client and an arn: (no OAuth) to the SigV4 proxy. Also covers the fail-closed contract (OAuth advertised + no buyer creds -> RuntimeError). 3 passed.
…anager credentials + CUSTOMER_VPC path Give the buyer an enterprise-grade durable-storage option (Req 12, Scope 1): the infrastructure + secret management + VPC reachability. The app-side Postgres backend that consumes it is a separate follow-up (Scope 2). - storage.yaml: add Aurora Serverless v2 (aurora-postgresql 16.9, db.serverless) beside Redis, with ServerlessV2ScalingConfiguration MinCapacity=0 (scale-to-zero) and ManageMasterUserPassword (RDS-managed, auto-rotating Secrets Manager secret — no plaintext password in SSM, CFN, or env). Export the secret ARN + Aurora endpoint/port/dbname. Redis untouched. - network-agentcore.yaml (new): AgentCore runtime SG, 443 self-ingress, and VPC interface endpoints (bedrock-agentcore/bedrock-runtime/sts/secretsmanager + ECR + Logs) so a private-subnet CUSTOMER_VPC runtime can reach its services. - main-agentcore.yaml (new): AgentCore root stack (network -> storage -> agentcore-network). Creates the Aurora SG in-stack so the shared network.yaml (ECS path) is untouched. - deploy.sh: add --storage postgres. Deploys the stack in CUSTOMER_VPC mode, forwards NON-secret env (DB_SECRET_ARN/AURORA_ENDPOINT/AURORA_PORT/DB_NAME/ REDIS_URL) — never the password — and idempotently grants the runtime role secretsmanager:GetSecretValue + scoped kms:Decrypt. Default stays sqlite. - tests: assert Aurora + RDS-managed secret + MinCapacity=0 + endpoints + 443 self-ingress + the deploy postgres branch (incl. that DB_PASSWORD never appears in env). - .gitignore: ignore the generated .packaged-agentcore.yaml build artifact. Verified live: stack CREATE_COMPLETE, Aurora available (16.9, MinCapacity 0.0), RDS-managed secret created (rds!cluster-...), runtime recreated in VPC mode and READY, secret grant applied.
…r credentials Route the buyer's stores to a durable Postgres backend on deployed AgentCore runtimes (opt-in via --storage postgres), replacing per-container SQLite. - connection_factory + db_secret: resolve the RDS-managed DATABASE_URL from a Secrets Manager secret at startup; fail-safe to SQLite when absent/unreadable. - http_main: resolve durable Postgres at startup from DB_SECRET_ARN (never logs the password); falls back to default storage with an explicit reason. - Wire the store connection seams (campaign/order/pacing/deal/adserver stores, audience_audit_log, health) to the shared connection factory. - deploy.sh: grant the DB-secret read policy to the SPECIFIC agent's runtime role (resolve via list/get-agent-runtime, then the agent-scoped yaml block) instead of the first execution_role in the file. - pyproject/requirements: synchronous psycopg[binary] + boto3 for the postgres/production extras.
…m discover_inventory The buyer proxy's check_avails called a non-existent seller MCP tool (discover_inventory), so it built an AvailsResponse missing every required field and raised ValidationError on every availability check — surfaced (and mislabeled) as 'Error parsing dates' 320x per crew run, driving retry loops. - check_avails now calls the seller's real check_avails MCP tool (honest availability), falling back to get_pricing for a seller runtime that predates it; either way it yields a contract-valid AvailsResponse. - get_product path points at the real get_product_details tool. - avails_check tool: split date parsing into its own try so a downstream response ValidationError is no longer mislabeled as a date-format error. - Unit tests for the real-avails path, the get_pricing fallback, and the error-result path.
…net 5 The buyer crew's latency is dominated by the level2/3 sub-agent fan-out (~165 LLM calls per plan). Default those sub-agents to Haiku 4.5 (fast, temperature-accepting) while keeping the single level-1 manager on Sonnet 5 for plan quality; both overridable at deploy via DEFAULT_LLM_MODEL / MANAGER_LLM_MODEL. - deploy.sh: LLM_MODEL default -> Haiku; new MANAGER_MODEL default -> Sonnet 5; wire MANAGER_LLM_MODEL from MANAGER_MODEL (was collapsed onto LLM_MODEL). - crew_tools: honor MANAGER_LLM_MODEL for the manager instead of forcing it to DEFAULT_LLM_MODEL. - conftest: per-test wall-clock timing summary for the AgentCore live suite.
Make the store-probe prompts imperative — name the store operation and forbid answering from memory — so the crew is more likely to call the campaign/order/ pacing/deal tool instead of replying conversationally. NL routing is still non-deterministic, so the skip-not-fail guard is unchanged.
Motivation: The proxy's hand-rolled _apply_filters diverged from the REST/ECS client's client-side catalog filter: it excluded products on channel where the REST path deliberately does not (buyer free-text channel vocab is not the seller inventory_type taxonomy), zeroing out channel-filtered searches on the deployed path. Changes: - agentcore_seller_proxy.py: list_products / list_products_tolerant now filter raw wire items through the OpenDirect client's _filter_wire_products (the single reconciled filter implementation) instead of the proxy-local copy, then map. Imported as a private peer-module symbol by design. - test_agentcore_seller_proxy.py: updated to assert the shared-filter behavior (channel not excluded; adFormat normalization preserved). Compatibility: In-boundary (agentcore). Behavioral fix: channel-filtered searches now return matching inventory instead of an empty list. Testing: Proxy unit tests pass.
Motivation: The buyer runtime mints its own short-lived Bedrock bearer token at startup for the Anthropic Messages path, but the toolkit-created execution role only carries InvokeModel (the SigV4/Converse path), so the runtime 403s on bedrock:CallWithBearerToken. Mirrors the seller's self-sustaining grant so both repos behave identically. Changes: - deploy.sh: extract _resolve_runtime_role_arn (resolves the role for THIS agent specifically -- the yaml holds every agent's config, so a bare grep|head -1 picks the wrong agent's role) and add grant_bedrock_bearer_token attaching a separately-named inline policy (survives the toolkit's per-deploy execution-policy rewrite). Best-effort + idempotent. grant_secret_access now uses the same resolver. Compatibility: In-boundary infra; best-effort grant, non-fatal on failure.
Motivation: Document how the buyer connects to an OAuth-protected AgentCore seller and the buyer's durable-storage path, surfaced during the enterprise-auth work. Changes: - docs/api/seller-discovery.md: connecting to a CUSTOM_JWT seller over OAuth -- OAuthTokenProvider (client_credentials, buyer-owned client id/secret from settings, cached + refreshed) and select_seller_client transport selection. - docs/architecture/storage-backends.md: buyer Aurora Serverless v2 + VPC durable path and the schema-pg thin-adapter backend. Compatibility: Docs-only.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Durable Postgres app backend for the buyer (Aurora Serverless v2 scale-to-zero,
Secrets-Manager credentials, CUSTOMER_VPC), real seller avails/product MCP calls,
a proxy filter aligned to the OpenDirect client, and a Haiku/Sonnet sub-agent split.
Changes
schema_pg.py(PG DDL derived fromschema.py),pg_connection.py(sqlite3-API psycopg adapter),
connection_factory.py,db_secret.py; storeseams (deal/order/campaign/pacing/adserver/audience-audit/health). Aurora v2 +
network/main agentcore CFN;
--storage postgresdeploy path.check_avails/product MCP calls (no phantomdiscover_inventory); reuse the OpenDirect client filter in the proxy; grantruntime role
bedrock:CallWithBearerToken(Req 11).Shared-surface note
Edits several non-agentcore storage files (
storage/*_store.py,pyproject.toml,cloudformation/storage.yaml) to add the Postgres seam — used by the FastAPI pathtoo. Covered by
test_pg_backend.py(231) +test_pg_round_trip.pyincl. theTestAllStoresOnPostgressweep, andtest_iac_templates.py.Note on buyer JSONB
The seller's JSONB codec bug does NOT apply here: the buyer stores JSON in TEXT
columns with symmetric store-side
json.dumps/json.loads, so psycopg returnsstridentically to sqlite3 — no driver codec needed.Testing
Buyer unit + integration suites green (pg-backend, pg-round-trip, iac-templates,
seller-proxy); full suite validated (~3,440) in prior runs.
Base:
buyer-mr2-crossorg-auth(stacked).