Skip to content

Add Redis subscribe usage ingest#138

Open
Willxup wants to merge 9 commits into
mainfrom
feature/cpa-redis-subscribe
Open

Add Redis subscribe usage ingest#138
Willxup wants to merge 9 commits into
mainfrom
feature/cpa-redis-subscribe

Conversation

@Willxup
Copy link
Copy Markdown
Owner

@Willxup Willxup commented May 22, 2026

Summary

  • Add Redis SUBSCRIBE usage ingest with Redis/HTTP fallback into the existing redis_usage_inboxes buffer.
  • Split remote ingest from local inbox processing so usage decoding and aggregation stay unchanged.
  • Harden Redis RESP parsing limits and ignore local AGENTS.md instructions.

Test plan

  • go test ./internal/poller/... ./internal/cpa ./internal/app
  • go test ./...
  • Restarted backend locally with project .env and verified ping endpoint during testing

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the usage ingestion architecture by replacing the legacy RedisDrain with a more modular system comprising a RedisIngestRunner and a RedisProcessRunner. The new implementation introduces a state machine that supports Redis SUBSCRIBE, Redis pull, and HTTP pull with built-in fallback and recovery logic. Additionally, the Redis client now includes security limits for RESP parsing to improve stability. Review feedback highlighted several areas for improvement: ensuring the backfill logic fully drains queues during startup to prevent data lag, allowing the ingest runner to "upgrade" back to higher-priority modes if they become available after a fallback, and making the Redis subscription channel configurable to ensure proper data isolation in shared environments.

Comment thread internal/poller/redis_ingest_runner.go Outdated
Comment thread internal/poller/redis_ingest_runner.go
Comment thread internal/poller/redis_ingest_state.go Outdated
Willxup added 6 commits May 23, 2026 00:23
Ensure subscribe reconnect catch-up drains legacy pull sources to an empty batch while keeping per-batch details at debug level.
Keep usage queue keys, subscribe channel, and Redis commands in the CPA endpoint constants so pull and subscribe paths share one source of truth.
Wait for asynchronous log writes before canceling test runners so slower CI schedulers do not miss expected backfill and subscribe debug logs.
Ensure fixed redis_pull mode clears stale status errors when Redis succeeds again after a full pull-chain failure.
Reject Redis queue batch sizes above the RESP array limit and make poller log capture safe for race-enabled tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant