Skip to content

feat: add asynchronous Omni Moderation for chat inputs and outputs - #591

Open
GeorgeXie2333 wants to merge 2 commits into
DEEIX-AI:devfrom
GeorgeXie2333:dev
Open

feat: add asynchronous Omni Moderation for chat inputs and outputs#591
GeorgeXie2333 wants to merge 2 commits into
DEEIX-AI:devfrom
GeorgeXie2333:dev

Conversation

@GeorgeXie2333

@GeorgeXie2333 GeorgeXie2333 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add asynchronous, OpenAI-compatible Omni Moderation for user prompt text, attached images, assistant text, and generated images.
  • Coordinate moderation with active conversation runs, emit live moderation events, and atomically block messages, redact unsafe output, and remove blocked generated files when a policy is triggered.
  • Preserve visible moderation alerts across stream completion and conversation reloads.
  • Add configurable policies for input/output text and images, including select-all category controls, service probing, concurrency limits, queue capacity, and timeout settings.
  • Add admin-only moderation configuration, event inspection, encrypted retained-content review, and anonymous statistics.
  • Correct moderation statistics so category breakdown rows do not inflate total hits or produce rates above 100%.
  • Add recovery and cleanup paths for delayed verdicts, worker failures, expired retained content, and interrupted streams.

Change type

  • Bug fix
  • Feature
  • Documentation
  • Refactor
  • Configuration / deployment
  • Security hardening
  • Other

Affected areas

  • Frontend / UI
  • Backend / API
  • Authentication / authorization
  • Conversations / streaming
  • Files / RAG / extraction
  • Model routing / providers
  • MCP / tools
  • Billing / payments
  • Admin console
  • Deployment / Docker / configuration
  • Documentation

Verification

  • Targeted Go tests for content moderation, conversation streaming, persistence, and secret encryption.
  • Go race-detector checks for content moderation concurrency paths.
  • Frontend TypeScript typecheck.
  • git diff --check.
  • Sanitized local streaming API smoke test confirmed a moderation block and an empty blocked assistant result.
  • Full repository test and production build suite; not run because validation was limited to targeted checks.

Screenshots, API examples, or logs

Configuration, migration, and compatibility notes

  • Adds the content_moderation system-settings namespace with:
    • base_url
    • api_key
    • model
    • timeout_seconds
    • max_concurrency
    • queue_capacity
    • policy_json
    • policy_version
  • Defaults to https://api.openai.com/v1 and omni-moderation-latest.
  • Moderation remains disabled by default because all policy category lists are initially empty.
  • The moderation API key is encrypted at rest and masked in admin responses.
  • AutoMigrate creates:
    • content_moderation_events
    • content_moderation_daily_stats
  • AutoMigrate adds moderation state, event ID, and category fields to conversation runs and messages. Existing rows use backward-compatible defaults; no data backfill is required.
  • Flagged text and isolated images are encrypted at rest. Retained content expires after 30 days, while metadata and anonymous statistics expire after 90 days.
  • Configuration and event details require super-admin access. Anonymous aggregate statistics are available to administrators.
  • Swagger and the generated TypeScript API contract were updated.
  • No new environment variables, public URLs, CORS rules, or Docker changes are required.

Documentation

  • Documentation is not needed for this change.
  • Documentation was updated.
  • Documentation still needs to be updated.

Swagger/API contract documentation includes the new administration endpoints and moderation response fields.

Security and privacy

  • No secrets, tokens, credentials, local config, or personal data are included.
  • User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
  • Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

Moderation event details and retained content are restricted to super-admin users. Ordinary administrators receive anonymous aggregate statistics only.

Checklist

  • I searched existing issues and pull requests.
  • Changes are focused and do not include unrelated refactors.
  • Tests or static verification were run where practical.
  • User-facing behavior, deployment steps, API contracts, and configuration changes are documented.
  • Generated artifacts are included only when this project explicitly requires them.
  • Caches, build output, .pyc files, .env files, and local storage data are not committed.

Modality: c.Query("modality"),
Result: c.Query("result"),
Category: c.Query("category"),
UserID: uint(userID),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants