feat(admin): add /admin dashboard with analytics, moderation, health by rafmacalaba · Pull Request #156 · worldbank/data-ai-chatbot

rafmacalaba · 2026-08-02T23:09:31Z

Summary

Adds the /admin dashboard (Analytics · Moderation · Health) with supporting /review shell for sole reviewers and an optional "Continue as guest" path on /login and /register. MSAL/AzureAD flow is untouched.

What ships (50 files, +6263 / −156)

Backend (12 files)

app/api/v1/admin/{analytics,moderation,health}.py — full rewrites
app/api/v1/auth.py — canViewTokenUsage flag (admin supersedes reviewer), canViewAdmin flag
app/api/deps.py — require_feedback_reviewer accepts admin OR reviewer
app/api/v1/utils/background_tasks.py — ChatTokenUsage recording
app/core/rate_limit.py — /api/admin/* exempt
app/main.py — dev-only Base.metadata.create_all gate
app/models/chat.py — deletedAt column
app/models/chat_token_usage.py — new model
alembic/versions/l3m4n5o6p7q8_*.py — deletedAt migration
alembic/versions/m4n5o6p7q8r9_*.py — ChatTokenUsage migration
6 new test files: test_admin_{auth,analytics,moderation,health,rate_limit_exempt}.py, test_role_matrix.py

Frontend (15 files)

app/(admin)/admin/{analytics,moderation,health}/page.tsx — full rewrites
app/(admin)/layout.tsx — sidebar + 403 gate + "Review feedback" link (gated on canViewTokenUsage)
app/(auth)/{login,register}/page.tsx — bounce fix, optional "Continue as guest"
app/review/{layout.tsx, review-shell.tsx} — auth-gated shell for sole reviewers
components/sidebar-user-nav.tsx — Dashboard menu item (gated on canViewAdmin)
lib/admin/{analytics,nav}.ts, lib/auth/registration.ts — new helpers
lib/{auth-service, auth-service-client}.ts — AuthError class, verbose toasts
4 new test files in lib/__tests__/

E2E + Docs

e2e/admin-dashboard.spec.ts — 18 Playwright tests
docs/admin-api.md — full /api/admin/* reference
CHANGELOG.md — Unreleased entry
README.md — Admin Dashboard section
.gitignore — ignores armada/, .opencode/, .agents/, DEFECTS.md, ADVERSARIAL_REVIEW.md, screenshots/, eval artifacts

Migration (run before deploy)

cd backend && alembic upgrade head

Applies: User.disabled column (already in upstream via 9bab245e8167), Chat.deletedAt (new l3m4n5o6p7q8), ChatTokenUsage table (new m4n5o6p7q8r9).

Config (set in prod `.env`)

ADMIN_EMAILS=real-admin@org.com (currently admin@test.com for local)
JWT_SECRET_KEY=<rotate to a real 32-byte secret> (currently a placeholder)
FEEDBACK_REVIEWER_EMAILS=... — keep as-is
Azure OpenAI placeholders: replace or set OPENAI_API_KEY + change MODEL_PROVIDER

Auth model

Flag	True when
`canViewAdmin`	email ∈ `ADMIN_EMAILS`
`canViewTokenUsage`	email ∈ `ADMIN_EMAILS` ∪ `FEEDBACK_REVIEWER_EMAILS` (admin supersedes)

All /api/admin/* routes guarded by Depends(require_admin)
Disabled users blocked at login, guest, refresh
/api/admin/* exempt from global rate limit (admin-gated)

Reviewer integration

The PR widens the existing /review/feedback reviewer-only flow so admins can also use it, and adds a sidebar shell for sole reviewers. The /api/feedback/review* endpoints themselves are unchanged — only the auth gate and the UI shell are new.

Role matrix

User type	`ADMIN_EMAILS`?	`FEEDBACK_REVIEWER_EMAILS`?	`canViewAdmin`	`canViewTokenUsage`	`/admin/*`	`/review/feedback`
Sole reviewer	no	yes	false	true	403 (no admin shell)	accessible via direct URL; has its own sidebar shell
Sole admin (super-role)	yes	no	true	true (admin supersedes)	accessible	accessible via sidebar "Review feedback" link
Both	yes	yes	true	true	accessible	accessible via sidebar
Regular user	no	no	false	false	403	403

Auth gate widening

app/api/deps.py — require_feedback_reviewer now accepts an email in ADMIN_EMAILS ∪ FEEDBACK_REVIEWER_EMAILS. When both lists are empty, the dep still returns 403 ("reviewer access is not configured").
app/api/v1/auth.py — canViewTokenUsage in /api/auth/me is admin ∪ reviewer (admin supersedes).

Cross-link in the admin sidebar

app/(admin)/layout.tsx — sidebar shows a "Review feedback" item gated on canViewTokenUsage === true. Admin users see it because the flag is true for them. Sole reviewers don't see the admin sidebar at all (they get a 403 on /admin).

Shell for sole reviewers

app/review/review-shell.tsx — auth-gated shell wrapping the existing /review/feedback pages. Mirrors the admin layout's look-and-feel. Sidebar shows "Feedback" + "Back to Chat"; conditionally shows "Admin Dashboard" if the user is also an admin. Inline 403 if canViewTokenUsage !== true.

Out of scope (deferred)

The /api/feedback/review* endpoints — unchanged. Reviewers see the same data as before.
FEEDBACK_REVIEWER_EMAILS env var — unchanged. The dep just also accepts admins now.
MSAL/AzureAD flow — untouched.

MSAL / AzureAD

Unchanged. When NEXT_PUBLIC_AUTH_PROVIDER=msal, the existing proxy.ts middleware handles auth as before; the login page still renders the "Sign in with Microsoft" button. The bounce fix on /login//register only affects guest and user modes.

Test gates

Backend pytest: 90/90 (auth, analytics, moderation, health, rate_limit_exempt, role_matrix)
Frontend unit (tsx --test): 33/33
Playwright e2e: 18/18

Out of scope (deferred to a follow-up)

Disabled-user cache key miss on Azure AD path
Soft-deleted chats stay live for owner
/api/admin/* rate-limit carve-out enables DB hammer
ChatTokenUsage unbounded growth, no FK, no cleanup
Admin can self-disable; no last-admin lockout

Phases 0-3 per REQUIREMENTS-admin-dashboard.md: Phase 0 - Admin Auth Foundation - ADMIN_EMAILS env var, require_admin dep, canViewAdmin flag in /api/auth/me - Disabled-user block on login, guest, refresh - /admin/* layout with auth gate, sidebar (Analytics/Moderation/Health/Review feedback) - Dashboard menu item in user nav for admins Phase 1 - Analytics Dashboard - GET /api/admin/analytics/{users,chats,feedback,tokens} with date range filter - Frontend: stat cards, 5 chart families, date filter, /admin/analytics page - Token usage block (totalTokens, promptTokens, completionTokens, tokensByModel, cost) Phase 2 - Moderation Panel - GET /api/admin/moderation/{users,chats} paginated, with search - POST /disable toggle, DELETE /chats/{id} soft delete (deletedAt) - Frontend: Users tab + Chats tab with search, pagination, actions Phase 3 - System Health Dashboard - GET /api/admin/health (status, db, mcp, uptimeSeconds) with MCP async-with fix - GET /api/admin/health/metrics (8 fields incl. totalTokens24h, tokenRate24h, costEstimate24h) - Frontend: status cards, metrics grid, 30s auto-refresh Cross-cutting - Rate-limit exemption for /api/admin/* - ChatTokenUsage model + 2 alembic migrations (deletedAt + ChatTokenUsage) - Admin canViewTokenUsage flag (admin supersedes reviewer) - require_feedback_reviewer accepts admin OR reviewer emails - /review/* auth-gated shell (ReviewShell) for sole reviewers - /register + /login: bounce fix + optional 'Continue as guest' path (MSAL untouched) - 90 backend tests + 33 frontend unit tests + 18 e2e tests - docs/admin-api.md, CHANGELOG.md, README.md updated - .gitignore: ignore armada/, .opencode/, .agents/, DEFECTS.md, ADVERSARIAL_REVIEW.md, screenshots/, eval artifacts

avsolatorio

Starting with this one, @rafmacalaba . :)

avsolatorio · 2026-08-04T20:27:29Z

+
+    now = datetime.utcnow()
+
+    # Hard-delete votes and streams


We should opt for soft deletion whenever possible. The Vote table, in particular, is very important because it contains feedback.

ack! fixed on the latest commit.

Thanks @avsolatorio — addressed, and took the principle further than just Vote.

Done in this round (3 new commits on feat/admin-dashboard):

Vote: soft-delete via new deletedAt column (migration n5o6p7q8r9s0). On re-vote the existing row reactivates on the same composite PK — no duplicates, no feedback loss.

Stream: same treatment (migration o6p7q8r9s0t1). Stream rows are needed to resume from Redis, so preserving them matters for audit.

Message was already soft-deletable in the base PR.

Caught while verifying:

Admin analytics was counting soft-deleted votes/chats, which would have inflated the upvote ratio and total-chats numbers. Now filters deletedAt IS NULL.

Verified live against the deployed chatbot: soft-delete a chat → vote row persists with deletedAt set, re-vote reuses the same PK, no duplicate, chat hidden from lists. 45 backend tests green.

The only paths that still hard-delete are delete_chat_by_id / delete_chats_by_user_id — those are by design (user-initiated / GDPR flows), kept distinct from admin moderation.

Address review feedback: Vote rows contain user feedback and must survive chat deletion for analytics. Adds deletedAt column to Vote_v2 (migration n5o6p7q8r9s0), switches soft_delete_chat_by_id to update votes instead of hard-deleting, and reactivates the existing row on re-vote via composite PK. Stream table left hard-deleted (reviewer scoped feedback to Vote). Verified live: vote row persists with deletedAt set; re-vote reuses same PK without duplicate.

avsolatorio

Some comments!

avsolatorio · 2026-08-05T16:22:17Z

+    )
+
+    # Hard-delete streams (no soft-delete column on Stream)
+    await session.execute(delete(Stream).where(Stream.chatId == chat_id))


Please also soft-delete this. Thanks!

Done in 7c40e6e. Stream now soft-deletes alongside Message and Vote — same deletedAt pattern, migration o6p7q8r9s0t1. soft_delete_chat_by_id updates Stream instead of deleting it. Test test_delete_chat_soft_deletes_streams proves the row stays in the DB with deletedAt set after chat deletion. Only delete_chat_by_id / delete_chats_by_user_id still hard-delete — those are by-design user-initiated paths.

avsolatorio · 2026-08-05T16:23:24Z

+
+    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    chat_id = Column(UUID(as_uuid=True), nullable=False, name="chatId")
+    message_id = Column(String(64), nullable=False, name="messageId")


Shouldn't this be a UUID type as well?

Fixed in c420f55. ChatTokenUsage.messageId is now UUID(as_uuid=True) to match the FK relationship with Message_v2.id. Migration p7q8r9s0t1u2 converts in place — pre-checked all existing rows (4/4) were valid UUID strings before the ALTER, so the conversion is lossless. background_tasks.py wraps the incoming message_id in UUID() at the boundary.

avsolatorio · 2026-08-05T16:25:27Z

You may also add the "admin" tag here. And remap the endpoint to "/api/admin/feedback".

Fixed in c420f55. /api/feedback → /api/admin/feedback with the "admin" OpenAPI tag, matching the other admin routers. Old path returns 404; new path is auth-gated like the rest. Migration p7q8r9s0t1u2 also bumps ChatTokenUsage.messageId from String(64) to UUID, addressing your other comment on chat_token_usage.py:20 — all existing rows were already valid UUIDs so the conversion is lossless. 89 tests green, live smoke confirms the new path is alive and the old one is gone.

avsolatorio · 2026-08-05T16:27:04Z

    """Lifespan context manager for startup and shutdown events."""
    # Startup
    logger.info("=== FastAPI app startup ===")
+    # Dev fallback: create tables for sqlite when migrations are not applied


Make sure the app environment is set to development before automatically creating this fallback db.

Fixed in 6290bf0. The old gate matched the sqlite filename in DATABASE_URL — a misconfigured prod URL containing 'chat.sqlite' would still trigger create_all against the real Postgres. New code gates on settings.ENVIRONMENT in (development, local, dev, test) first, then keeps the sqlite path sniff as a second guard. 9 tests in test_main.py cover: production skips regardless of URL, development+sqlite passes, development+postgres skips (don't create_all against prod-shaped URLs), staging skips. Live ENVIRONMENT=development confirmed.

The soft-delete change in af93fe1 (Vote, Message, Chat) left admin analytics counting soft-deleted rows. Add deletedAt.is_(None) to all vote/chat/token queries in analytics.py. Tests assert the generated SQL contains the filter.

Symmetric with the Vote/Message soft-delete from af93fe1. Stream rows contain the stream_id used to resume streaming from Redis and should be preserved on chat deletion for audit. Adds deletedAt column to Stream (migration o6p7q8r9s0t1, revises n5o6p7q8r9s0) and switches soft_delete_chat_by_id to update Stream instead of hard-deleting. Hard-delete variants delete_chat_by_id and delete_chats_by_user_id are unchanged (still hard-delete by design).

@avsolatorio

Two review comments from @avsolatorio: 1. main.py:207 — feedback router now lives under /api/admin/feedback with the 'admin' OpenAPI tag, consistent with the other admin routers (analytics/moderation/health). Old /api/feedback path removed (no backwards compat — dev/staging only). 2. chat_token_usage.py:20 — messageId was String(64), now UUID to match the FK relationship to Message_v2.id. Migration p7q8r9s0t1u2 converts in place (verified all 4 existing rows were valid UUIDs before the conversion). background_tasks.py now wraps the incoming message_id in UUID() at the boundary. 89 tests pass, live smoke confirms /api/admin/feedback/review returns 401 (auth-gated, route alive) and /api/feedback/review returns 404 (old path gone).

@avsolatorio

Address @avsolatorio comment on main.py:115. The old gate matched 'chat.sqlite' / 'chatbot.db' in DATABASE_URL — a misconfigured production URL containing those substrings would trigger create_all against the real Postgres. Now the FIRST guard is settings.ENVIRONMENT in (development, local, dev, test); the sqlite path sniff stays as a second guard. 9 new tests in test_main.py cover: production skips regardless of URL, development+sqlite passes, development+postgres skips (don't create_all against prod-shaped URLs), staging skips, all four env values verified. Live ENVIRONMENT=development confirmed via docker compose exec.

rafmacalaba force-pushed the feat/admin-dashboard branch from fb12aa1 to a947f2f Compare August 2, 2026 23:12

rafmacalaba requested a review from avsolatorio August 2, 2026 23:17

avsolatorio reviewed Aug 4, 2026

View reviewed changes

avsolatorio reviewed Aug 5, 2026

View reviewed changes

rafmacalaba added 4 commits August 5, 2026 14:47

Navigation Menu

Uh oh!

feat(admin): add /admin dashboard with analytics, moderation, health - #156

feat(admin): add /admin dashboard with analytics, moderation, health#156
rafmacalaba wants to merge 6 commits into
devfrom
feat/admin-dashboard

rafmacalaba commented Aug 2, 2026 •

edited

Loading

Uh oh!

avsolatorio left a comment

Uh oh!

avsolatorio Aug 4, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

avsolatorio left a comment

Uh oh!

avsolatorio Aug 5, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

avsolatorio Aug 5, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

avsolatorio Aug 5, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

avsolatorio Aug 5, 2026

Uh oh!

rafmacalaba Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rafmacalaba commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What ships (50 files, +6263 / −156)

Migration (run before deploy)

Config (set in prod .env)

Auth model

Reviewer integration

Role matrix

Auth gate widening

Cross-link in the admin sidebar

Shell for sole reviewers

Out of scope (deferred)

MSAL / AzureAD

Test gates

Out of scope (deferred to a follow-up)

Uh oh!

avsolatorio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avsolatorio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rafmacalaba commented Aug 2, 2026 •

edited

Loading

Config (set in prod `.env`)