From 61c2e06cf8f9529ea09370e619af96c9539f789d Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 14:34:11 +0200 Subject: [PATCH 01/21] Add marketing campaign vanity links with funnel attribution (#958) * Add marketing campaign vanity links with funnel attribution Marketing can now create campaigns and per-role links entirely from Django admin, publishing clean portal URLs (/join/builders/ethcc) that need no deployment per campaign. An Amplify rule proxies the reserved /join/ namespace to a new campaigns backend app, which resolves the alias, logs a privacy-scrubbed redirect hit with bot classification, and 302s to the stored destination with server-built UTMs (plus forwarded ad click IDs so paid traffic keeps its GA join). Unknown links fail with 404, expired ones with 410, and destinations are validated against a portal-path allowlist on write and again before every redirect. Attribution is durable and first-touch: the frontend stores a structured campaign touch (30-day localStorage first touch plus a session touch), sends the opaque link tracking ID with wallet login, and the backend resolves it server-side onto the pending signup, never trusting browser UTM text and never letting attribution fail a signup. Email confirmation copies the snapshot into a write-once per-user acquisition record inside the same transaction that creates the user (savepointed so a failure cannot abort registration). GA now receives a sign_up event exactly once per created account, and the visible URL is cleaned of tracking params (including ref) while preserving all other query parameters, fixing the old referral cleanup that erased the whole query string. Campaign performance (redirect hits, wallet connects, registrations, and per-role activations computed from durable records, with community activation gated by an explicit per-task flag) renders on the campaign admin page for a restricted Marketing staff group. ## Claude Implementation Notes - backend/campaigns/models.py: MarketingCampaign (unique tracking_key -> utm_campaign), CampaignLink (unique role+alias, server-generated immutable tracking_id -> utm_id, destination allowlist via validate_destination_path), CampaignRedirectHit (append-only, no PII), UserAcquisitionAttribution (OneToOne user, snapshot columns + SET_NULL link FK) - backend/campaigns/services.py: UA bot classification, resolver lookup, best-effort hit logging, click-ID forwarding (CLICK_ID_FORWARD_PARAMS only), apply_pending_attribution (first-touch, window check, reset on expired-pending reuse), record_user_acquisition (nested atomic savepoint), campaign_report (Portal-DB funnel counts, distinct users) - backend/campaigns/views.py + urls.py: anonymous GET/HEAD resolver at /campaigns/redirect//, 302 + Cache-Control no-store, 404/410 fail-closed, throttle scope campaign_redirect - backend/campaigns/admin.py: campaign admin with link inline + performance table, link admin with hit/signup annotations and locked role/alias/tracking_id, read-only hit and attribution admins - backend/campaigns/migrations/: 0001 models, 0002 Marketing group with campaign-only permissions (runs create_permissions explicitly for fresh DBs) - backend/campaigns/management/commands/purge_campaign_hits.py: retention purge, default CAMPAIGN_HIT_RETENTION_DAYS=90, --dry-run - backend/ethereum_auth/models.py + migration 0005: PendingWalletSignup gains acquisition_campaign_link/acquisition_snapshot/acquisition_captured_at - backend/ethereum_auth/views.py: login applies pending attribution after update_or_create, resetting stale data when an expired pending row is reused; wrapped so it can never fail signup - backend/ethereum_auth/email_verification.py: record_user_acquisition called inside the confirm transaction right after user creation - backend/social_tasks/models.py + admin.py + migration 0007: SocialTask.counts_as_activation explicit allowlist flag for community activation - backend/tally/settings.py: campaigns app, campaign_redirect throttle rate, CAMPAIGN_HIT_RETENTION_DAYS, CAMPAIGN_ATTRIBUTION_WINDOW_DAYS - backend/tally/urls.py + backend/utils/throttling.py: campaigns URLconf include and CampaignRedirectRateThrottle - backend/campaigns/tests/: models/resolver/attribution/reporting/admin-permission suites (61 tests), incl. SIWE login and email-confirm integration and savepoint verification - frontend/src/lib/analytics.js: structured first/session campaign touches, getAcquisitionAttribution, clearAcquisitionAttribution, cleanTrackingParamsFromUrl (preserves non-tracking params), trackSignUp (created:true only, clears touches) - frontend/src/lib/auth.js: login payload carries attribution via dynamic analytics import (avoids module cycle); not cleared on success - frontend/src/App.svelte: ref capture no longer rewrites the URL; consolidated cleanTrackingParamsFromUrl(['ref']) in onMount - frontend/src/components/ProfileCompletionGuard.svelte: trackSignUp fired from the email-confirm response before finishProfileCompletion - frontend/src/tests/analytics.test.js: new campaign-attribution describe block (9 tests) - amplify.yml: /join/<*> reverse-proxy rule before the SPA catch-all - backend/CLAUDE.md + frontend/CLAUDE.md: campaigns app and attribution documentation * Update changelog * Harden campaign links against history loss and stale attribution Campaign links can no longer be deleted or moved between campaigns from the admin (pausing replaces deletion, so redirect-hit history is never cascaded away), and a campaign's tracking key is now immutable at the model level once links exist, keeping historical utm_campaign meaning and snapshot reporting stable. Destination validation additionally rejects percent-encoded syntax so encoded traversal cannot bypass the allowlist, and rolling back the marketing-group migration no longer deletes a group it may not have created. On the frontend, only touches carrying the opaque campaign link ID are stored, so a plain utm_source share can no longer lock the 30-day first-touch slot and silently block a later real campaign click from being attributed. ## Claude Implementation Notes - backend/campaigns/models.py: reject '%' in validate_destination_path (encoded traversal); MarketingCampaign.clean() rejects tracking_key changes once links exist; CampaignRedirectHit docstring notes the deliberate non-BaseModel append-only-log exception - backend/campaigns/admin.py: CampaignLinkInline can_delete=False and save_formset no longer deletes; CampaignLinkAdmin freezes campaign FK after create and limits delete permission to superusers - backend/campaigns/migrations/0002_marketing_group.py: reverse operation is now RunPython.noop (a pre-existing Marketing group must survive rollback) - backend/campaigns/tests/test_models.py: %2e%2e destination cases; tracking_key immutability coverage (locked with links, editable without) - backend/campaigns/tests/test_admin_permissions.py: exact (app_label, codename) allowlist assertion instead of spot checks - frontend/src/lib/analytics.js: buildStructuredAttribution requires utm_id, so utm_id-less landings store nothing and cannot shadow the session touch in getAcquisitionAttribution - frontend/src/tests/analytics.test.js: regression test that a utm_id-less landing does not lock the first-touch slot - frontend/CLAUDE.md: blank line after the new analytics heading (MD022) * Ignore whitespace-only campaign IDs in browser attribution A campaign link ID consisting only of whitespace (for example a mangled utm_id=%20 share) is now treated as absent: it is never stored, never locks the 30-day first-touch slot, and a stored first touch without a resolvable ID falls through to the session touch instead of shadowing it. ## Claude Implementation Notes - frontend/src/lib/analytics.js: resolvableUtmId helper trims and validates utm_id; buildStructuredAttribution stores the trimmed ID or nothing; getAcquisitionAttribution prefers the first touch only when its ID is resolvable, else falls through to the session touch - frontend/src/tests/analytics.test.js: whitespace-only utm_id case added to the first-touch lock regression test --- CHANGELOG.md | 2 + amplify.yml | 7 + backend/CLAUDE.md | 20 ++ backend/campaigns/__init__.py | 0 backend/campaigns/admin.py | 177 ++++++++++ backend/campaigns/apps.py | 6 + backend/campaigns/management/__init__.py | 0 .../campaigns/management/commands/__init__.py | 0 .../commands/purge_campaign_hits.py | 30 ++ backend/campaigns/migrations/0001_initial.py | 108 ++++++ .../migrations/0002_marketing_group.py | 46 +++ backend/campaigns/migrations/__init__.py | 0 backend/campaigns/models.py | 307 +++++++++++++++++ backend/campaigns/services.py | 264 +++++++++++++++ backend/campaigns/tests/__init__.py | 0 .../campaigns/tests/test_admin_permissions.py | 76 +++++ backend/campaigns/tests/test_attribution.py | 317 ++++++++++++++++++ backend/campaigns/tests/test_models.py | 168 ++++++++++ backend/campaigns/tests/test_reporting.py | 126 +++++++ backend/campaigns/tests/test_resolver.py | 150 +++++++++ backend/campaigns/urls.py | 13 + backend/campaigns/views.py | 47 +++ backend/ethereum_auth/email_verification.py | 4 + ...gnup_acquisition_campaign_link_and_more.py | 30 ++ backend/ethereum_auth/models.py | 13 + backend/ethereum_auth/views.py | 14 + backend/social_tasks/admin.py | 3 +- .../0007_socialtask_counts_as_activation.py | 18 + backend/social_tasks/models.py | 6 + backend/tally/settings.py | 8 + backend/tally/urls.py | 5 +- backend/utils/throttling.py | 8 + frontend/CLAUDE.md | 8 + frontend/src/App.svelte | 18 +- .../components/ProfileCompletionGuard.svelte | 7 + frontend/src/lib/analytics.js | 152 +++++++++ frontend/src/lib/auth.js | 12 + frontend/src/tests/analytics.test.js | 158 +++++++++ 38 files changed, 2318 insertions(+), 10 deletions(-) create mode 100644 backend/campaigns/__init__.py create mode 100644 backend/campaigns/admin.py create mode 100644 backend/campaigns/apps.py create mode 100644 backend/campaigns/management/__init__.py create mode 100644 backend/campaigns/management/commands/__init__.py create mode 100644 backend/campaigns/management/commands/purge_campaign_hits.py create mode 100644 backend/campaigns/migrations/0001_initial.py create mode 100644 backend/campaigns/migrations/0002_marketing_group.py create mode 100644 backend/campaigns/migrations/__init__.py create mode 100644 backend/campaigns/models.py create mode 100644 backend/campaigns/services.py create mode 100644 backend/campaigns/tests/__init__.py create mode 100644 backend/campaigns/tests/test_admin_permissions.py create mode 100644 backend/campaigns/tests/test_attribution.py create mode 100644 backend/campaigns/tests/test_models.py create mode 100644 backend/campaigns/tests/test_reporting.py create mode 100644 backend/campaigns/tests/test_resolver.py create mode 100644 backend/campaigns/urls.py create mode 100644 backend/campaigns/views.py create mode 100644 backend/ethereum_auth/migrations/0005_pendingwalletsignup_acquisition_campaign_link_and_more.py create mode 100644 backend/social_tasks/migrations/0007_socialtask_counts_as_activation.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b94873e..4bb8af0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ All notable user-facing changes to this project will be documented in this file. ## Unreleased +- Marketing can create campaign links like portal.genlayer.foundation/join/builders/ethcc from the admin panel without a deployment; each link tracks visits, signups, and role activations per campaign, campaign traffic reaches Google Analytics with clean final URLs, and new accounts are attributed to the campaign that brought them (810bdc32) + - Notification bodies in the navbar dropdown now stop at 120 characters with an ellipsis while keeping their formatting and links (b4815f48) - Builder submissions now pass through AI review before standard steward review, high-point acceptances are escalated as proposals to top-level stewards, and apex stewards have a focused queue for accepted contributions marked interesting. Appeals and more-information resubmissions remain visible to both AI review stages. diff --git a/amplify.yml b/amplify.yml index 7453a254..56c8c5bf 100644 --- a/amplify.yml +++ b/amplify.yml @@ -25,6 +25,13 @@ applications: VITE_APP_NAME: Tally VITE_VALIDATOR_RPC_URL: https://rpc.testnet-chain.genlayer.com customRules: + # Campaign vanity links: reverse-proxy the reserved /join/ namespace to + # the Django resolver. Must stay BEFORE the SPA catch-all (rules apply + # in order). One dynamic rule for all campaigns; never add per-campaign + # rules here. + - source: '/join/<*>' + target: 'https://tally-backend.33qpgck0g28d0.us-east-1.cs.amazonlightsail.com/campaigns/redirect/<*>' + status: '200' - source: '' target: '/index.html' status: '200' diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 30b070c3..3c16f8f0 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -34,6 +34,7 @@ backend/ ├── gen_tv/ # Gen TV livestream index ├── notifications/ # Portal notification system ├── service_accounts/ # Machine identities + scoped bearer tokens (AI review agent) +├── campaigns/ # Marketing vanity links + campaign acquisition attribution ├── utils/ # Shared utilities └── tally/ # Django project settings (settings.py, urls.py) ``` @@ -120,6 +121,20 @@ backend/ - **Issue a token**: Django admin Service account change page -> "Issue token", or `python manage.py issue_service_account_token --scopes ... [--expires-days N]`; plaintext is shown exactly once. Rotate = issue new + revoke old (admin action on Service account tokens); kill switch = deactivate the account. - **Tests**: `service_accounts/tests/`; test helper `service_accounts.testing.service_account_auth_headers()` returns client auth kwargs. +### Campaigns (Marketing Vanity Links + Attribution) + +- **App**: `campaigns/`. Marketing creates campaigns and role links in Django admin; no deploy per campaign. Public URL contract: `{FRONTEND_URL}/join//`, reverse-proxied by an Amplify rule (`amplify.yml`, before the SPA catch-all) to `GET /campaigns/redirect//`. +- **Models** (`campaigns/models.py`): + - `MarketingCampaign` - name, unique `tracking_key` (published as utm_campaign, readonly after create), date window, `is_active`, `created_by`. + - `CampaignLink` - FK campaign, server-generated immutable `tracking_id` (published as utm_id), role, alias (UNIQUE role+alias, locked after create), `destination_path` (validated relative portal path: allowlist + reserved-prefix rejection in `validate_destination_path`, re-run by the resolver so corrupt data fails closed), required utm_source/utm_medium, optional content/term, optional window overrides. `redirect_target` builds the UTM query from stored fields only. + - `CampaignRedirectHit` - append-only resolver request log with UA bot classification (`services.classify_user_agent`, substring list). Privacy: never store IPs, full referrers, full UAs, wallets, emails. Retention via `python manage.py purge_campaign_hits [--days N] [--dry-run]` (default `CAMPAIGN_HIT_RETENTION_DAYS=90`). + - `UserAcquisitionAttribution` - write-once first-touch signup attribution (OneToOne user). Keeps immutable snapshot columns (campaign_key/source/medium/...) alongside the SET_NULL link FK so history survives campaign edits/deletes. +- **Resolver** (`campaigns/views.py:campaign_redirect`): anonymous GET/HEAD, throttle scope `campaign_redirect` (120/min), 302 + `Cache-Control: no-store` (never 301), 404 unknown/inactive/future, 410 expired, hit-log failure never blocks the redirect. Forwards ONLY allowlisted ad click IDs (`CLICK_ID_FORWARD_PARAMS`: gclid/gbraid/wbraid/fbclid/twclid/msclkid/ttclid, length-capped) from the request onto the destination; never a request-supplied destination. +- **Attribution flow** (mirrors the referral_code pattern): frontend sends `attribution: {utm_id, landing_path, captured_at}` in the `/api/auth/login/` body → `services.apply_pending_attribution` resolves the link server-side (never trusts browser UTM text; unknown/expired IDs silently ignored; first touch never overwritten; expired pending reuse resets stale data; window `CAMPAIGN_ATTRIBUTION_WINDOW_DAYS=30`) and writes `PendingWalletSignup.acquisition_*` → email confirm calls `services.record_user_acquisition` inside the signup transaction (internally savepointed: an attribution failure can never abort user creation). Attribution must never make signup fail. +- **Activation reporting**: `services.campaign_report(campaign)` returns Portal-DB-only funnel counts (redirect hits human/bot, attributed wallet connects, signups, distinct-user activations: builder = first builder-category SubmittedContribution any state, validator = `validator-waitlist` Contribution, community = completion of a `SocialTask` with `counts_as_activation=True`). Rendered on the campaign admin change page; a future internal-dashboard staff API should wrap this same service. +- **Admin**: `Marketing` group (data migration `campaigns/0002`) gets add/change/view on campaigns/links + view on hits/attributions; members need `is_staff` set manually. Hits and attributions are read-only in admin. +- **Tests**: `campaigns/tests/` (models, resolver, attribution incl. SIWE login + email-confirm integration, reporting, admin permissions). + ### Node Upgrade (Sub-app) - **Models**: `contributions/node_upgrade/models.py` - TargetNodeVersion - Active target version for node upgrades. Per-network, single @@ -510,6 +525,9 @@ GET /api/v1/notifications/ (requires auth, ?unread=true ?categor GET /api/v1/notifications/unread-count/ (requires auth) POST /api/v1/notifications/{id}/mark-read/ (requires auth) POST /api/v1/notifications/mark-all-read/ (requires auth) + +# Campaign vanity links (public; proxied from portal /join// by Amplify) +GET /campaigns/redirect/{role}/{alias} (anonymous, 302 with UTMs, throttled 120/min) ``` ### Leaderboard monthly date ranges @@ -556,6 +574,8 @@ Located in `.env` file: - `GRAFANA_ASIMOV_LABEL` / `GRAFANA_BRADBURY_LABEL` - Override the `network` label values Grafana queries use per testnet (defaults: `asimov-phase5`, `bradbury-phase1`) - `NODE_VERSION_SHAME_GRACE_DAYS` - Grace period (days) after a target's `target_date` before a node still behind it is version-shamed, applied globally (default `3`) - `NODE_VERSION_MIN_OPERATORS_FOR_AUTO_TARGET` - Minimum distinct operators that must be observed running a new stable node version before the Grafana sync auto-creates it as the fleet-wide upgrade target (default `1`: the first adopter creates the target; raise it to require corroboration if version spoofing ever becomes a concern) +- `CAMPAIGN_HIT_RETENTION_DAYS` - Retention window for detailed campaign redirect hits, used as the default of `purge_campaign_hits` (default `90`) +- `CAMPAIGN_ATTRIBUTION_WINDOW_DAYS` - Maximum age of a browser-captured campaign first touch accepted for signup attribution (default `30`) - `SORSA_API_BASE_URL` - Sorsa API base URL (default `https://api.sorsa.io/v3`); used for Twitter follow verification in social_tasks and X follower counts in overview metrics. - `SORSA_API_KEY` - Sorsa API key sent in the `ApiKey` header (secret, required). Store in AWS SSM (`/tally/{env}/sorsa_api_key`) for production. - Note: the Sorsa request timeout and follow endpoint path are intentionally code constants in `social_tasks/sorsa_client.py`, not env vars. Changing the endpoint requires a code deploy anyway because the response parser lives in the same file. diff --git a/backend/campaigns/__init__.py b/backend/campaigns/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/campaigns/admin.py b/backend/campaigns/admin.py new file mode 100644 index 00000000..fc1e0d9b --- /dev/null +++ b/backend/campaigns/admin.py @@ -0,0 +1,177 @@ +from django.contrib import admin +from django.db.models import Count, Q +from django.utils.html import format_html, format_html_join + +from .models import CampaignLink, CampaignRedirectHit, MarketingCampaign, UserAcquisitionAttribution +from .services import campaign_report + + +class CampaignLinkInline(admin.TabularInline): + model = CampaignLink + extra = 1 + # Deleting a link cascades away its redirect-hit history; pause with + # is_active instead. + can_delete = False + fields = ( + 'role', 'alias', 'destination_path', + 'utm_source', 'utm_medium', 'utm_content', 'utm_term', + 'is_active', 'link_url', + ) + readonly_fields = ('link_url',) + + @admin.display(description='Public URL') + def link_url(self, obj): + return obj.public_url if obj.pk else '' + + +@admin.register(MarketingCampaign) +class MarketingCampaignAdmin(admin.ModelAdmin): + list_display = ('name', 'tracking_key', 'is_active', 'starts_at', 'ends_at', 'link_count') + list_filter = ('is_active',) + search_fields = ('name', 'tracking_key') + inlines = [CampaignLinkInline] + + def get_queryset(self, request): + return super().get_queryset(request).annotate(link_count=Count('links', distinct=True)) + + @admin.display(ordering='link_count', description='Links') + def link_count(self, obj): + return obj.link_count + + def get_readonly_fields(self, request, obj=None): + # Published tracking keys are immutable: clone the campaign instead of + # rewriting history. + return ('tracking_key', 'performance') if obj else ('performance',) + + @admin.display(description='Performance (Portal DB)') + def performance(self, obj): + if not obj or not obj.pk: + return 'Available after the campaign is saved.' + report = campaign_report(obj) + activations = report['activations'] + rows = [ + ('Redirect hits (human)', report['redirect_hits_human']), + ('Redirect hits (probable bots)', report['redirect_hits_bot']), + ('Wallet connects (attributed pending signups)', report['wallet_connects']), + ('Registered users', report['signups']), + ('Activated builders (first builder submission)', activations['builder']), + ('Activated validators (joined waitlist)', activations['validator']), + ('Activated community (flagged task completed)', activations['community']), + ] + body = format_html_join( + '', + '{}{}', + rows, + ) + return format_html( + '{}
' + '

Source: Portal DB. Hits are redirect requests, not unique ' + 'visitors; use GA for session-level traffic.

', + body, + ) + + def save_model(self, request, obj, form, change): + if not change and not obj.created_by: + obj.created_by = request.user + super().save_model(request, obj, form, change) + + def save_formset(self, request, form, formset, change): + instances = formset.save(commit=False) + for instance in instances: + if isinstance(instance, CampaignLink) and not instance.created_by_id: + instance.created_by = request.user + instance.save() + formset.save_m2m() + + +@admin.register(CampaignLink) +class CampaignLinkAdmin(admin.ModelAdmin): + list_display = ( + 'alias', 'role', 'campaign', 'utm_source', 'utm_medium', + 'is_active', 'human_hits', 'bot_hits', 'signups', 'public_url', + ) + list_filter = ('role', 'is_active', 'campaign') + search_fields = ('alias', 'campaign__name', 'campaign__tracking_key', 'utm_source', 'utm_medium') + + def get_queryset(self, request): + return super().get_queryset(request).annotate( + human_hit_count=Count('hits', filter=Q(hits__is_probable_bot=False), distinct=True), + bot_hit_count=Count('hits', filter=Q(hits__is_probable_bot=True), distinct=True), + signup_count=Count('acquisitions', distinct=True), + ) + + @admin.display(ordering='human_hit_count', description='Hits (human)') + def human_hits(self, obj): + return obj.human_hit_count + + @admin.display(ordering='bot_hit_count', description='Hits (bots)') + def bot_hits(self, obj): + return obj.bot_hit_count + + @admin.display(ordering='signup_count', description='Signups') + def signups(self, obj): + return obj.signup_count + + def get_readonly_fields(self, request, obj=None): + base = ('tracking_id', 'redirect_preview') + # Campaign, role, and alias define the published link's identity; once + # it is live they must not silently change meaning (moving a link + # between campaigns would also move its hit history). Create a new + # link instead. + return base + ('campaign', 'role', 'alias') if obj else base + + def has_delete_permission(self, request, obj=None): + # Deleting a link cascades away its redirect-hit history; pause with + # is_active instead. Superuser escape hatch only. + return request.user.is_superuser + + @admin.display(description='Redirect target (preview)') + def redirect_preview(self, obj): + if not obj or not obj.pk: + return 'Available after the link is saved.' + return format_html( + 'Public URL: {0}
Redirects to: {1}', obj.public_url, obj.redirect_target, + ) + + def save_model(self, request, obj, form, change): + if not change and not obj.created_by: + obj.created_by = request.user + super().save_model(request, obj, form, change) + + +@admin.register(CampaignRedirectHit) +class CampaignRedirectHitAdmin(admin.ModelAdmin): + list_display = ( + 'campaign_link', 'occurred_at', 'referrer_host', + 'user_agent_family', 'device_category', 'is_probable_bot', + ) + list_filter = ('is_probable_bot', 'device_category') + date_hierarchy = 'occurred_at' + + def has_add_permission(self, request): + return False + + def has_change_permission(self, request, obj=None): + return False + + def has_delete_permission(self, request, obj=None): + # Escape hatch for manual cleanup; routine retention goes through the + # purge_campaign_hits management command. + return request.user.is_superuser + + +@admin.register(UserAcquisitionAttribution) +class UserAcquisitionAttributionAdmin(admin.ModelAdmin): + list_display = ('user', 'campaign_key', 'source', 'medium', 'link_role', 'registered_at') + list_filter = ('link_role', 'campaign_key') + search_fields = ('user__email', 'user__name', 'campaign_key', 'link_tracking_id') + + def has_add_permission(self, request): + return False + + def has_change_permission(self, request, obj=None): + # Acquisition facts are immutable. + return False + + def has_delete_permission(self, request, obj=None): + return request.user.is_superuser diff --git a/backend/campaigns/apps.py b/backend/campaigns/apps.py new file mode 100644 index 00000000..0cab42cc --- /dev/null +++ b/backend/campaigns/apps.py @@ -0,0 +1,6 @@ +from django.apps import AppConfig + + +class CampaignsConfig(AppConfig): + default_auto_field = 'django.db.models.BigAutoField' + name = 'campaigns' diff --git a/backend/campaigns/management/__init__.py b/backend/campaigns/management/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/campaigns/management/commands/__init__.py b/backend/campaigns/management/commands/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/campaigns/management/commands/purge_campaign_hits.py b/backend/campaigns/management/commands/purge_campaign_hits.py new file mode 100644 index 00000000..43659beb --- /dev/null +++ b/backend/campaigns/management/commands/purge_campaign_hits.py @@ -0,0 +1,30 @@ +from datetime import timedelta + +from django.conf import settings +from django.core.management.base import BaseCommand +from django.utils import timezone + +from campaigns.models import CampaignRedirectHit + + +class Command(BaseCommand): + help = 'Delete campaign redirect hits older than the retention window (default 90 days).' + + def add_arguments(self, parser): + parser.add_argument( + '--days', + type=int, + default=settings.CAMPAIGN_HIT_RETENTION_DAYS, + help='Retention window in days.', + ) + parser.add_argument('--dry-run', action='store_true', help='Only report what would be deleted.') + + def handle(self, *args, **options): + cutoff = timezone.now() - timedelta(days=options['days']) + queryset = CampaignRedirectHit.objects.filter(occurred_at__lt=cutoff) + count = queryset.count() + if options['dry_run']: + self.stdout.write(f'Would delete {count} redirect hits older than {cutoff.isoformat()}.') + return + queryset.delete() + self.stdout.write(self.style.SUCCESS(f'Deleted {count} redirect hits older than {cutoff.isoformat()}.')) diff --git a/backend/campaigns/migrations/0001_initial.py b/backend/campaigns/migrations/0001_initial.py new file mode 100644 index 00000000..dbdafd6f --- /dev/null +++ b/backend/campaigns/migrations/0001_initial.py @@ -0,0 +1,108 @@ +# Generated by Django 6.0.6 on 2026-07-29 11:09 + +import campaigns.models +import django.core.validators +import django.db.models.deletion +import django.utils.timezone +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name='CampaignLink', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('tracking_id', models.CharField(default=campaigns.models.generate_tracking_id, editable=False, help_text='Server-generated opaque ID published as utm_id. Immutable.', max_length=20, unique=True)), + ('role', models.CharField(choices=[('builder', 'Builder'), ('validator', 'Validator'), ('community', 'Community')], max_length=16)), + ('alias', models.CharField(help_text='URL segment after the role, e.g. "ethcc" for /join/builders/ethcc.', max_length=64, validators=[django.core.validators.RegexValidator('^[a-z0-9-]+$', 'Use only lowercase letters, digits, and hyphens.')])), + ('destination_path', models.CharField(help_text='Relative portal path the link redirects to, e.g. /builders.', max_length=200)), + ('utm_source', models.CharField(help_text='e.g. x, discord, newsletter, ethcc', max_length=64)), + ('utm_medium', models.CharField(help_text='e.g. organic_social, paid_social, email, event', max_length=64)), + ('utm_content', models.CharField(blank=True, help_text='Optional creative ID, e.g. launch_post_01', max_length=64)), + ('utm_term', models.CharField(blank=True, max_length=64)), + ('is_active', models.BooleanField(default=True, help_text='Prefer pausing over deleting.')), + ('starts_at', models.DateTimeField(blank=True, help_text='Optional override of the campaign window.', null=True)), + ('ends_at', models.DateTimeField(blank=True, help_text='Optional override of the campaign window.', null=True)), + ('created_by', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'ordering': ['-created_at'], + }, + ), + migrations.CreateModel( + name='CampaignRedirectHit', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('occurred_at', models.DateTimeField(db_index=True, default=django.utils.timezone.now)), + ('referrer_host', models.CharField(blank=True, max_length=100)), + ('user_agent_family', models.CharField(blank=True, max_length=32)), + ('device_category', models.CharField(choices=[('desktop', 'Desktop'), ('mobile', 'Mobile'), ('tablet', 'Tablet'), ('bot', 'Bot'), ('unknown', 'Unknown')], default='unknown', max_length=10)), + ('is_probable_bot', models.BooleanField(default=False)), + ('campaign_link', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='hits', to='campaigns.campaignlink')), + ], + options={ + 'ordering': ['-occurred_at'], + }, + ), + migrations.CreateModel( + name='MarketingCampaign', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('name', models.CharField(max_length=200)), + ('tracking_key', models.CharField(help_text='Published as utm_campaign, e.g. ethcc_role_recruitment. Immutable once links are live; clone the campaign instead of changing its meaning.', max_length=64, unique=True, validators=[django.core.validators.RegexValidator('^[a-z0-9_]+$', 'Use only lowercase letters, digits, and underscores.')])), + ('description', models.TextField(blank=True)), + ('starts_at', models.DateTimeField(blank=True, null=True)), + ('ends_at', models.DateTimeField(blank=True, null=True)), + ('is_active', models.BooleanField(default=True, help_text='Prefer deactivating over deleting.')), + ('created_by', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'ordering': ['-created_at'], + }, + ), + migrations.AddField( + model_name='campaignlink', + name='campaign', + field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='links', to='campaigns.marketingcampaign'), + ), + migrations.CreateModel( + name='UserAcquisitionAttribution', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('link_tracking_id', models.CharField(max_length=20)), + ('campaign_key', models.CharField(max_length=64)), + ('source', models.CharField(blank=True, max_length=64)), + ('medium', models.CharField(blank=True, max_length=64)), + ('content', models.CharField(blank=True, max_length=64)), + ('term', models.CharField(blank=True, max_length=64)), + ('link_role', models.CharField(blank=True, max_length=16)), + ('landing_path', models.CharField(blank=True, max_length=200)), + ('captured_at', models.DateTimeField(blank=True, null=True)), + ('registered_at', models.DateTimeField()), + ('campaign_link', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='acquisitions', to='campaigns.campaignlink')), + ('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='acquisition_attribution', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'ordering': ['-registered_at'], + }, + ), + migrations.AddConstraint( + model_name='campaignlink', + constraint=models.UniqueConstraint(fields=('role', 'alias'), name='unique_campaign_link_role_alias'), + ), + ] diff --git a/backend/campaigns/migrations/0002_marketing_group.py b/backend/campaigns/migrations/0002_marketing_group.py new file mode 100644 index 00000000..80ccc05a --- /dev/null +++ b/backend/campaigns/migrations/0002_marketing_group.py @@ -0,0 +1,46 @@ +from django.apps import apps as global_apps +from django.contrib.auth.management import create_permissions +from django.db import migrations + +MARKETING_GROUP_NAME = 'Marketing' + +# (model, actions) the marketing staff group needs. Members additionally need +# is_staff=True, assigned manually per user. +MARKETING_PERMISSIONS = [ + ('marketingcampaign', ('add', 'change', 'view')), + ('campaignlink', ('add', 'change', 'view')), + ('campaignredirecthit', ('view',)), + ('useracquisitionattribution', ('view',)), +] + + +def create_marketing_group(apps, schema_editor): + # Permissions are normally created by post_migrate, which has not run yet + # for this app on a fresh database; create them explicitly first. + app_config = global_apps.get_app_config('campaigns') + create_permissions(app_config, apps=apps, verbosity=0) + + Group = apps.get_model('auth', 'Group') + Permission = apps.get_model('auth', 'Permission') + group, _ = Group.objects.get_or_create(name=MARKETING_GROUP_NAME) + for model, actions in MARKETING_PERMISSIONS: + for action in actions: + permission = Permission.objects.filter( + content_type__app_label='campaigns', + codename=f'{action}_{model}', + ).first() + if permission: + group.permissions.add(permission) + + +class Migration(migrations.Migration): + + dependencies = [ + ('campaigns', '0001_initial'), + ] + + operations = [ + # Reverse is a no-op: the forward path may have reused a pre-existing + # Marketing group, so rollback must not delete it (or its members). + migrations.RunPython(create_marketing_group, migrations.RunPython.noop), + ] diff --git a/backend/campaigns/migrations/__init__.py b/backend/campaigns/migrations/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/campaigns/models.py b/backend/campaigns/models.py new file mode 100644 index 00000000..52dbd356 --- /dev/null +++ b/backend/campaigns/models.py @@ -0,0 +1,307 @@ +"""Marketing campaign vanity links and durable acquisition attribution. + +Marketing creates campaigns and role links in Django admin; the public URL is +always {FRONTEND_URL}/join//, reverse-proxied by Amplify +to the resolver in views.py. Creating a campaign is data only: no route, +Amplify, or DNS change is ever needed per campaign. +""" +import secrets +from urllib.parse import urlencode + +from django.conf import settings +from django.core.exceptions import ValidationError +from django.core.validators import RegexValidator +from django.db import models +from django.utils import timezone + +from utils.models import BaseModel + +ROLE_BUILDER = 'builder' +ROLE_VALIDATOR = 'validator' +ROLE_COMMUNITY = 'community' +ROLE_CHOICES = [ + (ROLE_BUILDER, 'Builder'), + (ROLE_VALIDATOR, 'Validator'), + (ROLE_COMMUNITY, 'Community'), +] + +# Public URL segment <-> canonical role (Category slug) mapping. +ROLE_SEGMENT_TO_ROLE = { + 'builders': ROLE_BUILDER, + 'validators': ROLE_VALIDATOR, + 'community': ROLE_COMMUNITY, +} +ROLE_TO_SEGMENT = {role: segment for segment, role in ROLE_SEGMENT_TO_ROLE.items()} + +# Ad click IDs forwarded from the vanity request onto the redirect target so +# auto-tagged paid traffic keeps its ad-platform join in GA. Keep in sync with +# ATTRIBUTION_PARAMS in frontend/src/lib/analytics.js. +CLICK_ID_FORWARD_PARAMS = ('gclid', 'gbraid', 'wbraid', 'fbclid', 'twclid', 'msclkid', 'ttclid') + +MAX_DESTINATION_LENGTH = 200 + +# Destinations must sit under one of these portal prefixes (exact match or +# prefix + '/'). Extend when marketing needs a new landing surface. +ALLOWED_DESTINATION_PREFIXES = ( + '/', + '/builders', + '/validators', + '/community', + '/how-it-works', + '/referral-program', + '/hackathon', + '/gen-tv', + '/gen-news', + '/ecosystem-partners', +) +RESERVED_DESTINATION_PREFIXES = ( + '/admin', + '/api', + '/oauth', + '/static', + '/media', + '/join', + '/swagger', + '/campaigns', +) + + +def generate_tracking_id(): + """Opaque, non-sensitive link ID published as utm_id.""" + return 'cl-' + secrets.token_hex(6) + + +def _matches_prefix(path, prefix): + return path == prefix or path.startswith(prefix.rstrip('/') + '/') + + +def validate_destination_path(path): + """Validate a campaign destination as a safe relative portal path. + + Runs on write (model clean) AND again in the resolver before every + redirect, so corrupt stored data fails closed instead of redirecting. + """ + if not isinstance(path, str) or not path: + raise ValidationError('Destination is required.') + if len(path) > MAX_DESTINATION_LENGTH: + raise ValidationError('Destination is too long.') + if not path.startswith('/') or path.startswith('//'): + raise ValidationError('Destination must be a relative portal path starting with "/".') + # '%' rejected so percent-encoded traversal (%2e%2e) cannot bypass the + # allowlist after URL normalization; portal paths never need encoding. + if any(ch in path for ch in ('#', '?', '@', '\\', ' ', '%')) or '..' in path or ':' in path: + raise ValidationError('Destination must not include a scheme, host, query, fragment, or encoded/traversal syntax.') + if any(_matches_prefix(path, prefix) for prefix in RESERVED_DESTINATION_PREFIXES): + raise ValidationError('Destination points at a reserved path.') + # '/' allows only the portal root, never every path. + allowed = path == '/' or any( + _matches_prefix(path, prefix) for prefix in ALLOWED_DESTINATION_PREFIXES if prefix != '/' + ) + if not allowed: + raise ValidationError('Destination is not an allowed portal path.') + + +class MarketingCampaign(BaseModel): + name = models.CharField(max_length=200) + tracking_key = models.CharField( + max_length=64, + unique=True, + validators=[RegexValidator(r'^[a-z0-9_]+$', 'Use only lowercase letters, digits, and underscores.')], + help_text=( + 'Published as utm_campaign, e.g. ethcc_role_recruitment. Immutable once ' + 'links are live; clone the campaign instead of changing its meaning.' + ), + ) + description = models.TextField(blank=True) + starts_at = models.DateTimeField(null=True, blank=True) + ends_at = models.DateTimeField(null=True, blank=True) + is_active = models.BooleanField(default=True, help_text='Prefer deactivating over deleting.') + created_by = models.ForeignKey( + settings.AUTH_USER_MODEL, null=True, blank=True, on_delete=models.SET_NULL, related_name='+', + ) + + class Meta: + ordering = ['-created_at'] + + def __str__(self): + return self.name + + def clean(self): + if self.tracking_key: + self.tracking_key = self.tracking_key.strip().lower() + if self.pk: + persisted_key = ( + MarketingCampaign.objects.filter(pk=self.pk) + .values_list('tracking_key', flat=True) + .first() + ) + if persisted_key and persisted_key != self.tracking_key and self.links.exists(): + raise ValidationError({ + 'tracking_key': ( + 'Published tracking keys are immutable once links exist; ' + 'clone the campaign instead.' + ), + }) + if self.starts_at and self.ends_at and self.ends_at <= self.starts_at: + raise ValidationError({'ends_at': 'End must be after start.'}) + + def is_expired_at(self, dt): + return bool(self.ends_at and dt >= self.ends_at) + + def is_live_at(self, dt): + if not self.is_active or self.is_expired_at(dt): + return False + return not (self.starts_at and dt < self.starts_at) + + +class CampaignLink(BaseModel): + campaign = models.ForeignKey(MarketingCampaign, on_delete=models.CASCADE, related_name='links') + tracking_id = models.CharField( + max_length=20, + unique=True, + editable=False, + default=generate_tracking_id, + help_text='Server-generated opaque ID published as utm_id. Immutable.', + ) + role = models.CharField(max_length=16, choices=ROLE_CHOICES) + alias = models.CharField( + max_length=64, + validators=[RegexValidator(r'^[a-z0-9-]+$', 'Use only lowercase letters, digits, and hyphens.')], + help_text='URL segment after the role, e.g. "ethcc" for /join/builders/ethcc.', + ) + destination_path = models.CharField( + max_length=MAX_DESTINATION_LENGTH, + help_text='Relative portal path the link redirects to, e.g. /builders.', + ) + utm_source = models.CharField(max_length=64, help_text='e.g. x, discord, newsletter, ethcc') + utm_medium = models.CharField(max_length=64, help_text='e.g. organic_social, paid_social, email, event') + utm_content = models.CharField(max_length=64, blank=True, help_text='Optional creative ID, e.g. launch_post_01') + utm_term = models.CharField(max_length=64, blank=True) + is_active = models.BooleanField(default=True, help_text='Prefer pausing over deleting.') + starts_at = models.DateTimeField(null=True, blank=True, help_text='Optional override of the campaign window.') + ends_at = models.DateTimeField(null=True, blank=True, help_text='Optional override of the campaign window.') + created_by = models.ForeignKey( + settings.AUTH_USER_MODEL, null=True, blank=True, on_delete=models.SET_NULL, related_name='+', + ) + + class Meta: + ordering = ['-created_at'] + constraints = [ + models.UniqueConstraint(fields=['role', 'alias'], name='unique_campaign_link_role_alias'), + ] + + def __str__(self): + return f'/join/{ROLE_TO_SEGMENT.get(self.role, self.role)}/{self.alias}' + + def clean(self): + for field in ('alias', 'utm_source', 'utm_medium', 'utm_content', 'utm_term'): + value = getattr(self, field) + if value: + setattr(self, field, value.strip().lower()) + if self.starts_at and self.ends_at and self.ends_at <= self.starts_at: + raise ValidationError({'ends_at': 'End must be after start.'}) + validate_destination_path(self.destination_path) + + @property + def public_url(self): + return f'{settings.FRONTEND_URL}/join/{ROLE_TO_SEGMENT.get(self.role, self.role)}/{self.alias}' + + @property + def utm_query(self): + params = { + 'utm_id': self.tracking_id, + 'utm_source': self.utm_source, + 'utm_medium': self.utm_medium, + 'utm_campaign': self.campaign.tracking_key, + } + if self.utm_content: + params['utm_content'] = self.utm_content + if self.utm_term: + params['utm_term'] = self.utm_term + return urlencode(params) + + @property + def redirect_target(self): + return f'{settings.FRONTEND_URL}{self.destination_path}?{self.utm_query}' + + def is_expired_at(self, dt): + return bool(self.ends_at and dt >= self.ends_at) or self.campaign.is_expired_at(dt) + + def is_live_at(self, dt): + if not self.is_active or not self.campaign.is_live_at(dt): + return False + if self.starts_at and dt < self.starts_at: + return False + return not (self.ends_at and dt >= self.ends_at) + + +class CampaignRedirectHit(models.Model): + """One resolver request. These are redirect requests, not unique humans: + link preview bots and scanners hit vanity URLs too (classified below). + + Deliberately NOT a BaseModel: this is an append-only, purgeable log table + (occurred_at is its only meaningful timestamp), matching the ethereum_auth + log-model precedent. + + Privacy: never add raw IPs, full referrer URLs, full user agents, wallet + addresses, or emails to this table. + """ + + DEVICE_DESKTOP = 'desktop' + DEVICE_MOBILE = 'mobile' + DEVICE_TABLET = 'tablet' + DEVICE_BOT = 'bot' + DEVICE_UNKNOWN = 'unknown' + DEVICE_CHOICES = [ + (DEVICE_DESKTOP, 'Desktop'), + (DEVICE_MOBILE, 'Mobile'), + (DEVICE_TABLET, 'Tablet'), + (DEVICE_BOT, 'Bot'), + (DEVICE_UNKNOWN, 'Unknown'), + ] + + campaign_link = models.ForeignKey(CampaignLink, on_delete=models.CASCADE, related_name='hits') + occurred_at = models.DateTimeField(default=timezone.now, db_index=True) + referrer_host = models.CharField(max_length=100, blank=True) + user_agent_family = models.CharField(max_length=32, blank=True) + device_category = models.CharField(max_length=10, choices=DEVICE_CHOICES, default=DEVICE_UNKNOWN) + is_probable_bot = models.BooleanField(default=False) + + class Meta: + ordering = ['-occurred_at'] + + def __str__(self): + return f'{self.campaign_link_id} @ {self.occurred_at:%Y-%m-%d %H:%M}' + + +class UserAcquisitionAttribution(BaseModel): + """Authoritative first-touch signup attribution, written once when the + user is created from a pending wallet signup. + + The snapshot columns duplicate the FK on purpose: a campaign may later be + renamed, archived, or deleted, and historical acquisition facts must not + silently change. + """ + + user = models.OneToOneField( + settings.AUTH_USER_MODEL, on_delete=models.CASCADE, related_name='acquisition_attribution', + ) + campaign_link = models.ForeignKey( + CampaignLink, null=True, blank=True, on_delete=models.SET_NULL, related_name='acquisitions', + ) + link_tracking_id = models.CharField(max_length=20) + campaign_key = models.CharField(max_length=64) + source = models.CharField(max_length=64, blank=True) + medium = models.CharField(max_length=64, blank=True) + content = models.CharField(max_length=64, blank=True) + term = models.CharField(max_length=64, blank=True) + link_role = models.CharField(max_length=16, blank=True) + landing_path = models.CharField(max_length=MAX_DESTINATION_LENGTH, blank=True) + captured_at = models.DateTimeField(null=True, blank=True) + registered_at = models.DateTimeField() + + class Meta: + ordering = ['-registered_at'] + + def __str__(self): + return f'{self.user_id} <- {self.campaign_key}' diff --git a/backend/campaigns/services.py b/backend/campaigns/services.py new file mode 100644 index 00000000..cfd0437f --- /dev/null +++ b/backend/campaigns/services.py @@ -0,0 +1,264 @@ +import logging +from datetime import timedelta +from urllib.parse import urlencode, urlparse + +from django.conf import settings +from django.db import transaction +from django.db.models import Count, Q +from django.utils import timezone +from django.utils.dateparse import parse_datetime + +from .models import ( + CLICK_ID_FORWARD_PARAMS, + MAX_DESTINATION_LENGTH, + ROLE_SEGMENT_TO_ROLE, + CampaignLink, + CampaignRedirectHit, + UserAcquisitionAttribution, +) + +logger = logging.getLogger(__name__) + +# ponytail: substring UA classification, swap for a UA-parser dependency only +# if the bot split ever proves too coarse. +BOT_UA_SUBSTRINGS = ( + 'bot', + 'crawler', + 'spider', + 'slurp', + 'preview', + 'facebookexternalhit', + 'whatsapp', + 'embedly', + 'curl', + 'wget', + 'python-requests', + 'httpclient', + 'headless', + 'lighthouse', + 'scanner', +) +_BROWSER_FAMILIES = ( + ('edg', 'edge'), + ('opr', 'opera'), + ('firefox', 'firefox'), + ('chrome', 'chrome'), + ('safari', 'safari'), +) +_MAX_FORWARDED_CLICK_ID_LENGTH = 100 + + +def classify_user_agent(user_agent): + """Return (family, device_category, is_probable_bot) from a raw UA string.""" + ua = (user_agent or '').lower() + if not ua: + return ('', CampaignRedirectHit.DEVICE_UNKNOWN, True) + for marker in BOT_UA_SUBSTRINGS: + if marker in ua: + return (marker[:32], CampaignRedirectHit.DEVICE_BOT, True) + if 'ipad' in ua or 'tablet' in ua: + device = CampaignRedirectHit.DEVICE_TABLET + elif 'mobi' in ua or 'android' in ua: + device = CampaignRedirectHit.DEVICE_MOBILE + else: + device = CampaignRedirectHit.DEVICE_DESKTOP + family = 'other' + for marker, name in _BROWSER_FAMILIES: + if marker in ua: + family = name + break + return (family, device, False) + + +def resolve_campaign_link(role_segment, alias): + role = ROLE_SEGMENT_TO_ROLE.get((role_segment or '').lower()) + if not role: + return None + return ( + CampaignLink.objects.select_related('campaign') + .filter(role=role, alias=(alias or '').lower()) + .first() + ) + + +def record_redirect_hit(link, request): + """Best effort: a failed hit insert must never block the redirect.""" + try: + referrer_host = '' + referer = request.META.get('HTTP_REFERER', '') + if referer: + referrer_host = (urlparse(referer).hostname or '')[:100] + family, device, is_bot = classify_user_agent(request.META.get('HTTP_USER_AGENT', '')) + CampaignRedirectHit.objects.create( + campaign_link=link, + referrer_host=referrer_host, + user_agent_family=family, + device_category=device, + is_probable_bot=is_bot, + ) + except Exception: + logger.warning('Campaign redirect hit logging failed for link %s', link.pk, exc_info=True) + + +def build_redirect_url(link, request): + """Stored destination + stored UTMs, plus allowlisted ad click IDs + forwarded from the incoming request (nothing else is ever forwarded).""" + url = link.redirect_target + forwarded = { + key: request.GET[key][:_MAX_FORWARDED_CLICK_ID_LENGTH] + for key in CLICK_ID_FORWARD_PARAMS + if request.GET.get(key) + } + if forwarded: + url = f'{url}&{urlencode(forwarded)}' + return url + + +def _clear_pending_attribution_fields(pending): + pending.acquisition_campaign_link = None + pending.acquisition_snapshot = {} + pending.acquisition_captured_at = None + + +_ATTRIBUTION_UPDATE_FIELDS = [ + 'acquisition_campaign_link', 'acquisition_snapshot', 'acquisition_captured_at', 'updated_at', +] + + +def apply_pending_attribution(pending, payload, reset=False): + """Write first-touch campaign attribution onto a pending wallet signup. + + Defensive by design: unknown or expired IDs and malformed payloads are + ignored silently, never surfaced as errors (attribution must never make + signup fail). The snapshot is built ONLY from the resolved link and its + campaign, never from browser-supplied UTM text. + """ + if reset and pending.acquisition_captured_at: + # The pending row is being reused after expiry; stale acquisition data + # must not leak into the new signup attempt. + _clear_pending_attribution_fields(pending) + pending.save(update_fields=_ATTRIBUTION_UPDATE_FIELDS) + if pending.acquisition_captured_at: + return # first touch wins + if not isinstance(payload, dict): + return + utm_id = payload.get('utm_id') + if not isinstance(utm_id, str) or not utm_id or len(utm_id) > 64: + return + captured_raw = payload.get('captured_at') + captured_dt = parse_datetime(captured_raw) if isinstance(captured_raw, str) else None + if captured_dt is None or timezone.is_naive(captured_dt): + return + now = timezone.now() + window = timedelta(days=settings.CAMPAIGN_ATTRIBUTION_WINDOW_DAYS) + if captured_dt > now + timedelta(minutes=5) or captured_dt < now - window: + return + landing_path = payload.get('landing_path') + if ( + not isinstance(landing_path, str) + or not landing_path.startswith('/') + or len(landing_path) > MAX_DESTINATION_LENGTH + ): + landing_path = '' + else: + landing_path = landing_path.split('?')[0].split('#')[0] + link = CampaignLink.objects.select_related('campaign').filter(tracking_id=utm_id).first() + if link is None or not link.is_live_at(captured_dt): + return + pending.acquisition_campaign_link = link + pending.acquisition_snapshot = { + 'link_tracking_id': link.tracking_id, + 'campaign_key': link.campaign.tracking_key, + 'source': link.utm_source, + 'medium': link.utm_medium, + 'content': link.utm_content, + 'term': link.utm_term, + 'link_role': link.role, + 'landing_path': landing_path, + 'captured_at': captured_dt.isoformat(), + } + pending.acquisition_captured_at = captured_dt + pending.save(update_fields=_ATTRIBUTION_UPDATE_FIELDS) + + +def record_user_acquisition(user, pending_signup): + """Copy pending-signup attribution into the write-once acquisition record. + + Called inside the signup transaction so the record commits atomically with + the new User. The nested atomic() creates a savepoint: a failure here rolls + back only this insert and never aborts user creation (a bare try/except + would poison the outer Postgres transaction). + """ + if pending_signup is None or not pending_signup.acquisition_captured_at: + return + try: + with transaction.atomic(): + if UserAcquisitionAttribution.objects.filter(user=user).exists(): + return + snapshot = pending_signup.acquisition_snapshot or {} + + def _text(key, max_length): + value = snapshot.get(key) + return value[:max_length] if isinstance(value, str) else '' + + UserAcquisitionAttribution.objects.create( + user=user, + campaign_link=pending_signup.acquisition_campaign_link, + link_tracking_id=_text('link_tracking_id', 20), + campaign_key=_text('campaign_key', 64), + source=_text('source', 64), + medium=_text('medium', 64), + content=_text('content', 64), + term=_text('term', 64), + link_role=_text('link_role', 16), + landing_path=_text('landing_path', MAX_DESTINATION_LENGTH), + captured_at=pending_signup.acquisition_captured_at, + registered_at=timezone.now(), + ) + except Exception: + logger.exception('Failed to record acquisition attribution for user %s', user.pk) + + +def campaign_report(campaign): + """Campaign funnel numbers from durable portal records, for the admin + change page (and, later, the internal dashboard staff API). + + All user-level numbers are distinct users reaching their first qualifying + outcome, never event counts. Source: Portal DB only; GA remains the + session/multi-touch layer. + """ + from contributions.models import Contribution, SubmittedContribution + from ethereum_auth.models import PendingWalletSignup + from social_tasks.models import SocialTaskCompletion + + hit_counts = CampaignRedirectHit.objects.filter(campaign_link__campaign=campaign).aggregate( + human=Count('id', filter=Q(is_probable_bot=False)), + bot=Count('id', filter=Q(is_probable_bot=True)), + ) + wallet_connects = PendingWalletSignup.objects.filter( + acquisition_campaign_link__campaign=campaign, + ).count() + # Filter on the snapshot key so acquisitions survive link deletion. + user_ids = list( + UserAcquisitionAttribution.objects.filter( + Q(campaign_link__campaign=campaign) | Q(campaign_key=campaign.tracking_key) + ).values_list('user_id', flat=True).distinct() + ) + return { + 'source': 'portal_db', + 'redirect_hits_human': hit_counts['human'] or 0, + 'redirect_hits_bot': hit_counts['bot'] or 0, + 'wallet_connects': wallet_connects, + 'signups': len(user_ids), + 'activations': { + 'builder': SubmittedContribution.objects.filter( + user_id__in=user_ids, contribution_type__category__slug='builder', + ).values('user_id').distinct().count(), + 'validator': Contribution.objects.filter( + user_id__in=user_ids, contribution_type__slug='validator-waitlist', + ).values('user_id').distinct().count(), + 'community': SocialTaskCompletion.objects.filter( + user_id__in=user_ids, task__counts_as_activation=True, + ).values('user_id').distinct().count(), + }, + } diff --git a/backend/campaigns/tests/__init__.py b/backend/campaigns/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/campaigns/tests/test_admin_permissions.py b/backend/campaigns/tests/test_admin_permissions.py new file mode 100644 index 00000000..e5cbce5d --- /dev/null +++ b/backend/campaigns/tests/test_admin_permissions.py @@ -0,0 +1,76 @@ +from importlib import import_module + +from django.apps import apps +from django.contrib.auth import get_user_model +from django.contrib.auth.models import Group +from django.test import Client, TestCase + +from campaigns.models import MarketingCampaign + +User = get_user_model() + + +def _run_marketing_group_migration(): + # Migrations do not run under tally.test_settings; invoke the data + # migration function directly against the live apps registry. + module = import_module('campaigns.migrations.0002_marketing_group') + module.create_marketing_group(apps, None) + + +class MarketingGroupTests(TestCase): + def setUp(self): + _run_marketing_group_migration() + self.group = Group.objects.get(name='Marketing') + self.marketer = User.objects.create_user( + email='marketing@example.com', password='pass12345', is_staff=True, + ) + self.marketer.groups.add(self.group) + self.client = Client() + self.client.force_login(self.marketer) + + def test_group_has_exactly_the_campaign_permissions(self): + granted = set( + self.group.permissions.values_list('content_type__app_label', 'codename') + ) + expected = { + ('campaigns', 'add_marketingcampaign'), + ('campaigns', 'change_marketingcampaign'), + ('campaigns', 'view_marketingcampaign'), + ('campaigns', 'add_campaignlink'), + ('campaigns', 'change_campaignlink'), + ('campaigns', 'view_campaignlink'), + ('campaigns', 'view_campaignredirecthit'), + ('campaigns', 'view_useracquisitionattribution'), + } + self.assertEqual(granted, expected) + + def test_marketing_user_can_manage_campaigns(self): + response = self.client.get('/admin/campaigns/marketingcampaign/') + self.assertEqual(response.status_code, 200) + response = self.client.post('/admin/campaigns/marketingcampaign/add/', { + 'name': 'Test Campaign', + 'tracking_key': 'test_campaign', + 'description': '', + 'is_active': 'on', + 'links-TOTAL_FORMS': '0', + 'links-INITIAL_FORMS': '0', + }) + self.assertEqual(response.status_code, 302) + self.assertTrue(MarketingCampaign.objects.filter(tracking_key='test_campaign').exists()) + + def test_marketing_user_cannot_access_other_apps(self): + response = self.client.get('/admin/users/user/') + self.assertEqual(response.status_code, 403) + + def test_anonymous_denied(self): + anonymous = Client() + response = anonymous.get('/admin/campaigns/marketingcampaign/') + self.assertEqual(response.status_code, 302) + self.assertIn('/admin/login', response['Location']) + + def test_non_staff_denied(self): + plain = User.objects.create_user(email='plain@example.com', password='pass12345') + client = Client() + client.force_login(plain) + response = client.get('/admin/campaigns/marketingcampaign/') + self.assertEqual(response.status_code, 302) diff --git a/backend/campaigns/tests/test_attribution.py b/backend/campaigns/tests/test_attribution.py new file mode 100644 index 00000000..c8b999e3 --- /dev/null +++ b/backend/campaigns/tests/test_attribution.py @@ -0,0 +1,317 @@ +from datetime import timedelta +from types import SimpleNamespace +from unittest.mock import Mock, patch + +from django.contrib.auth import get_user_model +from django.core.cache import cache +from django.db import transaction +from django.test import TestCase, override_settings +from django.utils import timezone +from eth_account import Account +from eth_account.messages import encode_defunct +from rest_framework.test import APIClient + +from campaigns.models import CampaignLink, UserAcquisitionAttribution +from campaigns.services import apply_pending_attribution, record_user_acquisition +from campaigns.tests.test_models import make_campaign, make_link +from ethereum_auth.models import Nonce, PendingWalletSignup + +User = get_user_model() + + +def make_pending(address='0xabcdefabcdefabcdefabcdefabcdefabcdefabcd', **kwargs): + defaults = {'address': address, 'expires_at': timezone.now() + timedelta(minutes=30)} + defaults.update(kwargs) + return PendingWalletSignup.objects.create(**defaults) + + +def attribution_payload(link, **overrides): + payload = { + 'utm_id': link.tracking_id, + 'landing_path': '/builders', + 'captured_at': timezone.now().isoformat(), + } + payload.update(overrides) + return payload + + +class ApplyPendingAttributionTests(TestCase): + def setUp(self): + self.link = make_link() + self.pending = make_pending() + + def test_valid_utm_id_writes_fk_and_snapshot(self): + apply_pending_attribution(self.pending, attribution_payload(self.link)) + self.pending.refresh_from_db() + self.assertEqual(self.pending.acquisition_campaign_link, self.link) + self.assertIsNotNone(self.pending.acquisition_captured_at) + snapshot = self.pending.acquisition_snapshot + self.assertEqual(snapshot['link_tracking_id'], self.link.tracking_id) + self.assertEqual(snapshot['campaign_key'], 'ethcc_role_recruitment') + self.assertEqual(snapshot['source'], 'x') + self.assertEqual(snapshot['medium'], 'organic_social') + self.assertEqual(snapshot['link_role'], 'builder') + self.assertEqual(snapshot['landing_path'], '/builders') + + def test_snapshot_never_copies_browser_utm_text(self): + payload = attribution_payload(self.link, utm_source='SPOOFED', campaign='SPOOFED') + apply_pending_attribution(self.pending, payload) + self.pending.refresh_from_db() + self.assertEqual(self.pending.acquisition_snapshot['source'], 'x') + self.assertNotIn('SPOOFED', str(self.pending.acquisition_snapshot)) + + def test_unknown_utm_id_is_silently_ignored(self): + apply_pending_attribution(self.pending, attribution_payload(self.link, utm_id='cl-unknown')) + self.pending.refresh_from_db() + self.assertIsNone(self.pending.acquisition_captured_at) + + def test_link_not_live_at_capture_time_ignored(self): + CampaignLink.objects.filter(pk=self.link.pk).update(is_active=False) + apply_pending_attribution(self.pending, attribution_payload(self.link)) + self.pending.refresh_from_db() + self.assertIsNone(self.pending.acquisition_captured_at) + + def test_captured_at_outside_window_ignored(self): + stale = (timezone.now() - timedelta(days=45)).isoformat() + apply_pending_attribution(self.pending, attribution_payload(self.link, captured_at=stale)) + future = (timezone.now() + timedelta(days=2)).isoformat() + apply_pending_attribution(self.pending, attribution_payload(self.link, captured_at=future)) + self.pending.refresh_from_db() + self.assertIsNone(self.pending.acquisition_captured_at) + + def test_first_touch_never_overwritten(self): + apply_pending_attribution(self.pending, attribution_payload(self.link)) + other_link = make_link(make_campaign(tracking_key='other_campaign'), alias='other') + apply_pending_attribution(self.pending, attribution_payload(other_link)) + self.pending.refresh_from_db() + self.assertEqual(self.pending.acquisition_campaign_link, self.link) + + def test_reset_clears_stale_attribution(self): + apply_pending_attribution(self.pending, attribution_payload(self.link)) + apply_pending_attribution(self.pending, None, reset=True) + self.pending.refresh_from_db() + self.assertIsNone(self.pending.acquisition_campaign_link) + self.assertEqual(self.pending.acquisition_snapshot, {}) + self.assertIsNone(self.pending.acquisition_captured_at) + + def test_reset_then_new_attribution_applies(self): + apply_pending_attribution(self.pending, attribution_payload(self.link)) + other_link = make_link(make_campaign(tracking_key='other_campaign'), alias='other') + apply_pending_attribution(self.pending, attribution_payload(other_link), reset=True) + self.pending.refresh_from_db() + self.assertEqual(self.pending.acquisition_campaign_link, other_link) + + def test_malformed_payloads_never_raise(self): + for payload in ( + None, + [], + 'string', + {}, + {'utm_id': 42}, + {'utm_id': 'x' * 200}, + {'utm_id': self.link.tracking_id}, # missing captured_at + {'utm_id': self.link.tracking_id, 'captured_at': 'not-a-date'}, + {'utm_id': self.link.tracking_id, 'captured_at': '2026-01-01T00:00:00'}, # naive + attribution_payload(self.link, landing_path='https://evil.example.com'), + attribution_payload(self.link, landing_path='x' * 500), + ): + apply_pending_attribution(self.pending, payload) + self.pending.refresh_from_db() + # The two payloads with a bad landing_path are otherwise valid: they + # attribute with an empty landing path rather than failing. + self.assertEqual(self.pending.acquisition_snapshot.get('landing_path'), '') + + +class RecordUserAcquisitionTests(TestCase): + def setUp(self): + self.link = make_link() + self.pending = make_pending() + apply_pending_attribution(self.pending, attribution_payload(self.link)) + self.pending.refresh_from_db() + self.user = User.objects.create_user(email='acq@example.com', password='x') + + def test_creates_write_once_record_from_snapshot(self): + record_user_acquisition(self.user, self.pending) + record = UserAcquisitionAttribution.objects.get(user=self.user) + self.assertEqual(record.campaign_link, self.link) + self.assertEqual(record.link_tracking_id, self.link.tracking_id) + self.assertEqual(record.campaign_key, 'ethcc_role_recruitment') + self.assertEqual(record.source, 'x') + self.assertEqual(record.link_role, 'builder') + self.assertIsNotNone(record.registered_at) + + # Second call is a no-op. + record_user_acquisition(self.user, self.pending) + self.assertEqual(UserAcquisitionAttribution.objects.filter(user=self.user).count(), 1) + + def test_no_attribution_no_record(self): + blank_pending = make_pending(address='0x1111111111111111111111111111111111111111') + record_user_acquisition(self.user, blank_pending) + record_user_acquisition(self.user, None) + self.assertFalse(UserAcquisitionAttribution.objects.exists()) + + def test_failure_does_not_poison_outer_transaction(self): + with transaction.atomic(): + with patch( + 'campaigns.services.UserAcquisitionAttribution.objects.create', + side_effect=RuntimeError('boom'), + ): + record_user_acquisition(self.user, self.pending) + # The outer transaction must still be usable after the failure. + marker = User.objects.create_user(email='still-works@example.com', password='x') + self.assertTrue(User.objects.filter(pk=marker.pk).exists()) + self.assertFalse(UserAcquisitionAttribution.objects.exists()) + + +class LoginAttributionIntegrationTests(TestCase): + """The SIWE login view resolves browser attribution onto the pending signup.""" + + def setUp(self): + cache.clear() + self.client = APIClient() + self.link = make_link() + + def _session_key(self): + session = self.client.session + session.save() + return session.session_key + + def _nonce(self, value): + return Nonce.objects.create( + value=value, + session_key=self._session_key(), + purpose=Nonce.PURPOSE_LOGIN, + expires_at=timezone.now() + timedelta(minutes=5), + ) + + def _login_message(self, account, nonce_value): + return ( + 'localhost:5173 wants you to sign in with your Ethereum account:\n' + f'{account.address}\n\n' + 'Sign in with Ethereum to GenLayer Testnet Contributions\n\n' + 'URI: http://localhost:5173\n' + 'Version: 1\n' + 'Chain ID: 1\n' + f'Nonce: {nonce_value}\n' + f'Issued At: {timezone.now().isoformat()}' + ) + + def _login(self, account, nonce_value, attribution=None): + nonce = self._nonce(nonce_value) + message = self._login_message(account, nonce.value) + signature = Account.sign_message( + encode_defunct(text=message), private_key=account.key, + ).signature.hex() + payload = {'message': message, 'signature': signature} + if attribution is not None: + payload['attribution'] = attribution + return self.client.post('/api/auth/login/', payload, format='json') + + def test_login_with_attribution_populates_pending(self): + account = Account.create() + response = self._login(account, 'attribNonce1', attribution_payload(self.link)) + self.assertEqual(response.status_code, 200) + self.assertTrue(response.data['pending_signup']) + pending = PendingWalletSignup.objects.get(address=account.address.lower()) + self.assertEqual(pending.acquisition_campaign_link, self.link) + + def test_repeated_login_keeps_first_touch(self): + account = Account.create() + self._login(account, 'attribNonce2', attribution_payload(self.link)) + other_link = make_link(make_campaign(tracking_key='other_campaign'), alias='other') + self._login(account, 'attribNonce3', attribution_payload(other_link)) + pending = PendingWalletSignup.objects.get(address=account.address.lower()) + self.assertEqual(pending.acquisition_campaign_link, self.link) + + def test_expired_pending_reuse_resets_stale_attribution(self): + account = Account.create() + self._login(account, 'attribNonce4', attribution_payload(self.link)) + PendingWalletSignup.objects.filter(address=account.address.lower()).update( + expires_at=timezone.now() - timedelta(minutes=1), + ) + self._login(account, 'attribNonce5') + pending = PendingWalletSignup.objects.get(address=account.address.lower()) + self.assertIsNone(pending.acquisition_campaign_link) + self.assertIsNone(pending.acquisition_captured_at) + + def test_garbage_attribution_never_blocks_signup(self): + account = Account.create() + response = self._login( + account, 'attribNonce6', {'utm_id': ['not', 'a', 'string'], 'captured_at': 12345}, + ) + self.assertEqual(response.status_code, 200) + self.assertTrue(response.data['pending_signup']) + + def test_existing_user_login_ignores_attribution(self): + account = Account.create() + User.objects.create_user( + email='existing@example.com', password='x', address=account.address.lower(), + ) + response = self._login(account, 'attribNonce7', attribution_payload(self.link)) + self.assertEqual(response.status_code, 200) + self.assertTrue(response.data['authenticated']) + self.assertFalse(response.data['created']) + self.assertFalse(PendingWalletSignup.objects.filter(address=account.address.lower()).exists()) + self.assertFalse(UserAcquisitionAttribution.objects.exists()) + + +@override_settings(TURNSTILE_SECRET_KEY='test-secret', TURNSTILE_ALLOWED_HOSTNAMES=[]) +class EmailConfirmAttributionTests(TestCase): + """Email confirmation copies the pending attribution into the durable + acquisition record in the same transaction that creates the user.""" + + def setUp(self): + cache.clear() + self.client = APIClient() + self.link = make_link() + + def _pending_signup_in_session(self): + pending = make_pending() + apply_pending_attribution(pending, attribution_payload(self.link)) + pending.refresh_from_db() + session = self.client.session + session['pending_wallet_signup_id'] = pending.id + session['pending_wallet_address'] = pending.address + session.save() + return pending + + def _start_and_confirm(self, email='campaign-user@example.com'): + with ( + patch('ethereum_auth.email_verification._generate_verification_code', return_value='123456'), + patch('ethereum_auth.email_verification.validate_email') as mock_validate_email, + patch('ethereum_auth.email_verification.requests.post') as mock_post, + ): + mock_post.return_value = Mock(json=lambda: {'success': True, 'hostname': 'localhost'}) + mock_validate_email.return_value = SimpleNamespace( + normalized=email, domain=email.split('@', 1)[1], + ) + start = self.client.post('/api/auth/signup/email/start/', { + 'email': email, + 'name': 'Campaign User', + 'turnstile_token': 'ok-token', + }, format='json') + self.assertEqual(start.status_code, 200, start.data) + # Confirm re-validates the email, so it must run inside the patches. + return self.client.post('/api/auth/signup/email/confirm/', {'code': '123456'}, format='json') + + def test_confirm_creates_acquisition_record(self): + pending = self._pending_signup_in_session() + response = self._start_and_confirm() + self.assertEqual(response.status_code, 200, response.data) + self.assertTrue(response.data['created']) + user = User.objects.get(address__iexact=pending.address) + record = UserAcquisitionAttribution.objects.get(user=user) + self.assertEqual(record.campaign_link, self.link) + self.assertEqual(record.campaign_key, 'ethcc_role_recruitment') + + def test_confirm_survives_attribution_failure(self): + pending = self._pending_signup_in_session() + with patch( + 'campaigns.services.UserAcquisitionAttribution.objects.create', + side_effect=RuntimeError('boom'), + ): + response = self._start_and_confirm(email='resilient@example.com') + self.assertEqual(response.status_code, 200) + self.assertTrue(response.data['created']) + self.assertTrue(User.objects.filter(address__iexact=pending.address).exists()) + self.assertFalse(UserAcquisitionAttribution.objects.exists()) diff --git a/backend/campaigns/tests/test_models.py b/backend/campaigns/tests/test_models.py new file mode 100644 index 00000000..9249c35c --- /dev/null +++ b/backend/campaigns/tests/test_models.py @@ -0,0 +1,168 @@ +from django.core.exceptions import ValidationError +from django.db import IntegrityError +from django.test import TestCase +from django.utils import timezone + +from campaigns.models import ( + CampaignLink, + MarketingCampaign, + generate_tracking_id, + validate_destination_path, +) + + +def make_campaign(**kwargs): + defaults = {'name': 'ETHCC Role Recruitment', 'tracking_key': 'ethcc_role_recruitment'} + defaults.update(kwargs) + return MarketingCampaign.objects.create(**defaults) + + +def make_link(campaign=None, **kwargs): + campaign = campaign or make_campaign() + defaults = { + 'campaign': campaign, + 'role': 'builder', + 'alias': 'ethcc', + 'destination_path': '/builders', + 'utm_source': 'x', + 'utm_medium': 'organic_social', + } + defaults.update(kwargs) + return CampaignLink.objects.create(**defaults) + + +class DestinationValidationTests(TestCase): + def test_valid_destinations(self): + for path in ('/', '/builders', '/builders/tasks', '/community', '/how-it-works'): + validate_destination_path(path) + + def test_invalid_destinations_rejected(self): + bad = [ + '', + None, + 'https://evil.example.com', + '//evil.example.com', + '/builders/../admin', + '/builders/%2e%2e/admin', + '/builders/%2E%2E/admin', + '/admin', + '/admin/login', + '/api/v1/users', + '/join/builders/x', + '/oauth', + '/static/app.js', + '/campaigns/redirect/builders/x', + '/builders#frag', + '/builders?x=1', + '/builders with space', + '/not-a-real-prefix', + '/' + 'a' * 400, + ] + for path in bad: + with self.assertRaises(ValidationError, msg=f'accepted: {path!r}'): + validate_destination_path(path) + + +class CampaignModelTests(TestCase): + def test_tracking_key_rejects_bad_characters(self): + campaign = MarketingCampaign(name='X', tracking_key='Bad-Key!') + with self.assertRaises(ValidationError): + campaign.full_clean() + + def test_date_window_validation(self): + now = timezone.now() + campaign = MarketingCampaign( + name='X', tracking_key='x_campaign', starts_at=now, ends_at=now - timezone.timedelta(days=1), + ) + with self.assertRaises(ValidationError): + campaign.full_clean() + + def test_tracking_key_immutable_once_links_exist(self): + link = make_link() + campaign = link.campaign + campaign.tracking_key = 'renamed_key' + with self.assertRaises(ValidationError): + campaign.full_clean() + + def test_tracking_key_editable_while_campaign_has_no_links(self): + campaign = make_campaign(tracking_key='draft_key') + campaign.tracking_key = 'renamed_key' + campaign.full_clean() + + +class CampaignLinkModelTests(TestCase): + def test_clean_normalizes_utm_values(self): + link = make_link() + link.utm_source = ' X ' + link.utm_medium = 'Organic_Social' + link.utm_content = ' Launch_Post_01 ' + link.full_clean() + self.assertEqual(link.utm_source, 'x') + self.assertEqual(link.utm_medium, 'organic_social') + self.assertEqual(link.utm_content, 'launch_post_01') + + def test_alias_rejects_script_html_and_uppercase(self): + link = make_link() + for alias in (' diff --git a/frontend/src/routes/VerifyEmail.svelte b/frontend/src/routes/VerifyEmail.svelte index 0230ea69..fac8f027 100644 --- a/frontend/src/routes/VerifyEmail.svelte +++ b/frontend/src/routes/VerifyEmail.svelte @@ -104,7 +104,7 @@ let currentUser = userStore.getUser(); try { - currentUser = currentUser || await userStore.loadUser(); + currentUser = currentUser || await userStore.loadUser({ force: true }); } catch {} modalEmail = currentUser?.email || ''; modalDestination = state.address ? `/participant/${state.address}` : '/'; @@ -140,7 +140,7 @@ } } else { await confirmEmailVerification(token); - await userStore.loadUser(); + await userStore.loadUser({ force: true }); destination = '/profile'; } diff --git a/frontend/src/tests/authSession.test.js b/frontend/src/tests/authSession.test.js index cb403b8f..42071a1c 100644 --- a/frontend/src/tests/authSession.test.js +++ b/frontend/src/tests/authSession.test.js @@ -115,4 +115,73 @@ describe('auth session refresh', () => { await vi.advanceTimersByTimeAsync(5 * 60 * 1000); expect(mocks.post).toHaveBeenCalledTimes(1); }); + + // resetModules() re-imports auth.js, which registers another visibilitychange + // listener on the shared document, so earlier tests leave listeners behind. + // These assertions therefore measure growth across flips, not absolute counts. + it('throttles the visibility refresh so tab flipping is not one request per flip', async () => { + const { authState } = await importAuth(); + mocks.post.mockResolvedValue({ data: {} }); + authState.setAuthenticated(true, '0x123'); + + document.dispatchEvent(new Event('visibilitychange')); + await vi.advanceTimersByTimeAsync(1000); + const afterFirstFlip = mocks.post.mock.calls.length; + expect(afterFirstFlip).toBeGreaterThan(0); + + for (let flip = 0; flip < 4; flip += 1) { + setDocumentHidden(true); + document.dispatchEvent(new Event('visibilitychange')); + setDocumentHidden(false); + document.dispatchEvent(new Event('visibilitychange')); + await vi.advanceTimersByTimeAsync(1000); + } + + expect(mocks.post.mock.calls.length).toBe(afterFirstFlip); + }); + + it('catches up on visibility once the throttle window has passed', async () => { + const { authState } = await importAuth(); + mocks.post.mockResolvedValue({ data: {} }); + authState.setAuthenticated(true, '0x123'); + + document.dispatchEvent(new Event('visibilitychange')); + await vi.advanceTimersByTimeAsync(0); + const afterFirstFlip = mocks.post.mock.calls.length; + expect(afterFirstFlip).toBeGreaterThan(0); + + await vi.advanceTimersByTimeAsync(61 * 1000); + document.dispatchEvent(new Event('visibilitychange')); + await vi.advanceTimersByTimeAsync(0); + + expect(mocks.post.mock.calls.length).toBeGreaterThan(afterFirstFlip); + }); + + it('does not re-verify immediately after a 5xx', async () => { + const { verifyAuth } = await importAuth(); + const serverError = new Error('boom'); + serverError.response = { status: 500 }; + mocks.get.mockRejectedValue(serverError); + + await verifyAuth({ force: true }); + await verifyAuth({ force: true }); + await verifyAuth({ force: true }); + + // One attempt, then the cooldown absorbs the rest. + expect(mocks.get).toHaveBeenCalledTimes(1); + + await vi.advanceTimersByTimeAsync(31 * 1000); + await verifyAuth({ force: true }); + expect(mocks.get).toHaveBeenCalledTimes(2); + }); + + it('still logs out on a definitive rejection', async () => { + const { verifyAuth, authState } = await importAuth(); + const authError = new Error('gone'); + authError.response = { status: 403 }; + mocks.get.mockRejectedValue(authError); + + await expect(verifyAuth({ force: true })).resolves.toBe(false); + expect(authState.get().isAuthenticated).toBe(false); + }); }); diff --git a/frontend/src/tests/csrf.test.js b/frontend/src/tests/csrf.test.js new file mode 100644 index 00000000..5b38c9f3 --- /dev/null +++ b/frontend/src/tests/csrf.test.js @@ -0,0 +1,138 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +const mocks = vi.hoisted(() => ({ + get: vi.fn(), +})); + +vi.mock('axios', () => ({ + default: { get: mocks.get }, +})); + +async function loadCsrf() { + vi.resetModules(); + return import('../lib/csrf.js'); +} + +function csrfResponse(token = 'token-1') { + return { data: { csrfToken: token, csrfCookieName: 'csrftoken' } }; +} + +describe('csrf token cache', () => { + beforeEach(() => { + mocks.get.mockReset(); + // Production splits the SPA and API across hosts, so document.cookie never + // carries the CSRF cookie. Model that here. + document.cookie = ''; + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('does not touch the endpoint for safe methods', async () => { + const { attachCsrfToken } = await loadCsrf(); + + const config = await attachCsrfToken({ method: 'get' }); + + expect(mocks.get).not.toHaveBeenCalled(); + expect(config.headers?.['X-CSRFToken']).toBeUndefined(); + }); + + it('fetches once and reuses the token for sequential unsafe requests', async () => { + mocks.get.mockResolvedValue(csrfResponse()); + const { attachCsrfToken } = await loadCsrf(); + + const first = await attachCsrfToken({ method: 'post' }); + const second = await attachCsrfToken({ method: 'patch' }); + const third = await attachCsrfToken({ method: 'delete' }); + + expect(mocks.get).toHaveBeenCalledTimes(1); + expect(first.headers['X-CSRFToken']).toBe('token-1'); + expect(second.headers['X-CSRFToken']).toBe('token-1'); + expect(third.headers['X-CSRFToken']).toBe('token-1'); + }); + + it('coalesces simultaneous unsafe requests into one fetch', async () => { + let resolveGet; + mocks.get.mockImplementation( + () => new Promise((resolve) => { resolveGet = resolve; }) + ); + const { attachCsrfToken } = await loadCsrf(); + + const pending = Promise.all([ + attachCsrfToken({ method: 'post' }), + attachCsrfToken({ method: 'post' }), + ]); + resolveGet(csrfResponse()); + const [first, second] = await pending; + + expect(mocks.get).toHaveBeenCalledTimes(1); + expect(first.headers['X-CSRFToken']).toBe('token-1'); + expect(second.headers['X-CSRFToken']).toBe('token-1'); + }); + + it('refetches after clearCsrfToken', async () => { + mocks.get + .mockResolvedValueOnce(csrfResponse('token-1')) + .mockResolvedValueOnce(csrfResponse('token-2')); + const { attachCsrfToken, clearCsrfToken } = await loadCsrf(); + + await attachCsrfToken({ method: 'post' }); + clearCsrfToken(); + const after = await attachCsrfToken({ method: 'post' }); + + expect(mocks.get).toHaveBeenCalledTimes(2); + expect(after.headers['X-CSRFToken']).toBe('token-2'); + }); + + it('never writes the token to persistent storage', async () => { + mocks.get.mockResolvedValue(csrfResponse('secret-token')); + const localSpy = vi.spyOn(Storage.prototype, 'setItem'); + const { attachCsrfToken } = await loadCsrf(); + + await attachCsrfToken({ method: 'post' }); + + const persisted = localSpy.mock.calls.map(([, value]) => String(value)); + expect(persisted.some((value) => value.includes('secret-token'))).toBe(false); + }); + + it('prefers a readable cookie over the cached token', async () => { + document.cookie = 'csrftoken=cookie-token'; + const { attachCsrfToken } = await loadCsrf(); + + const config = await attachCsrfToken({ method: 'post' }); + + expect(mocks.get).not.toHaveBeenCalled(); + expect(config.headers['X-CSRFToken']).toBe('cookie-token'); + }); +}); + +describe('isCsrfFailure', () => { + it('recognises a DRF CSRF rejection', async () => { + const { isCsrfFailure } = await loadCsrf(); + + expect(isCsrfFailure({ + response: { status: 403, data: { detail: 'CSRF Failed: Origin checking failed.' } }, + })).toBe(true); + }); + + it('does not treat a permission 403 as a CSRF failure', async () => { + const { isCsrfFailure } = await loadCsrf(); + + expect(isCsrfFailure({ + response: { + status: 403, + data: { detail: 'You do not have permission to perform this action.' }, + }, + })).toBe(false); + }); + + it('ignores non-403 responses and network errors', async () => { + const { isCsrfFailure } = await loadCsrf(); + + expect(isCsrfFailure({ response: { status: 401, data: { detail: 'CSRF Failed: x' } } })).toBe(false); + expect(isCsrfFailure({ response: { status: 500, data: {} } })).toBe(false); + expect(isCsrfFailure({})).toBe(false); + expect(isCsrfFailure(undefined)).toBe(false); + }); +}); diff --git a/frontend/src/tests/notificationCenterRequests.test.js b/frontend/src/tests/notificationCenterRequests.test.js new file mode 100644 index 00000000..4d481b19 --- /dev/null +++ b/frontend/src/tests/notificationCenterRequests.test.js @@ -0,0 +1,142 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { render } from '@testing-library/svelte/svelte5'; + +/** + * The navbar bell used to call loadLatest() on every route change even while + * closed, which cost a list request plus a redundant unread-count. Nothing + * rendered the component in tests, so the bug shipped unnoticed. These tests + * render it. + */ + +const mocks = vi.hoisted(() => { + // A minimal writable: vi.hoisted runs before imports, so svelte/store is not + // available here. + function store(initial) { + let value = initial; + const subscribers = new Set(); + return { + subscribe(run) { + subscribers.add(run); + run(value); + return () => subscribers.delete(run); + }, + set(next) { + value = next; + subscribers.forEach((run) => run(value)); + }, + }; + } + + return { + store, + loadLatest: vi.fn(), + loadUnreadCount: vi.fn(), + startPolling: vi.fn(() => () => {}), + reset: vi.fn(), + markRead: vi.fn(), + markAllRead: vi.fn(), + location: store('/'), + authState: store({ isAuthenticated: true }), + notifications: store({ + items: [], + unreadCount: 0, + loading: false, + error: null, + }), + }; +}); + +vi.mock('svelte-spa-router', () => ({ + push: vi.fn(), + location: mocks.location, +})); + +vi.mock('../lib/auth.js', () => ({ + authState: mocks.authState, +})); + +vi.mock('../lib/notificationStore.js', () => ({ + notificationStore: { + subscribe: mocks.notifications.subscribe, + loadLatest: mocks.loadLatest, + loadUnreadCount: mocks.loadUnreadCount, + startPolling: mocks.startPolling, + reset: mocks.reset, + markRead: mocks.markRead, + markAllRead: mocks.markAllRead, + }, +})); + +async function flush() { + await new Promise((resolve) => setTimeout(resolve, 0)); +} + +describe('NotificationCenter request volume', () => { + beforeEach(() => { + mocks.loadLatest.mockReset(); + mocks.loadUnreadCount.mockReset(); + mocks.location.set('/'); + mocks.authState.set({ isAuthenticated: true }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('only refreshes the unread count on route changes while closed', async () => { + const NotificationCenter = (await import('../components/NotificationCenter.svelte')).default; + render(NotificationCenter); + await flush(); + + mocks.loadLatest.mockClear(); + mocks.loadUnreadCount.mockClear(); + + for (const path of ['/builders', '/validators', '/community/poaps', '/profile']) { + mocks.location.set(path); + await flush(); + } + + expect(mocks.loadLatest).not.toHaveBeenCalled(); + expect(mocks.loadUnreadCount).toHaveBeenCalledTimes(4); + }); + + it('loads the list when the panel is opened', async () => { + const NotificationCenter = (await import('../components/NotificationCenter.svelte')).default; + const { getByRole } = render(NotificationCenter); + await flush(); + mocks.loadLatest.mockClear(); + + getByRole('button', { name: /notification/i }).click(); + await flush(); + + expect(mocks.loadLatest).toHaveBeenCalledTimes(1); + }); + + it('does not load the list on the notifications route', async () => { + const NotificationCenter = (await import('../components/NotificationCenter.svelte')).default; + render(NotificationCenter); + await flush(); + mocks.loadLatest.mockClear(); + mocks.loadUnreadCount.mockClear(); + + mocks.location.set('/notifications'); + await flush(); + + expect(mocks.loadLatest).not.toHaveBeenCalled(); + expect(mocks.loadUnreadCount).toHaveBeenCalledTimes(1); + }); + + it('resets instead of fetching when unauthenticated', async () => { + const NotificationCenter = (await import('../components/NotificationCenter.svelte')).default; + render(NotificationCenter); + await flush(); + mocks.loadUnreadCount.mockClear(); + mocks.reset.mockClear(); + + mocks.authState.set({ isAuthenticated: false }); + await flush(); + + expect(mocks.loadUnreadCount).not.toHaveBeenCalled(); + expect(mocks.reset).toHaveBeenCalled(); + }); +}); diff --git a/frontend/src/tests/userStore.test.js b/frontend/src/tests/userStore.test.js index 4e5f517d..8f56491e 100644 --- a/frontend/src/tests/userStore.test.js +++ b/frontend/src/tests/userStore.test.js @@ -233,4 +233,76 @@ describe('userStore', () => { unsubscribe(); }); }); + + // Every role-gated navigation calls loadUser(). In-flight coalescing only + // covers overlapping calls, so without a TTL each navigation refetched. + describe('success-cache TTL', () => { + const mockUser = { id: 1, name: 'Cached User', address: '0xabc' }; + + it('serves sequential loads from cache within the TTL', async () => { + getCurrentUser.mockResolvedValue(mockUser); + + await userStore.loadUser(); + const second = await userStore.loadUser(); + await userStore.loadUser(); + + expect(getCurrentUser).toHaveBeenCalledTimes(1); + expect(second).toEqual(mockUser); + }); + + it('refetches once the TTL has elapsed', async () => { + getCurrentUser.mockResolvedValue(mockUser); + const nowSpy = vi.spyOn(Date, 'now'); + + nowSpy.mockReturnValue(1_000_000); + await userStore.loadUser(); + nowSpy.mockReturnValue(1_000_000 + userStore.USER_CACHE_TTL_MS + 1); + await userStore.loadUser(); + + expect(getCurrentUser).toHaveBeenCalledTimes(2); + nowSpy.mockRestore(); + }); + + it('always refetches with force', async () => { + getCurrentUser.mockResolvedValue(mockUser); + + await userStore.loadUser(); + await userStore.loadUser({ force: true }); + + expect(getCurrentUser).toHaveBeenCalledTimes(2); + }); + + it('does not extend the TTL after a 5xx, and keeps the known user', async () => { + getCurrentUser.mockResolvedValueOnce(mockUser); + await userStore.loadUser(); + + const serverError = new Error('boom'); + serverError.response = { status: 500 }; + getCurrentUser.mockRejectedValueOnce(serverError); + await expect(userStore.loadUser({ force: true })).rejects.toThrow('boom'); + expect(get(userStore).user).toEqual(mockUser); + + // The failure must not have refreshed the timestamp, so the next + // uncached read goes back to the network rather than serving stale data + // off a TTL the failure extended. + getCurrentUser.mockResolvedValueOnce(mockUser); + await userStore.loadUser({ force: true }); + expect(getCurrentUser).toHaveBeenCalledTimes(3); + }); + + it('clears the cache on 401 so the next load refetches', async () => { + getCurrentUser.mockResolvedValueOnce(mockUser); + await userStore.loadUser(); + + const authError = new Error('unauthenticated'); + authError.response = { status: 401 }; + getCurrentUser.mockRejectedValueOnce(authError); + await expect(userStore.loadUser({ force: true })).rejects.toThrow('unauthenticated'); + expect(get(userStore).user).toBeNull(); + + getCurrentUser.mockResolvedValueOnce(mockUser); + await userStore.loadUser(); + expect(getCurrentUser).toHaveBeenCalledTimes(3); + }); + }); }); From f5692c51b597947f81d90f78572dca428095162d Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 15:08:53 +0200 Subject: [PATCH 06/21] Make slow requests visible in production logs Production logged only server errors, so the 27 July degradation left almost no application trace: tens of thousands of successful requests averaging nearly five seconds produced no log lines at all. The middleware already measured duration and built the message, then discarded both for anything that was not a 5xx. Requests at or above a configurable threshold, one second by default, now emit a single warning. Everything needed was already in place and is reused rather than rebuilt: the existing duration measurement, the existing path redaction, the existing skip list for static and health paths, and the request path rather than the full URL, so query strings were never logged and still are not. Server errors keep their existing error-level line and are not logged twice. ## Claude Implementation Notes - backend/tally/middleware/api_logging.py: Add an `elif duration_ms >= settings.SLOW_REQUEST_LOG_MS` warning branch after the existing 5xx and DEBUG branches, so a slow 5xx logs once as an error. Update the module docstring. - backend/tally/settings.py: SLOW_REQUEST_LOG_MS, env-overridable, default 1000, so the threshold is tunable without a deploy. The tally.api logger is already at WARNING in production, so the record emits through the existing JSON handler. - backend/tally/tests/test_api_logging.py: New SlowRequestLoggingTest covering the threshold, fast requests logging nothing, a slow 5xx logging only the error, slow 4xx, claim-link redaction in the warning, query strings never appearing, and skipped paths staying silent. --- backend/tally/middleware/api_logging.py | 9 ++- backend/tally/settings.py | 5 ++ backend/tally/tests/test_api_logging.py | 85 +++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 1 deletion(-) diff --git a/backend/tally/middleware/api_logging.py b/backend/tally/middleware/api_logging.py index aa239d25..4d4a728a 100644 --- a/backend/tally/middleware/api_logging.py +++ b/backend/tally/middleware/api_logging.py @@ -3,7 +3,8 @@ Logs HTTP requests/responses for the [API] layer with smart trace breakdown. - DEBUG=true: Logs all requests; shows breakdown for slow requests (>100ms) -- DEBUG=false: Logs only 5xx errors with breakdown +- DEBUG=false: Logs 5xx errors with breakdown, plus one warning per request + slower than settings.SLOW_REQUEST_LOG_MS """ import re import time @@ -122,6 +123,12 @@ def __call__(self, request): logger.error(log_message) elif settings.DEBUG: logger.debug(log_message) + elif duration_ms >= settings.SLOW_REQUEST_LOG_MS: + # Without this, production logs only 5xx, so a flood of slow-but- + # successful requests leaves no trace at all. One warning per slow + # request; the message carries method, redacted path, status and + # duration, and never a query string or body. + logger.warning(log_message) # Clear request tracking clear_correlation_id() diff --git a/backend/tally/settings.py b/backend/tally/settings.py index 4da3eb72..0236464e 100644 --- a/backend/tally/settings.py +++ b/backend/tally/settings.py @@ -383,6 +383,11 @@ def get_required_env(key): 'http://127.0.0.1:55010', ])) +# Production logs only 5xx, so slow-but-successful requests were invisible. +# Requests at or above this duration emit one sanitized WARNING. Tunable +# without a deploy. +SLOW_REQUEST_LOG_MS = int(os.environ.get('SLOW_REQUEST_LOG_MS', 1000)) + # Session settings SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_SAMESITE = 'Lax' # 'None' if using cross-site cookies, but requires HTTPS diff --git a/backend/tally/tests/test_api_logging.py b/backend/tally/tests/test_api_logging.py index 86eae9ed..3fae59ee 100644 --- a/backend/tally/tests/test_api_logging.py +++ b/backend/tally/tests/test_api_logging.py @@ -1,3 +1,5 @@ +import time + from django.http import HttpResponse from django.test import RequestFactory, SimpleTestCase, override_settings @@ -42,3 +44,86 @@ def test_server_error_logging_redacts_poap_claim_link_token(self): self.assertIn('/api/v1/poaps/claim-link//', logs) self.assertIn('500', logs) self.assertNotIn(token, logs) + + +@override_settings(DEBUG=False, SLOW_REQUEST_LOG_MS=50) +class SlowRequestLoggingTest(SimpleTestCase): + """ + Production logged only 5xx, so a flood of slow successful requests left no + trace. These pin the threshold, the redaction, and the no-double-log rule. + """ + + def setUp(self): + self.factory = RequestFactory() + + def _middleware(self, status=200, delay=0.0): + def view(_request): + if delay: + time.sleep(delay) + return HttpResponse('ok', status=status) + return APILoggingMiddleware(view) + + def test_slow_successful_request_logs_one_warning(self): + request = self.factory.get('/api/v1/notifications/unread-count/') + + with self.assertLogs('tally.api', level='WARNING') as captured: + self._middleware(delay=0.08)(request) + + self.assertEqual(len(captured.records), 1) + record = captured.records[0] + self.assertEqual(record.levelname, 'WARNING') + self.assertIn('GET', record.getMessage()) + self.assertIn('/api/v1/notifications/unread-count/', record.getMessage()) + self.assertIn('200', record.getMessage()) + self.assertIn('ms', record.getMessage()) + + def test_fast_request_logs_nothing(self): + request = self.factory.get('/api/v1/notifications/unread-count/') + + with self.assertNoLogs('tally.api', level='DEBUG'): + self._middleware()(request) + + def test_slow_server_error_logs_only_the_error(self): + request = self.factory.get('/api/v1/notifications/unread-count/') + + with self.assertLogs('tally.api', level='DEBUG') as captured: + self._middleware(status=500, delay=0.08)(request) + + levels = [record.levelname for record in captured.records] + self.assertEqual(levels, ['ERROR']) + + def test_slow_4xx_still_logs_a_warning(self): + request = self.factory.get('/api/v1/notifications/unread-count/') + + with self.assertLogs('tally.api', level='WARNING') as captured: + self._middleware(status=403, delay=0.08)(request) + + self.assertEqual(len(captured.records), 1) + self.assertIn('403', captured.records[0].getMessage()) + + def test_slow_request_redacts_sensitive_path(self): + token = 'slow-synthetic-claim-token' + request = self.factory.post(f'/api/v1/poaps/claim-link/{token}/') + + with self.assertLogs('tally.api', level='WARNING') as captured: + self._middleware(delay=0.08)(request) + + logs = '\n'.join(captured.output) + self.assertIn('/api/v1/poaps/claim-link//', logs) + self.assertNotIn(token, logs) + + def test_query_string_is_never_logged(self): + request = self.factory.get( + '/api/v1/notifications/', {'secret_token': 'do-not-log-me'} + ) + + with self.assertLogs('tally.api', level='WARNING') as captured: + self._middleware(delay=0.08)(request) + + self.assertNotIn('do-not-log-me', '\n'.join(captured.output)) + + def test_skipped_paths_log_nothing_even_when_slow(self): + request = self.factory.get('/health/') + + with self.assertNoLogs('tally.api', level='DEBUG'): + self._middleware(delay=0.08)(request) From 8634e0808a4a5675c0d6ff778c6e3da4ab68ee33 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 15:08:53 +0200 Subject: [PATCH 07/21] Silence the Gunicorn control socket warning at startup Gunicorn 26 opens a control socket in the working directory, which the container owns as root while the application process runs unprivileged, so every boot logs a permission warning. Nothing in the application uses the control socket. Unrelated to the 27 July degradation. ## Claude Implementation Notes - backend/Dockerfile, backend/deploy-apprunner.sh, backend/deploy-apprunner-dev.sh: Add --no-control-socket to all five copies of the Gunicorn start command. Worker count, bind, timeout and logging flags unchanged. Flag confirmed present in the pinned gunicorn 26.0.0. --- backend/Dockerfile | 2 +- backend/deploy-apprunner-dev.sh | 4 ++-- backend/deploy-apprunner.sh | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/backend/Dockerfile b/backend/Dockerfile index 9360464c..3dd93ce5 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -42,4 +42,4 @@ EXPOSE 8000 USER app # Run startup script -CMD ["./startup.sh", "gunicorn", "--bind", "0.0.0.0:8000", "--timeout", "180", "--workers", "2", "--access-logfile", "-", "--error-logfile", "-", "--capture-output", "--log-level", "info", "tally.wsgi:application"] +CMD ["./startup.sh", "gunicorn", "--no-control-socket", "--bind", "0.0.0.0:8000", "--timeout", "180", "--workers", "2", "--access-logfile", "-", "--error-logfile", "-", "--capture-output", "--log-level", "info", "tally.wsgi:application"] diff --git a/backend/deploy-apprunner-dev.sh b/backend/deploy-apprunner-dev.sh index e01416e5..ea7ce13c 100755 --- a/backend/deploy-apprunner-dev.sh +++ b/backend/deploy-apprunner-dev.sh @@ -168,7 +168,7 @@ if aws apprunner describe-service --service-arn arn:aws:apprunner:$REGION:$ACCOU "TWITTER_REDIRECT_URI": "$SSM_PREFIX/$SSM_ENV/twitter_redirect_uri", "DISCORD_REDIRECT_URI": "$SSM_PREFIX/$SSM_ENV/discord_redirect_uri" }, - "StartCommand": "./startup.sh gunicorn --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" + "StartCommand": "./startup.sh gunicorn --no-control-socket --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" }, "ImageRepositoryType": "ECR" }, @@ -308,7 +308,7 @@ EOF "TWITTER_REDIRECT_URI": "$SSM_PREFIX/$SSM_ENV/twitter_redirect_uri", "DISCORD_REDIRECT_URI": "$SSM_PREFIX/$SSM_ENV/discord_redirect_uri" }, - "StartCommand": "./startup.sh gunicorn --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" + "StartCommand": "./startup.sh gunicorn --no-control-socket --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" }, "ImageRepositoryType": "ECR" }, diff --git a/backend/deploy-apprunner.sh b/backend/deploy-apprunner.sh index 55626ffa..4a6ba787 100755 --- a/backend/deploy-apprunner.sh +++ b/backend/deploy-apprunner.sh @@ -250,7 +250,7 @@ if aws apprunner describe-service --service-arn arn:aws:apprunner:$REGION:$ACCOU "DISCORD_NEUROCREATIVE_ROLE_ID": "$SSM_PREFIX/prod/discord_neurocreative_role_id", "DISCORD_REDIRECT_URI": "$SSM_PREFIX/prod/discord_redirect_uri" }, - "StartCommand": "./startup.sh gunicorn --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" + "StartCommand": "./startup.sh gunicorn --no-control-socket --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" }, "ImageRepositoryType": "ECR" }, @@ -368,7 +368,7 @@ else "DISCORD_NEUROCREATIVE_ROLE_ID": "$SSM_PREFIX/prod/discord_neurocreative_role_id", "DISCORD_REDIRECT_URI": "$SSM_PREFIX/prod/discord_redirect_uri" }, - "StartCommand": "./startup.sh gunicorn --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" + "StartCommand": "./startup.sh gunicorn --no-control-socket --bind 0.0.0.0:8000 --timeout 180 --workers 2 --access-logfile - --error-logfile - --capture-output --log-level info tally.wsgi:application" }, "ImageRepositoryType": "ECR" }, From 459db6e12369163b75aaafa807883b1c34e7c51d Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 15:08:53 +0200 Subject: [PATCH 08/21] Document the request-volume and query-shape constraints ## Claude Implementation Notes - backend/CLAUDE.md: Record that EthereumAuthentication runs on every DRF request and must never look a user up by address; document the case-insensitive wallet binding, the login_wallet_session test helper, the Upper(address) index, the community aggregate cache and its clear_community_caches() test requirement, and SLOW_REQUEST_LOG_MS. - frontend/CLAUDE.md: Document the notification route-change rule and poll backoff, why markRead refetches rather than decrementing, the new CSRF section (cross-host cookie, in-memory-only cache, clear paths, why the 403 re-verify branch must stay), the userStore TTL and its force call sites, and the verify cooldown plus refresh throttle. --- backend/CLAUDE.md | 34 ++++++++++++++++++++++++++++++++++ frontend/CLAUDE.md | 19 +++++++++++++++++-- 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 3c16f8f0..dac34978 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -566,6 +566,7 @@ Located in `.env` file: - `RECAPTCHA_PRIVATE_KEY` - Google reCAPTCHA secret key (required - use test key from .env.example for development) - `RECAPTCHA_ALLOW_TEST_KEYS` - Optional opt-in flag for non-production deployments that intentionally use Google's reCAPTCHA test keys with `DEBUG=False`. Set to `true` to silence `django_recaptcha.recaptcha_test_key_error`; production must not set this flag. The logic lives in `tally/settings.py` near `_RECAPTCHA_TEST_PUBLIC_KEY` and `SILENCED_SYSTEM_CHECKS`. - `CRON_SYNC_TOKEN` - Cron-protected endpoint auth (used by `sync` and `sync-grafana`) +- `SLOW_REQUEST_LOG_MS` - Duration in ms (default `1000`) at or above which `APILoggingMiddleware` emits one sanitized WARNING for a non-5xx request. Production otherwise logs only 5xx, so slow successful requests leave no trace. The message carries method, redacted path, status and duration only, never query strings or bodies. - `DISCORD_SYNAPSE_ROLE_ID` / `DISCORD_BRAIN_ROLE_ID` / `DISCORD_NEUROCREATIVE_ROLE_ID` - Discord role IDs for the earned community role automation (Synapse/Brain assignment). All three must be set or the assignment job is a no-op. - `GRAFANA_BASE_URL` - Grafana Cloud base URL (default `https://genlayerfoundation.grafana.net`) - `GRAFANA_API_TOKEN` - Grafana service-account bearer token (required for Wall of Shame). Store in AWS SSM (`/tally/{env}/grafana_api_token`) for production. @@ -663,6 +664,39 @@ The project uses **context-aware serialization** to optimize API performance: - `LightEvidenceURLTypeSerializer` - Minimal (id, name, slug, is_generic) for nested use in Evidence responses - `EvidenceURLTypeSerializer` - Full serializer with url_patterns for client-side detection, used in ContributionType responses +### Per-request user lookup (do not reintroduce the address scan) + +`EthereumAuthentication` (`ethereum_auth/authentication.py`) runs on EVERY DRF request: +it is first in `DEFAULT_AUTHENTICATION_CLASSES` and DRF resolves `request.user` +unconditionally. It resolves the user from Django's own session machinery via +`request._request.user` (one primary-key lookup, plus `_auth_user_backend` / +`_auth_user_hash` / `is_active` validation), then checks that the session's +`ethereum_address` still binds to that user **case-insensitively** (login stores the +lowercased SIWE address, `signup_email_confirm` stores database casing, and production +holds mixed-case rows). A wallet mismatch RAISES rather than returning `None`, so the +next authenticator in the chain cannot grant the request off the same session user. + +Never look the user up by `address__iexact` in a per-request path. `iexact` compiles to +`UPPER(address) = UPPER(...)` on PostgreSQL and the only unique index on the column is +case-sensitive, so it is a sequential scan. Migration `users/0022` adds a non-unique +functional index on `Upper('address')` for the case-insensitive lookups that legitimately +remain (`users/utils.py::user_lookup_kwargs`, address search). Guards: +`ethereum_auth/test_authentication.py::WalletSessionQueryShapeTests`. + +Session-based tests must build a real session with +`ethereum_auth.testing.login_wallet_session(client, user)`; hand-seeding only +`ethereum_address` + `authenticated` produces a session the authenticator rejects. + +### Community aggregate caching + +`community_xp/cache.py` holds a 60s cache for the two shared, non-personalized community +aggregates: the ranking snapshot (`list[(user_id, total_points)]`) and the stats summary. +Both take no request input. Search, `user_rank`, `profile_context` and hydration stay live +per request. There is no `CACHES` setting, so this is per-process `LocMemCache`: the relief +scales with worker/container count and is strongest at steady state. Tests that touch the +community endpoints must call `clear_community_caches()` in `setUp` (LocMemCache is not +reset between tests). Query-count guards: `leaderboard/tests/test_community_query_counts.py`. + ## Testing - **Test Organization Best Practice**: Use `{app}/tests/` folder structure for better organization - Create `{app}/tests/__init__.py` to make it a Python package diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 18886390..28375c36 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -294,7 +294,9 @@ frontend/src/ - **Notifications**: `src/components/NotificationCenter.svelte` - Bell icon button in the navbar, left of the search bar on desktop, before the auth button on mobile; only when authenticated - Unread badge, dropdown with latest notifications, mark-all-read, "View all" linking to `/notifications`. Dropdown body previews retain sanitized markdown formatting and links while being capped at 120 characters and visually clamped to two lines; the full notifications page retains rich markdown. - - Polls unread count every 60s while the tab is visible; the desktop/mobile instances share one refcounted timer and visibility listener, and returning to the tab refreshes immediately. Clicking a notification marks it read (non-blocking) and follows its `link_url` (internal routes push in-app, http(s) opens a new tab) + - Polls unread count every 60s while the tab is visible; the desktop/mobile instances share one refcounted timer and visibility listener, and returning to the tab refreshes immediately. Failed polls back off (60s → 180s → 240s, reset on the first success) so a degraded backend is not hammered at a fixed rate by every open tab. Clicking a notification marks it read (non-blocking) and follows its `link_url` (internal routes push in-app, http(s) opens a new tab) + - **Route changes only refresh the unread count, never the list.** The `$effect` tracks `$location` and `$authState`, so it reruns on every navigation and every auth-store emission; calling `loadLatest()` there cost a list request plus a redundant unread-count on each one, doubled by the two mounted instances. `loadLatest()` now runs only when the panel is opened. Guard: `src/tests/notificationCenterRequests.test.js` (renders the component; `notificationPolling.test.js` only exercises the store). + - `markRead` deliberately refetches the count instead of decrementing locally: a count request can observe the server-side mark-read before the POST resolves, and a blind decrement would then subtract it twice (pinned by `notificationPolling.test.js`). `markAllRead` is local because it sets the count to zero outright. - Full feed page: `src/routes/Notifications.svelte` (All/Unread filter pills, load-more pagination). Bodies render as sanitized image-free markdown via `parseUserMarkdown()` (no ``, so private campaign opens can't ping external tracking pixels); rows are `div[role=button]` so markdown links stay clickable, inline anchor clicks don't also trigger the row's `link_url` redirect, and rows without a `link_url` show a default cursor (pure announcements) - Shared utils: `src/lib/notificationUtils.js` (`asList` payload normalization, `notificationBodyPreview` compact dropdown copy, `followNotificationLink` link handling) and `src/lib/relativeTime.js` for compact timestamps - **Sidebar**: `src/components/Sidebar.svelte` @@ -494,6 +496,18 @@ const routes = { - `/api/auth/login/` - `/api/auth/verify/` - `/api/auth/logout/` +- **Verify cooldown**: a 5xx/network verify failure deliberately leaves `hasVerified` unset (so the session is never dropped on a transient error), which used to mean every subsequent `verifyAuth()` re-hit a degraded backend. A 30s cooldown now absorbs the repeats. A definitive `<500` rejection clears the cooldown and still logs out immediately. +- **Refresh throttle**: `refreshSession()` runs on a 5-minute interval AND on `visibilitychange`. The visibility path is throttled to once per 60s, so flipping between tabs no longer produces one `POST /auth/refresh/` per flip. + +### CSRF (`src/lib/csrf.js`) + +In production the SPA (Amplify) and the API are on different hosts and `CSRF_COOKIE_DOMAIN` is unset, so the CSRF cookie is host-only on the API and `document.cookie` can never read it. Every unsafe request therefore used to fetch `/api/csrf/` (the in-flight promise is cleared in `.finally()`, so only literally simultaneous requests shared one). + +- `attachCsrfToken(config)` - adds `X-CSRFToken` to unsafe methods only. Token order: readable cookie (same-origin dev, where Django rotates it for us) → in-memory cache → network. +- **The cached token is in module memory only. Never `localStorage`, `sessionStorage`, or any persistent browser storage.** +- `clearCsrfToken()` - must be called from every path that rotates the server-side token: login and `signup_email_confirm` (Django cycles the token inside `login()`), logout (`session.flush()`), and wallet switch (which goes through logout + sign-in). Session refresh does NOT rotate it. +- `isCsrfFailure(error)` - distinguishes a real CSRF rejection from an authorization 403 by the `CSRF Failed` detail prefix DRF's `enforce_csrf` produces. Both are 403. +- The `api.js` response interceptor clears the cached token on a real CSRF failure and **does not retry**: POAP claims and other non-idempotent mutations share that axios instance. The 401/403 → `verifyAuth({force:true})` branch must stay: DRF answers **403, not 401**, for an expired session, because the first authenticator in the chain supplies no `authenticate_header`. ### Analytics & Campaign Attribution (`src/lib/analytics.js`) @@ -506,12 +520,13 @@ const routes = { ### User Store (`src/lib/userStore.js`) - **Central store for logged-in user data** - **Key Functions**: - - `loadUser()` - Fetch user data from API + - `loadUser({ force } = {})` - Fetch user data from API - `updateUser(updates)` - Partial update of user data - `setUser(userData)` - Set full user data - `clearUser()` - Clear on logout - **Auto-managed**: Loaded on login, cleared on logout - **Reactive**: Updates reflect immediately in all components using `$userStore` +- **30s success-cache TTL** (`USER_CACHE_TTL_MS`): `requireRoleForRoute` calls `loadUser()` on every one of the ~20 role-gated navigations, and in-flight coalescing only covers overlapping calls, so sequential navigation used to refetch `/users/me/` every time. Route guards are a UX gate, not the security boundary (the backend enforces permissions on every request), so a short cache is safe; a role change takes up to 30s to reflect in client-side gating while the backend refuses the data immediately. Only successful loads start the TTL; failures never extend it, and `clearUser()` resets it. **Every other call site passes `{ force: true }`** so its behavior is unchanged: login, wallet switch, profile update, email confirm, journey completion, and the 401/403 recovery path. `performVerification` forwards its own `force` flag, so a forced verify re-reads the profile. - **Steward hierarchy**: authenticated user payloads expose `steward_tier`; the submission queue defaults a missing tier to reviewer tier 1. ### Components From 1fa14989787da0d7daa899fa029c667d8da33c96 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 15:49:48 +0200 Subject: [PATCH 09/21] Repair an interrupted address index build and honour forced profile reads Addresses review of the outage fixes. An interrupted concurrent index build leaves the index in the catalog marked invalid. Retrying with "if not exists" then silently skips it rather than rebuilding, so a deploy that was interrupted once would leave the case-insensitive address lookups permanently unindexed while appearing to succeed. Verified against PostgreSQL: the retry logs "already exists, skipping" and the index stays invalid. The migration now detects that state and clears the leftover before rebuilding. Forcing a profile refresh could also settle for a response that predated the change it was meant to observe. Callers force a refresh precisely because they just altered server state, so a forced read now queues behind any request already in flight instead of joining it, and the newer response is the one that lands in the store. Unforced callers still share in-flight work. Notification poll backoff no longer reacts to superseded responses, so a stale failure cannot re-arm a delay that a newer success already cleared, or reintroduce one after sign-out. ## Claude Implementation Notes - backend/users/migrations/0022_user_address_upper_index.py: Add _index_is_invalid() checking pg_index.indisvalid via to_regclass (search-path aware), and DROP INDEX CONCURRENTLY the leftover before recreating. Non-PostgreSQL path unchanged. Verified end to end against a scratch database: forced an invalid index via the catalog, confirmed CREATE INDEX CONCURRENTLY IF NOT EXISTS no-ops on it, then confirmed create_index() takes it INVALID to VALID and that drop/recreate still works. Not unit-tested: migrations are disabled under test settings and CONCURRENTLY cannot run inside the test transaction. - frontend/src/lib/userStore.js: loadUser({ force: true }) no longer returns an in-flight promise; it awaits the previous load, then issues its own request. A token guard stops a superseded load from clearing the newer in-flight handle, and clearUser() resets both. - frontend/src/lib/notificationStore.js: Move notePollResult() after the epoch/version guard in both handlers. - frontend/src/tests/userStore.test.js: The 5xx TTL test now advances mocked time past the original TTL and reads UNFORCED, which is the only way to catch a failure wrongly refreshing the timestamp; the previous forced read hit the network regardless and proved nothing. Add a test that a forced load started during an in-flight load resolves with the newer response. - backend/tally/settings.py: String default for os.environ.get (Ruff PLW1508). - frontend/src/components/NotificationCenter.svelte: Reword comment to "unread count" per the project terminology rule. --- backend/tally/settings.py | 2 +- .../0022_user_address_upper_index.py | 31 ++++++++++++++- .../src/components/NotificationCenter.svelte | 6 +-- frontend/src/lib/notificationStore.js | 7 +++- frontend/src/lib/userStore.js | 32 ++++++++++++--- frontend/src/tests/userStore.test.js | 39 +++++++++++++++++-- 6 files changed, 100 insertions(+), 17 deletions(-) diff --git a/backend/tally/settings.py b/backend/tally/settings.py index 0236464e..9d118bc6 100644 --- a/backend/tally/settings.py +++ b/backend/tally/settings.py @@ -386,7 +386,7 @@ def get_required_env(key): # Production logs only 5xx, so slow-but-successful requests were invisible. # Requests at or above this duration emit one sanitized WARNING. Tunable # without a deploy. -SLOW_REQUEST_LOG_MS = int(os.environ.get('SLOW_REQUEST_LOG_MS', 1000)) +SLOW_REQUEST_LOG_MS = int(os.environ.get('SLOW_REQUEST_LOG_MS', '1000')) # Session settings SESSION_COOKIE_HTTPONLY = True diff --git a/backend/users/migrations/0022_user_address_upper_index.py b/backend/users/migrations/0022_user_address_upper_index.py index 106ed67b..ffe1fe75 100644 --- a/backend/users/migrations/0022_user_address_upper_index.py +++ b/backend/users/migrations/0022_user_address_upper_index.py @@ -19,10 +19,37 @@ INDEX_NAME = 'users_user_address_upper_idx' +def _index_is_invalid(schema_editor): + """ + True when the index exists but is marked invalid. + + An interrupted CREATE INDEX CONCURRENTLY leaves the index in the catalog + with indisvalid = false. The planner ignores it, but it still carries write + overhead, and a retried CREATE INDEX CONCURRENTLY IF NOT EXISTS silently + skips rather than rebuilding it, so the lookups would stay unindexed. + """ + with schema_editor.connection.cursor() as cursor: + cursor.execute( + 'SELECT indisvalid FROM pg_index WHERE indexrelid = to_regclass(%s)', + [INDEX_NAME], + ) + row = cursor.fetchone() + return row is not None and row[0] is False + + def create_index(apps, schema_editor): - concurrently = 'CONCURRENTLY ' if schema_editor.connection.vendor == 'postgresql' else '' + if schema_editor.connection.vendor != 'postgresql': + schema_editor.execute( + f'CREATE INDEX IF NOT EXISTS {INDEX_NAME} ON users_user (UPPER(address))' + ) + return + + if _index_is_invalid(schema_editor): + # Clear the leftover first; IF NOT EXISTS would skip over it. + schema_editor.execute(f'DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}') + schema_editor.execute( - f'CREATE INDEX {concurrently}IF NOT EXISTS {INDEX_NAME} ' + f'CREATE INDEX CONCURRENTLY IF NOT EXISTS {INDEX_NAME} ' 'ON users_user (UPPER(address))' ) diff --git a/frontend/src/components/NotificationCenter.svelte b/frontend/src/components/NotificationCenter.svelte index d7e07bdc..ecab3ff5 100644 --- a/frontend/src/components/NotificationCenter.svelte +++ b/frontend/src/components/NotificationCenter.svelte @@ -55,9 +55,9 @@ closePanel(); } - // Route changes and auth-store emissions only refresh the badge. The list - // body is fetched when the panel actually opens (toggleOpen), so navigating - // with the bell closed costs one request instead of two. + // Route changes and auth-store emissions only refresh the unread count. The + // list body is fetched when the panel actually opens (toggleOpen), so + // navigating with the bell closed costs one request instead of two. $effect(() => { void $location; if ($authState.isAuthenticated) { diff --git a/frontend/src/lib/notificationStore.js b/frontend/src/lib/notificationStore.js index 74deadb7..2d253eb3 100644 --- a/frontend/src/lib/notificationStore.js +++ b/frontend/src/lib/notificationStore.js @@ -110,13 +110,16 @@ function createNotificationStore() { const request = notificationsAPI .unreadCount() .then((response) => { - notePollResult(true); + // Record the outcome only for the current, unsuperseded request: a + // stale failure would otherwise re-arm backoff after reset() or after a + // newer success had already cleared it. if (requestEpoch !== epoch || requestUnreadVersion !== unreadWriteVersion) return; + notePollResult(true); update((state) => ({ ...state, unreadCount: response.data?.count || 0 })); }) .catch((error) => { - notePollResult(false); if (requestEpoch !== epoch || requestUnreadVersion !== unreadWriteVersion) return; + notePollResult(false); update((state) => ({ ...state, error })); }) .finally(() => { diff --git a/frontend/src/lib/userStore.js b/frontend/src/lib/userStore.js index 8a0fc411..dacabd1e 100644 --- a/frontend/src/lib/userStore.js +++ b/frontend/src/lib/userStore.js @@ -11,6 +11,9 @@ function createUserStore() { error: null }); let loadUserPromise = null; + // Identifies the newest queued load, so a superseded one cannot clear the + // in-flight handle out from under it. + let currentLoadToken = null; // Every role-gated navigation calls loadUser(), and in-flight coalescing only // covers overlapping calls, so sequential navigation used to refetch every // time. Route guards are a UX gate, not a security boundary (the backend @@ -25,7 +28,8 @@ function createUserStore() { // Load user data from API async loadUser({ force = false } = {}) { - if (loadUserPromise) { + // Unforced callers share whatever is already in flight. + if (!force && loadUserPromise) { return loadUserPromise; } @@ -38,9 +42,22 @@ function createUserStore() { return state.user; } - update(state => ({ ...state, loading: true, error: null })); + // A forced caller has just changed server state (login, wallet switch, + // profile edit, role change), so it must not settle for a response to a + // request that started before that change. Queue behind any in-flight + // load instead of joining it; sequencing also keeps the older response + // from landing after the newer one. + const previous = loadUserPromise; + const token = {}; + currentLoadToken = token; + + const request = (async () => { + if (previous) { + await previous.catch(() => {}); + } + + update(state => ({ ...state, loading: true, error: null })); - loadUserPromise = (async () => { try { const userData = await getCurrentUser(); // Only successful loads start the TTL; failures must never extend it. @@ -66,11 +83,14 @@ function createUserStore() { })); throw err; } finally { - loadUserPromise = null; + if (currentLoadToken === token) { + loadUserPromise = null; + } } })(); - return loadUserPromise; + loadUserPromise = request; + return request; }, // Update user data (partial update) @@ -94,6 +114,8 @@ function createUserStore() { // Clear user data (on logout) clearUser() { lastLoadedAt = 0; + currentLoadToken = null; + loadUserPromise = null; set({ user: null, loading: false, diff --git a/frontend/src/tests/userStore.test.js b/frontend/src/tests/userStore.test.js index 8f56491e..c0fff8dd 100644 --- a/frontend/src/tests/userStore.test.js +++ b/frontend/src/tests/userStore.test.js @@ -273,21 +273,52 @@ describe('userStore', () => { }); it('does not extend the TTL after a 5xx, and keeps the known user', async () => { + const nowSpy = vi.spyOn(Date, 'now'); + const start = 1_000_000; + getCurrentUser.mockResolvedValueOnce(mockUser); + nowSpy.mockReturnValue(start); await userStore.loadUser(); const serverError = new Error('boom'); serverError.response = { status: 500 }; getCurrentUser.mockRejectedValueOnce(serverError); + nowSpy.mockReturnValue(start + 20_000); await expect(userStore.loadUser({ force: true })).rejects.toThrow('boom'); expect(get(userStore).user).toEqual(mockUser); - // The failure must not have refreshed the timestamp, so the next - // uncached read goes back to the network rather than serving stale data - // off a TTL the failure extended. + // Past the original TTL but still inside one measured from the failure. + // An UNFORCED read is the only thing that can catch a failure wrongly + // refreshing the timestamp; a forced read would hit the network anyway. getCurrentUser.mockResolvedValueOnce(mockUser); - await userStore.loadUser({ force: true }); + nowSpy.mockReturnValue(start + userStore.USER_CACHE_TTL_MS + 1_000); + await userStore.loadUser(); + expect(getCurrentUser).toHaveBeenCalledTimes(3); + nowSpy.mockRestore(); + }); + + it('a forced load does not settle for a request that predates it', async () => { + // The forced caller has just mutated server state, so joining the + // in-flight response would hand back pre-mutation data. + let resolveFirst; + getCurrentUser.mockImplementationOnce( + () => new Promise((resolve) => { resolveFirst = resolve; }) + ); + const stale = { ...mockUser, name: 'Stale' }; + const fresh = { ...mockUser, name: 'Fresh' }; + + const backgroundLoad = userStore.loadUser(); + const forcedLoad = userStore.loadUser({ force: true }); + + getCurrentUser.mockResolvedValueOnce(fresh); + resolveFirst(stale); + + await expect(backgroundLoad).resolves.toEqual(stale); + await expect(forcedLoad).resolves.toEqual(fresh); + expect(getCurrentUser).toHaveBeenCalledTimes(2); + // The newer response must be the one left in the store. + expect(get(userStore).user).toEqual(fresh); }); it('clears the cache on 401 so the next load refetches', async () => { From dd60f926db204b0a78152274c17678cd4a02cd51 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 16:41:31 +0200 Subject: [PATCH 10/21] Discard profile responses that arrive after sign-out A profile request already in flight when the account changes would still write its result into the store when it resolved, restoring the previous account after a sign-out or a wallet switch and starting a freshness window for data that no longer applied. The notification store already guards against exactly this with its epoch counter; the user store did not. Every write is now gated on the load still being the current one, and clearing the user invalidates whatever is in flight. ## Claude Implementation Notes - frontend/src/lib/userStore.js: Gate the success write, lastLoadedAt, and the error write on currentLoadToken === token, and bail before the loading flag if the load was superseded while queued. clearUser() already drops the token, which is what invalidates in-flight work. The request still resolves or rejects for its caller; only the shared store state is protected. - frontend/src/tests/userStore.test.js: Two regression tests covering a deferred request, clearUser(), then resolution: the store must stay empty, and the discarded response must not seed the TTL. Both verified to fail without the guard (the first asserts the logged-out account was being restored), so they are not vacuous. - frontend/CLAUDE.md: Document the token guard and why forced loads queue rather than join. --- frontend/CLAUDE.md | 1 + frontend/src/lib/userStore.js | 29 +++++++++++++++------- frontend/src/tests/userStore.test.js | 36 ++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 8 deletions(-) diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 28375c36..b48794a7 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -527,6 +527,7 @@ In production the SPA (Amplify) and the API are on different hosts and `CSRF_COO - **Auto-managed**: Loaded on login, cleared on logout - **Reactive**: Updates reflect immediately in all components using `$userStore` - **30s success-cache TTL** (`USER_CACHE_TTL_MS`): `requireRoleForRoute` calls `loadUser()` on every one of the ~20 role-gated navigations, and in-flight coalescing only covers overlapping calls, so sequential navigation used to refetch `/users/me/` every time. Route guards are a UX gate, not the security boundary (the backend enforces permissions on every request), so a short cache is safe; a role change takes up to 30s to reflect in client-side gating while the backend refuses the data immediately. Only successful loads start the TTL; failures never extend it, and `clearUser()` resets it. **Every other call site passes `{ force: true }`** so its behavior is unchanged: login, wallet switch, profile update, email confirm, journey completion, and the 401/403 recovery path. `performVerification` forwards its own `force` flag, so a forced verify re-reads the profile. +- **In-flight responses are token-guarded.** A forced load does not join an in-flight request (the caller forced it because it just changed server state, so an earlier response would be pre-change); it queues behind that request and issues its own. Every store write, including `lastLoadedAt`, is gated on the load still being the current one, and `clearUser()` drops the token. Without that gate a `/users/me/` response arriving after logout or a wallet switch restores the previous account into the store, which is the same hazard `notificationStore`'s `epoch` counter exists to prevent. - **Steward hierarchy**: authenticated user payloads expose `steward_tier`; the submission queue defaults a missing tier to reviewer tier 1. ### Components diff --git a/frontend/src/lib/userStore.js b/frontend/src/lib/userStore.js index dacabd1e..6d9f4a3e 100644 --- a/frontend/src/lib/userStore.js +++ b/frontend/src/lib/userStore.js @@ -56,10 +56,21 @@ function createUserStore() { await previous.catch(() => {}); } + // Superseded by clearUser() or a newer load while we were queued. + if (currentLoadToken !== token) { + return get({ subscribe }).user; + } + update(state => ({ ...state, loading: true, error: null })); try { const userData = await getCurrentUser(); + // A response that lands after logout or a wallet switch must not + // restore the previous account, nor start a TTL for data that never + // reached the store. clearUser() drops the token to invalidate it. + if (currentLoadToken !== token) { + return userData; + } // Only successful loads start the TTL; failures must never extend it. lastLoadedAt = Date.now(); update(state => ({ @@ -73,14 +84,16 @@ function createUserStore() { // Only a definitive auth rejection means "no user". On network/5xx // failures keep any previously loaded user so role gating and journey // state don't reset while the backend is down. - const status = err.response?.status; - const unauthenticated = status === 401 || status === 403; - update(state => ({ - ...state, - user: unauthenticated ? null : state.user, - loading: false, - error: err.message || 'Failed to load user data' - })); + if (currentLoadToken === token) { + const status = err.response?.status; + const unauthenticated = status === 401 || status === 403; + update(state => ({ + ...state, + user: unauthenticated ? null : state.user, + loading: false, + error: err.message || 'Failed to load user data' + })); + } throw err; } finally { if (currentLoadToken === token) { diff --git a/frontend/src/tests/userStore.test.js b/frontend/src/tests/userStore.test.js index c0fff8dd..fb27d497 100644 --- a/frontend/src/tests/userStore.test.js +++ b/frontend/src/tests/userStore.test.js @@ -298,6 +298,42 @@ describe('userStore', () => { nowSpy.mockRestore(); }); + it('discards a response that lands after clearUser', async () => { + // Logout and wallet switch both clear the store while a load may still + // be in flight; the old account must not reappear when it resolves. + let resolveLoad; + getCurrentUser.mockImplementationOnce( + () => new Promise((resolve) => { resolveLoad = resolve; }) + ); + + const pending = userStore.loadUser(); + userStore.clearUser(); + resolveLoad(mockUser); + await pending; + + expect(get(userStore).user).toBeNull(); + }); + + it('does not let a post-clearUser response seed the cache', async () => { + let resolveLoad; + getCurrentUser.mockImplementationOnce( + () => new Promise((resolve) => { resolveLoad = resolve; }) + ); + + const pending = userStore.loadUser(); + userStore.clearUser(); + resolveLoad(mockUser); + await pending; + + // The discarded response must not have started a TTL, so the next read + // goes to the network instead of serving an account that was logged out. + getCurrentUser.mockResolvedValueOnce(mockUser); + await userStore.loadUser(); + + expect(getCurrentUser).toHaveBeenCalledTimes(2); + expect(get(userStore).user).toEqual(mockUser); + }); + it('a forced load does not settle for a request that predates it', async () => { // The forced caller has just mutated server state, so joining the // in-flight response would hand back pre-mutation data. From 91da6b64f6bb92a5d2147a6ae9d0fa2fc26d69c9 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 17:27:43 +0200 Subject: [PATCH 11/21] Stop a late profile read from reverting a just-saved profile Writing the user store directly always carries state that is newer than any read already in flight: the server's own user from a profile save, a role join or a claim. Those writes could still be overwritten when an earlier request resolved, reverting a saved profile or dropping a merged field. Clearing the session already invalidated in-flight work. The two direct writes now do the same, through one shared helper, so all three behave alike. A read started after such a write takes a fresh token and is unaffected. ## Claude Implementation Notes - frontend/src/lib/userStore.js: Extract invalidateInFlightLoad() and call it from setUser, updateUser and clearUser. updateUser was not in the review finding but carries the identical defect across ~10 call sites, and its partial merge is fully replaced by a stale response; fixing only setUser would have left the bug next door and an asymmetry that reads as intentional. - frontend/src/tests/userStore.test.js: Regression tests for a pending load followed by setUser, and by updateUser, then a late response. Both verified to fail without the fix ('Stale' beating 'Saved' and 'Merged'). - frontend/CLAUDE.md: Note that all three direct writes invalidate, and why. --- frontend/CLAUDE.md | 2 +- frontend/src/lib/userStore.js | 16 ++++++++++--- frontend/src/tests/userStore.test.js | 34 ++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 4 deletions(-) diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index b48794a7..1ccba9f8 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -527,7 +527,7 @@ In production the SPA (Amplify) and the API are on different hosts and `CSRF_COO - **Auto-managed**: Loaded on login, cleared on logout - **Reactive**: Updates reflect immediately in all components using `$userStore` - **30s success-cache TTL** (`USER_CACHE_TTL_MS`): `requireRoleForRoute` calls `loadUser()` on every one of the ~20 role-gated navigations, and in-flight coalescing only covers overlapping calls, so sequential navigation used to refetch `/users/me/` every time. Route guards are a UX gate, not the security boundary (the backend enforces permissions on every request), so a short cache is safe; a role change takes up to 30s to reflect in client-side gating while the backend refuses the data immediately. Only successful loads start the TTL; failures never extend it, and `clearUser()` resets it. **Every other call site passes `{ force: true }`** so its behavior is unchanged: login, wallet switch, profile update, email confirm, journey completion, and the 401/403 recovery path. `performVerification` forwards its own `force` flag, so a forced verify re-reads the profile. -- **In-flight responses are token-guarded.** A forced load does not join an in-flight request (the caller forced it because it just changed server state, so an earlier response would be pre-change); it queues behind that request and issues its own. Every store write, including `lastLoadedAt`, is gated on the load still being the current one, and `clearUser()` drops the token. Without that gate a `/users/me/` response arriving after logout or a wallet switch restores the previous account into the store, which is the same hazard `notificationStore`'s `epoch` counter exists to prevent. +- **In-flight responses are token-guarded.** A forced load does not join an in-flight request (the caller forced it because it just changed server state, so an earlier response would be pre-change); it queues behind that request and issues its own. Every store write, including `lastLoadedAt`, is gated on the load still being the current one. `clearUser()`, `setUser()` and `updateUser()` all drop the token first, because each writes state that is newer than any read already in flight: a session change, or the server's own post-mutation user from a profile save, role join or claim. Without that gate a `/users/me/` response arriving late restores the previous account after logout, or reverts a just-saved profile, which is the same hazard `notificationStore`'s `epoch` counter exists to prevent. A load started *after* one of those writes takes a fresh token and is unaffected. - **Steward hierarchy**: authenticated user payloads expose `steward_tier`; the submission queue defaults a missing tier to reviewer tier 1. ### Components diff --git a/frontend/src/lib/userStore.js b/frontend/src/lib/userStore.js index 6d9f4a3e..f240e983 100644 --- a/frontend/src/lib/userStore.js +++ b/frontend/src/lib/userStore.js @@ -21,6 +21,15 @@ function createUserStore() { // Pass { force: true } wherever state must be re-read immediately. let lastLoadedAt = 0; + // Callers that write the store directly are holding authoritative + // post-mutation state (a profile save, a role join, a claim) or clearing the + // session, so a read that started earlier must not land on top of it. Loads + // started afterwards take a fresh token and are unaffected. + function invalidateInFlightLoad() { + currentLoadToken = null; + loadUserPromise = null; + } + return { subscribe, @@ -108,14 +117,16 @@ function createUserStore() { // Update user data (partial update) updateUser(updates) { + invalidateInFlightLoad(); update(state => ({ ...state, user: state.user ? { ...state.user, ...updates } : null })); }, - + // Set full user data setUser(userData) { + invalidateInFlightLoad(); lastLoadedAt = Date.now(); update(state => ({ ...state, @@ -127,8 +138,7 @@ function createUserStore() { // Clear user data (on logout) clearUser() { lastLoadedAt = 0; - currentLoadToken = null; - loadUserPromise = null; + invalidateInFlightLoad(); set({ user: null, loading: false, diff --git a/frontend/src/tests/userStore.test.js b/frontend/src/tests/userStore.test.js index fb27d497..f4d32649 100644 --- a/frontend/src/tests/userStore.test.js +++ b/frontend/src/tests/userStore.test.js @@ -334,6 +334,40 @@ describe('userStore', () => { expect(get(userStore).user).toEqual(mockUser); }); + it('a late response does not overwrite setUser', async () => { + // setUser carries the server's post-mutation user (profile save, role + // join, claim), so a read that started before it must not win. + let resolveLoad; + getCurrentUser.mockImplementationOnce( + () => new Promise((resolve) => { resolveLoad = resolve; }) + ); + const saved = { ...mockUser, name: 'Saved' }; + const stale = { ...mockUser, name: 'Stale' }; + + const pending = userStore.loadUser(); + userStore.setUser(saved); + resolveLoad(stale); + await pending; + + expect(get(userStore).user).toEqual(saved); + }); + + it('a late response does not overwrite updateUser', async () => { + let resolveLoad; + getCurrentUser.mockImplementationOnce( + () => new Promise((resolve) => { resolveLoad = resolve; }) + ); + userStore.setUser({ ...mockUser, name: 'Before' }); + + // setUser seeds the TTL, so force past it to get a real request in flight. + const pending = userStore.loadUser({ force: true }); + userStore.updateUser({ name: 'Merged' }); + resolveLoad({ ...mockUser, name: 'Stale' }); + await pending; + + expect(get(userStore).user.name).toBe('Merged'); + }); + it('a forced load does not settle for a request that predates it', async () => { // The forced caller has just mutated server state, so joining the // in-flight response would hand back pre-mutation data. From cf6186261ede836b8f38c3dcc32f40477844f862 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 18:04:46 +0200 Subject: [PATCH 12/21] Keep signup-in-progress sessions from reading as signed in Starting a wallet signup with an unregistered address marks the session as not authenticated but deliberately leaves the previous sign-in and address in place. The verification and refresh endpoints had been narrowed to the resolved user alone, so that session read as fully signed in as the previous account: verification never reported the pending signup the client was waiting for, and refresh kept extending a session the user was in the middle of leaving. Journey and waitlist flows also refresh the profile after acting, and several of those refreshes could be served from the new short-lived cache instead of rereading, leaving waitlist, journey and points state a step behind. Those call sites now ask for a fresh read; the navigation guard, which is where the saved requests actually come from, still uses the cache. Auth requests additionally had no way to notice a rejected security token. They are the only users of their own client, so a token rotated in another tab was never cleared and session refresh would keep failing every five minutes until the tab reloaded. ## Claude Implementation Notes - backend/ethereum_auth/views.py: Restore the session 'authenticated' flag alongside request.user.is_authenticated in verify_auth and refresh_session. The pending-signup branch of login() sets it False while leaving _auth_user_id and ethereum_address, so DRF's SessionAuthentication still resolves the old user; the flag is what distinguishes the two. - backend/ethereum_auth/test_authentication.py: Two regression tests for that session shape, verified to fail against the previous gate. - frontend/src/lib/auth.js: Add the authAxios response interceptor mirroring api.js (clear on a real CSRF failure, never retry). Force the post-login and signup-confirm profile reads, both session boundaries. - frontend/src/routes/{BuilderJourney,CommunityJourney,ValidatorWaitlist}.svelte, frontend/src/components/funnel/RoleLanding.svelte, frontend/src/components/social-tasks/SocialTaskCard.svelte: Pass { force: true } on post-mutation refreshes. These call loadUser?.() with optional chaining, which an earlier literal search for loadUser() missed. ValidatorWaitlist's read straight after setUser stays cached on purpose: setUser already wrote the authoritative user. - frontend/src/tests/authSession.test.js: Mock interceptors.response, capture the handler at registration since importAuth clears mock calls, and cover CSRF-403 clearing versus permission-403 not clearing. --- backend/ethereum_auth/test_authentication.py | 32 ++++++++++++ backend/ethereum_auth/views.py | 18 ++++--- .../src/components/funnel/RoleLanding.svelte | 2 +- .../social-tasks/SocialTaskCard.svelte | 2 +- frontend/src/lib/auth.js | 20 ++++++-- frontend/src/routes/BuilderJourney.svelte | 8 +-- frontend/src/routes/CommunityJourney.svelte | 8 +-- frontend/src/routes/ValidatorWaitlist.svelte | 2 +- frontend/src/tests/authSession.test.js | 51 +++++++++++++++++-- 9 files changed, 118 insertions(+), 25 deletions(-) diff --git a/backend/ethereum_auth/test_authentication.py b/backend/ethereum_auth/test_authentication.py index 94add30d..b2308c87 100644 --- a/backend/ethereum_auth/test_authentication.py +++ b/backend/ethereum_auth/test_authentication.py @@ -339,6 +339,38 @@ def test_refresh_succeeds_for_real_session(self): self.assertEqual(response.status_code, 200) self.assertEqual(response.data['message'], 'Session refreshed successfully.') + def test_pending_signup_does_not_report_the_previous_account(self): + """ + Starting a signup with an unregistered wallet sets authenticated=False + but leaves the previous Django login and address in the session, so the + wallet-session flag has to stay part of the gate. Otherwise DRF's + SessionAuthentication resolves the old user and verify reports them as + signed in, hiding the pending-signup branch from the client. + """ + login_wallet_session(self.client, self.user, address=self.user.address) + + session = self.client.session + session['authenticated'] = False + session['pending_wallet_address'] = '0x' + 'a' * 40 + session.save() + + response = self.client.get(VERIFY_URL) + + self.assertEqual(response.status_code, 200) + self.assertFalse(response.data['authenticated']) + self.assertIsNone(response.data['user_id']) + + def test_pending_signup_session_is_not_refreshed(self): + login_wallet_session(self.client, self.user, address=self.user.address) + + session = self.client.session + session['authenticated'] = False + session.save() + + response = self.client.post(REFRESH_URL) + + self.assertEqual(response.status_code, 401) + def test_refresh_rejects_session_without_django_auth_id(self): session = self.client.session session['authenticated'] = True diff --git a/backend/ethereum_auth/views.py b/backend/ethereum_auth/views.py index dfa88792..32d03a63 100644 --- a/backend/ethereum_auth/views.py +++ b/backend/ethereum_auth/views.py @@ -318,11 +318,15 @@ def verify_auth(request): Verify if the user is authenticated. """ ethereum_address = request.session.get('ethereum_address') + authenticated = request.session.get('authenticated', False) # request.user is resolved by EthereumAuthentication, so this reuses that - # lookup instead of querying the user a second time. Requiring the session - # address keeps admin-only sessions reporting unauthenticated, as before. - if ethereum_address and request.user.is_authenticated: + # lookup instead of querying the user a second time. The wallet-session + # flag stays part of the gate: starting a signup with an unregistered + # wallet sets it to False while leaving the previous Django login and + # address in place, and that session must read as unauthenticated so the + # pending-signup branch below runs. + if authenticated and ethereum_address and request.user.is_authenticated: return Response({ 'authenticated': True, 'address': ethereum_address, @@ -354,10 +358,12 @@ def refresh_session(request): Refresh the session to prevent expiration. """ ethereum_address = request.session.get('ethereum_address') + authenticated = request.session.get('authenticated', False) - # Gate on the resolved user, not just the session flag, so a session the - # authenticator rejects stops rolling its own expiry forward every 5 minutes. - if ethereum_address and request.user.is_authenticated: + # Gate on the resolved user as well as the session flag, so a session the + # authenticator rejects stops rolling its own expiry forward every 5 + # minutes, and a session mid-signup with a new wallet is not extended. + if authenticated and ethereum_address and request.user.is_authenticated: # Simply touching the session extends its lifetime request.session.modified = True return Response({'message': 'Session refreshed successfully.'}) diff --git a/frontend/src/components/funnel/RoleLanding.svelte b/frontend/src/components/funnel/RoleLanding.svelte index 0f987ddb..b8d7d97c 100644 --- a/frontend/src/components/funnel/RoleLanding.svelte +++ b/frontend/src/components/funnel/RoleLanding.svelte @@ -82,7 +82,7 @@ if (response.data?.user) { userStore.setUser(response.data.user); } else { - userStore.loadUser?.()?.catch(() => {}); + userStore.loadUser?.({ force: true })?.catch(() => {}); } markFunnelTime(`journey_start:${role}`); markLifecycleTime(`first_journey_start:${role}`); diff --git a/frontend/src/components/social-tasks/SocialTaskCard.svelte b/frontend/src/components/social-tasks/SocialTaskCard.svelte index cbee4dda..1f7cb85a 100644 --- a/frontend/src/components/social-tasks/SocialTaskCard.svelte +++ b/frontend/src/components/social-tasks/SocialTaskCard.svelte @@ -183,7 +183,7 @@ if (updatedUser) { userStore.updateUser(updatedUser); } else { - userStore.loadUser?.(); + userStore.loadUser?.({ force: true }); } } diff --git a/frontend/src/lib/auth.js b/frontend/src/lib/auth.js index a5980329..e95c8a4e 100644 --- a/frontend/src/lib/auth.js +++ b/frontend/src/lib/auth.js @@ -3,7 +3,7 @@ import axios from 'axios'; import { writable } from 'svelte/store'; import { userStore } from './userStore'; import { API_BASE_URL } from './config.js'; -import { attachCsrfToken, clearCsrfToken } from './csrf.js'; +import { attachCsrfToken, clearCsrfToken, isCsrfFailure } from './csrf.js'; import { detectCategoryFromRoute } from '../stores/category.js'; import { roleForCategory } from './roleState.js'; @@ -137,6 +137,20 @@ authAxios.interceptors.request.use( (error) => Promise.reject(error) ); +// Mirrors the api.js interceptor. Without it a token rotated by another tab +// would keep being sent from this one: the auth endpoints are the only callers +// here, so nothing else would ever clear it and session refresh would fail +// every five minutes until reload. Not retried, for the same reason as api.js. +authAxios.interceptors.response.use( + (response) => response, + (error) => { + if (isCsrfFailure(error)) { + clearCsrfToken(); + } + return Promise.reject(error); + } +); + // Authentication API endpoints (relative to base URL, not api/v1) const API_ENDPOINTS = { NONCE: `${API_BASE_URL}/api/auth/nonce/`, @@ -387,7 +401,7 @@ export async function signInWithEthereum(provider = null, walletName = 'wallet', // Load user data into the store let userData = null; try { - userData = await userStore.loadUser(); + userData = await userStore.loadUser({ force: true }); } catch (err) { // Silently handle user data load failure } @@ -601,7 +615,7 @@ export async function confirmPendingSignupEmail(credential) { clearCsrfToken(); authState.setAuthenticated(true, response.data.address); try { - await userStore.loadUser(); + await userStore.loadUser({ force: true }); } catch (err) { // Silently handle user data load failure } diff --git a/frontend/src/routes/BuilderJourney.svelte b/frontend/src/routes/BuilderJourney.svelte index 8a9643af..5974fab0 100644 --- a/frontend/src/routes/BuilderJourney.svelte +++ b/frontend/src/routes/BuilderJourney.svelte @@ -262,7 +262,7 @@ .startBuilderJourney() .then((res) => { if (res.data?.user) userStore.updateUser(res.data.user); - else userStore.loadUser?.(); + else userStore.loadUser?.({ force: true }); markFunnelTime('journey_start:builder'); markLifecycleTime('first_journey_start:builder'); trackEvent('journey_started', getAnalyticsContext({ @@ -589,7 +589,7 @@ try { const res = await journeyAPI.linkGithubAccount(); if (res.data?.user) userStore.updateUser(res.data.user); - else await userStore.loadUser?.(); + else await userStore.loadUser?.({ force: true }); trackBuilderStepEvent('journey_step_verified', 'github'); showSuccess('GitHub linked. 25 BP awarded.'); } catch (err) { @@ -605,7 +605,7 @@ function handleGithubLinked(updatedUser) { if (updatedUser) userStore.updateUser(updatedUser); - else userStore.loadUser?.(); + else userStore.loadUser?.({ force: true }); } function handleTaskCompleted(result) { @@ -619,7 +619,7 @@ : task ); loadTasks({ showLoading: false }); - userStore.loadUser?.(); + userStore.loadUser?.({ force: true }); } function triggerWalletConnect() { diff --git a/frontend/src/routes/CommunityJourney.svelte b/frontend/src/routes/CommunityJourney.svelte index 4367e1cb..a4716eec 100644 --- a/frontend/src/routes/CommunityJourney.svelte +++ b/frontend/src/routes/CommunityJourney.svelte @@ -174,7 +174,7 @@ time_from_wallet_auth_success_ms: getFunnelDurationMs('wallet_auth_success'), time_from_profile_completion_ms: getFunnelDurationMs('profile_completion'), })); - userStore.loadUser?.(); + userStore.loadUser?.({ force: true }); }) .catch((err) => { trackEvent('journey_start_error', getAnalyticsContext({ @@ -339,7 +339,7 @@ try { const res = isX ? await journeyAPI.linkXAccount() : await journeyAPI.linkDiscordAccount(); if (res.data?.user) userStore.updateUser(res.data.user); - else await userStore.loadUser?.(); + else await userStore.loadUser?.({ force: true }); trackCommunityStepEvent('journey_step_verified', stepId); markStepDone(stepId); showSuccess(isX ? 'X account linked for community points.' : 'Discord account linked for community points.'); @@ -379,7 +379,7 @@ : task ); loadJourney({ showLoading: false }); - userStore.loadUser?.(); + userStore.loadUser?.({ force: true }); } async function copyShareText() { @@ -494,7 +494,7 @@ try { const res = await journeyAPI.completeCommunityJourney(); if (res.data?.user) userStore.updateUser(res.data.user); - await userStore.loadUser?.(); + await userStore.loadUser?.({ force: true }); markLifecycleTime('role_unlocked:community'); trackEvent('community_role_claim_success', getAnalyticsContext(claimParams)); trackEvent('journey_completed', getAnalyticsContext({ diff --git a/frontend/src/routes/ValidatorWaitlist.svelte b/frontend/src/routes/ValidatorWaitlist.svelte index c724104c..599474a9 100644 --- a/frontend/src/routes/ValidatorWaitlist.svelte +++ b/frontend/src/routes/ValidatorWaitlist.svelte @@ -282,7 +282,7 @@ } // Joined server-side. The user refresh and success-banner write are both // best-effort and must not block (or undo) the success redirect. - userStore.loadUser?.()?.catch(() => {}); + userStore.loadUser?.({ force: true })?.catch(() => {}); markLifecycleTime('validator_waitlist_joined'); trackEvent('validator_waitlist_joined', getAnalyticsContext({ role_context: 'validator', diff --git a/frontend/src/tests/authSession.test.js b/frontend/src/tests/authSession.test.js index 42071a1c..e47b4d4a 100644 --- a/frontend/src/tests/authSession.test.js +++ b/frontend/src/tests/authSession.test.js @@ -1,10 +1,21 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -const mocks = vi.hoisted(() => ({ - get: vi.fn(), - post: vi.fn(), - requestUse: vi.fn(), -})); +const mocks = vi.hoisted(() => { + const store = { + get: vi.fn(), + post: vi.fn(), + requestUse: vi.fn(), + clearCsrfToken: vi.fn(), + isCsrfFailure: vi.fn(), + // Captured at registration: importAuth() runs vi.clearAllMocks(), which + // would otherwise wipe the recorded interceptor arguments. + capturedResponseRejected: null, + }; + store.responseUse = vi.fn((onFulfilled, onRejected) => { + store.capturedResponseRejected = onRejected; + }); + return store; +}); vi.mock('axios', () => ({ default: { @@ -13,6 +24,7 @@ vi.mock('axios', () => ({ post: mocks.post, interceptors: { request: { use: mocks.requestUse }, + response: { use: mocks.responseUse }, }, })), }, @@ -24,6 +36,8 @@ vi.mock('../lib/config.js', () => ({ vi.mock('../lib/csrf.js', () => ({ attachCsrfToken: vi.fn((config) => config), + clearCsrfToken: mocks.clearCsrfToken, + isCsrfFailure: mocks.isCsrfFailure, })); vi.mock('../lib/userStore.js', () => ({ @@ -175,6 +189,33 @@ describe('auth session refresh', () => { expect(mocks.get).toHaveBeenCalledTimes(2); }); + // The auth endpoints are the only callers on authAxios, so if a CSRF failure + // there did not clear the cached token nothing else ever would, and the + // 5-minute session refresh would keep failing until the tab reloaded. + it('clears the cached CSRF token when an auth request is rejected for CSRF', async () => { + await importAuth(); + const onRejected = mocks.capturedResponseRejected; + const csrfError = { response: { status: 403, data: { detail: 'CSRF Failed: x' } } }; + mocks.isCsrfFailure.mockReturnValue(true); + + await expect(onRejected(csrfError)).rejects.toBe(csrfError); + + expect(mocks.clearCsrfToken).toHaveBeenCalledTimes(1); + }); + + it('leaves the cached CSRF token alone for a permission rejection', async () => { + await importAuth(); + const onRejected = mocks.capturedResponseRejected; + const permissionError = { + response: { status: 403, data: { detail: 'You do not have permission.' } }, + }; + mocks.isCsrfFailure.mockReturnValue(false); + + await expect(onRejected(permissionError)).rejects.toBe(permissionError); + + expect(mocks.clearCsrfToken).not.toHaveBeenCalled(); + }); + it('still logs out on a definitive rejection', async () => { const { verifyAuth, authState } = await importAuth(); const authError = new Error('gone'); From 45090ec169f54fba69cec5e1a022f637c8548485 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Wed, 29 Jul 2026 20:30:02 +0200 Subject: [PATCH 13/21] Assert the pending-signup response and catch best-effort profile refreshes The pending-signup regression test only set the session key, without the signup record it points at, so the endpoint took the "no pending signup" path and the test never checked the response the client actually depends on. It now creates the record and asserts the pending signup and its address, which is the half of the contract the fix was really about. The fire-and-forget profile refreshes in the journey and task flows also had no rejection handler, so a failed refresh surfaced as an unhandled rejection rather than being quietly ignored as intended. ## Claude Implementation Notes - backend/ethereum_auth/test_authentication.py: Create an active PendingWalletSignup and set pending_wallet_signup_id, then assert pending_signup is True and address is the pending one. get_pending_signup_from_session() filters on STATUS_PENDING and is_active(), so the session key alone resolved to None. Re-verified by reverting the gate in place: both tests fail (True is not false, 200 != 401) and pass with it. - frontend/src/routes/{BuilderJourney,CommunityJourney}.svelte, frontend/src/components/social-tasks/SocialTaskCard.svelte: Add ?.catch(() => {}) to the six non-awaited refreshes, matching the existing convention in RoleLanding and ValidatorWaitlist. The three awaited calls are inside try/catch blocks and are already handled, so they are left alone. --- backend/ethereum_auth/test_authentication.py | 18 +++++++++++++++++- .../social-tasks/SocialTaskCard.svelte | 2 +- frontend/src/routes/BuilderJourney.svelte | 6 +++--- frontend/src/routes/CommunityJourney.svelte | 4 ++-- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/backend/ethereum_auth/test_authentication.py b/backend/ethereum_auth/test_authentication.py index b2308c87..d88d6f44 100644 --- a/backend/ethereum_auth/test_authentication.py +++ b/backend/ethereum_auth/test_authentication.py @@ -7,12 +7,16 @@ behaviour and the query shape. """ +from datetime import timedelta + from django.contrib.auth import get_user_model from django.db import connection from django.test import TestCase from django.test.utils import CaptureQueriesContext +from django.utils import timezone from rest_framework.test import APIClient +from .models import PendingWalletSignup from .testing import login_wallet_session @@ -349,9 +353,17 @@ def test_pending_signup_does_not_report_the_previous_account(self): """ login_wallet_session(self.client, self.user, address=self.user.address) + pending_address = '0x' + 'a' * 40 session = self.client.session + pending = PendingWalletSignup.objects.create( + address=pending_address, + session_key=session.session_key or '', + status=PendingWalletSignup.STATUS_PENDING, + expires_at=timezone.now() + timedelta(minutes=10), + ) session['authenticated'] = False - session['pending_wallet_address'] = '0x' + 'a' * 40 + session['pending_wallet_signup_id'] = pending.id + session['pending_wallet_address'] = pending_address session.save() response = self.client.get(VERIFY_URL) @@ -359,6 +371,10 @@ def test_pending_signup_does_not_report_the_previous_account(self): self.assertEqual(response.status_code, 200) self.assertFalse(response.data['authenticated']) self.assertIsNone(response.data['user_id']) + # The point of the flag: the client needs the pending-signup branch, + # which never runs if the previous account reads as still signed in. + self.assertTrue(response.data['pending_signup']) + self.assertEqual(response.data['address'], pending_address) def test_pending_signup_session_is_not_refreshed(self): login_wallet_session(self.client, self.user, address=self.user.address) diff --git a/frontend/src/components/social-tasks/SocialTaskCard.svelte b/frontend/src/components/social-tasks/SocialTaskCard.svelte index 1f7cb85a..34c69fef 100644 --- a/frontend/src/components/social-tasks/SocialTaskCard.svelte +++ b/frontend/src/components/social-tasks/SocialTaskCard.svelte @@ -183,7 +183,7 @@ if (updatedUser) { userStore.updateUser(updatedUser); } else { - userStore.loadUser?.({ force: true }); + userStore.loadUser?.({ force: true })?.catch(() => {}); } } diff --git a/frontend/src/routes/BuilderJourney.svelte b/frontend/src/routes/BuilderJourney.svelte index 5974fab0..e2e5460e 100644 --- a/frontend/src/routes/BuilderJourney.svelte +++ b/frontend/src/routes/BuilderJourney.svelte @@ -262,7 +262,7 @@ .startBuilderJourney() .then((res) => { if (res.data?.user) userStore.updateUser(res.data.user); - else userStore.loadUser?.({ force: true }); + else userStore.loadUser?.({ force: true })?.catch(() => {}); markFunnelTime('journey_start:builder'); markLifecycleTime('first_journey_start:builder'); trackEvent('journey_started', getAnalyticsContext({ @@ -605,7 +605,7 @@ function handleGithubLinked(updatedUser) { if (updatedUser) userStore.updateUser(updatedUser); - else userStore.loadUser?.({ force: true }); + else userStore.loadUser?.({ force: true })?.catch(() => {}); } function handleTaskCompleted(result) { @@ -619,7 +619,7 @@ : task ); loadTasks({ showLoading: false }); - userStore.loadUser?.({ force: true }); + userStore.loadUser?.({ force: true })?.catch(() => {}); } function triggerWalletConnect() { diff --git a/frontend/src/routes/CommunityJourney.svelte b/frontend/src/routes/CommunityJourney.svelte index a4716eec..5523f802 100644 --- a/frontend/src/routes/CommunityJourney.svelte +++ b/frontend/src/routes/CommunityJourney.svelte @@ -174,7 +174,7 @@ time_from_wallet_auth_success_ms: getFunnelDurationMs('wallet_auth_success'), time_from_profile_completion_ms: getFunnelDurationMs('profile_completion'), })); - userStore.loadUser?.({ force: true }); + userStore.loadUser?.({ force: true })?.catch(() => {}); }) .catch((err) => { trackEvent('journey_start_error', getAnalyticsContext({ @@ -379,7 +379,7 @@ : task ); loadJourney({ showLoading: false }); - userStore.loadUser?.({ force: true }); + userStore.loadUser?.({ force: true })?.catch(() => {}); } async function copyShareText() { From 0cd7e5f3bcd9512da4de5dcdef1c8e9b6b3e3d43 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iva=CC=81n=20Raskovsky?= Date: Thu, 30 Jul 2026 12:37:08 +0200 Subject: [PATCH 14/21] Let validators link Telegram support groups through the Deckard bot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validators can now connect Telegram groups to their validator from the portal. A new "Telegram Support" page in the validator area issues one-time group bind codes: the validator creates a Telegram group, adds the Deckard support bot, and runs /bindcode there; the bot redeems the code server-to-server and the group is bound to the validator. A validator can bind multiple groups (one code per group), codes expire after 48 hours, and unredeemed codes can be revoked. The bind code follows the service-account token design: only a SHA-256 digest is stored, lookup is by a non-secret identifier embedded in the code, and redemption compares digests in constant time. Redemption is gated by a new telegram_bind:redeem service-account scope reserved for the Deckard bot and is atomic and single-use: it records the bound group chat id and the redeeming Telegram uid, and upserts a new TelegramConnection (social connection pattern, bot-confirmed instead of OAuth) so the portal account carries a verifiable Telegram identity. Issuance is throttled per user; the plaintext code is returned exactly once and never listed again. ## Claude Implementation Notes - backend/validators/models.py: TelegramGroupBindCode model — identifier + digest storage (hash_code/identifier_from_plaintext/issue mirroring ServiceAccountToken), issued/redeemed/expired/revoked statuses, 48h default TTL, lazy effective_status, is_redeemable helper - backend/validators/migrations/0018_telegramgroupbindcode.py: schema - backend/validators/views.py: TelegramBindCodeViewSet (create issues + returns plaintext once, mine lists metadata, revoke with row lock and 409 for redeemed codes) and TelegramBindCodeRedeemView (service account auth, required_scopes {'*': telegram_bind:redeem}, payload validation, select_for_update single-use redemption, constant-time digest compare, TelegramConnection upsert that never blanks an existing username) - backend/validators/serializers.py: TelegramBindCodeSerializer — metadata only, status is the lazy-expiry effective status - backend/validators/urls.py: explicit redeem path before the router so the viewset detail route never swallows it; telegram-bind-codes registered before the catch-all validator route - backend/social_connections/models.py: TelegramConnection subclass of the SocialConnection abstract base (numeric uid identity, display-only username, no OAuth), db_table social_connections_telegram - backend/social_connections/migrations/0008_telegramconnection.py: schema - backend/social_connections/admin.py: TelegramConnection admin - backend/service_accounts/scopes.py: TELEGRAM_BIND_REDEEM_SCOPE added to ALLOWED_SERVICE_ACCOUNT_SCOPES so admin/command token issuance accepts it - backend/utils/throttling.py + backend/tally/settings.py: TelegramBindCodeIssueRateThrottle, telegram_bind_issue 10/hour - backend/validators/tests/test_telegram_bind_codes.py: 22 tests — issuance auth + validator gate + single-shot plaintext, mine isolation and lazy expiry, revoke rules, redeem scope gate, single-use, expiry, invalid/tampered codes, connection upsert - frontend/src/routes/ValidatorTelegram.svelte: Link-a-Telegram-group page (Svelte 5 runes) — generate code, show once with /bindcode copy command, list codes with status chips and revoke - frontend/src/App.svelte: /validators/telegram role-gated route - frontend/src/components/Sidebar.svelte: Telegram Support link in both validator sections (role-locked pattern) - frontend/src/lib/api.js: issue/list/revoke bind-code helpers - frontend/src/lib/config.js: DECKARD_BOT_USERNAME from VITE_DECKARD_BOT_USERNAME with generic-wording fallback - backend/CLAUDE.md + frontend/CLAUDE.md: documented the new model, endpoints, scope, route, and API helpers --- backend/CLAUDE.md | 14 +- backend/service_accounts/scopes.py | 8 +- backend/social_connections/admin.py | 8 + .../migrations/0008_telegramconnection.py | 35 ++ backend/social_connections/models.py | 17 + backend/tally/settings.py | 2 + backend/utils/throttling.py | 9 + .../migrations/0018_telegramgroupbindcode.py | 36 ++ backend/validators/models.py | 114 ++++++ backend/validators/serializers.py | 27 +- .../tests/test_telegram_bind_codes.py | 331 ++++++++++++++++++ backend/validators/urls.py | 17 +- backend/validators/views.py | 216 +++++++++++- frontend/CLAUDE.md | 2 + frontend/src/App.svelte | 2 + frontend/src/components/Sidebar.svelte | 26 ++ frontend/src/lib/api.js | 7 +- frontend/src/lib/config.js | 4 + frontend/src/routes/ValidatorTelegram.svelte | 249 +++++++++++++ 19 files changed, 1116 insertions(+), 8 deletions(-) create mode 100644 backend/social_connections/migrations/0008_telegramconnection.py create mode 100644 backend/validators/migrations/0018_telegramgroupbindcode.py create mode 100644 backend/validators/tests/test_telegram_bind_codes.py create mode 100644 frontend/src/routes/ValidatorTelegram.svelte diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 3c16f8f0..a324158b 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -27,7 +27,7 @@ backend/ ├── api/ # Core API app ├── contributions/ # Contribution tracking ├── leaderboard/ # Leaderboard and rankings -├── social_connections/ # OAuth (GitHub, Twitter, Discord) + encrypted token storage +├── social_connections/ # OAuth (GitHub, Twitter, Discord) + encrypted token storage + Telegram (bot-confirmed, no OAuth) ├── social_tasks/ # Repeatable social tasks (follow, join, like) and completions ├── users/ # User management and auth ├── partners/ # Ecosystem partners directory @@ -118,6 +118,7 @@ backend/ - **Models**: `ServiceAccount` (name, description, is_active; acts as the DRF request principal; it is NOT a User and can never become a session) and `ServiceAccountToken` (unique non-secret `identifier`; unique SHA-256 `digest` of the plaintext, which is never stored; `scopes` list; `expires_at`/`revoked_at`/`last_used_at`, with `last_used_at` writes throttled to once per minute). - **Auth class**: `service_accounts.authentication.ServiceAccountAuthentication` parses `Bearer sa__`, looks up the token by non-secret id, compares digests in constant time, rejects expired/revoked/inactive with a generic 401. Non-`sa_` credentials pass through to other authenticators. - **Permission**: `service_accounts.permissions.HasServiceAccountScope`; views declare `required_scopes = {'': '', '*': ''}`. +- **Scopes** (`service_accounts/scopes.py`): `ai_review:read`, `ai_review:propose` (AI review agent), `telegram_bind:redeem` (Deckard Telegram bot redeeming validator group bind codes). New scopes must be added to `ALLOWED_SERVICE_ACCOUNT_SCOPES` or admin token issuance rejects them. - **Issue a token**: Django admin Service account change page -> "Issue token", or `python manage.py issue_service_account_token --scopes ... [--expires-days N]`; plaintext is shown exactly once. Rotate = issue new + revoke old (admin action on Service account tokens); kill switch = deactivate the account. - **Tests**: `service_accounts/tests/`; test helper `service_accounts.testing.service_account_auth_headers()` returns client auth kwargs. @@ -295,6 +296,7 @@ backend/ - ValidatorWalletStatusSnapshot - Daily wallet rollup. On-chain `status` (owned by the on-chain sync, for uptime lookback) PLUS the latched observability verdict written by the Grafana sync: `metrics_status` / `logs_status` / `version_status`, `metrics_samples` / `logs_samples` counters, and `node_version`. **Metrics and logs latch pessimistically** (worst-of-day: shame at ANY observation → the day is shame). **Version latches optimistically** (best-of-day: a single up-to-date observation → the day is OK, since once a node upgrades that day an earlier stale reading must not shame it; `on` > `warning` > `shame`). A day is "clean" only if `status=='active'` and both sample counters are ≥1 and neither metrics nor logs is `shame` and version is not `shame`. The two syncs write disjoint columns (bulk_create update_conflicts on `(wallet, date)`), so neither clobbers the other. - ValidatorWalletObservation - Append-only raw log; one row per active wallet per Grafana sync run (`observed_at`, `onchain_status`, `metrics_status`, `logs_status`, `version_status`, `node_version`). Source of truth the daily rollup is materialised from and rebuildable via `rebuild_daily_snapshots`. - SyncLock - Database-backed sync coordination row with owner token for cross-worker locking + - TelegramGroupBindCode - One-time code binding a Telegram group to a validator via the Deckard support bot. Plaintext (`tgb__`) is returned exactly once at issuance; only the SHA-256 digest is stored (identifier lookup + constant-time compare, mirroring ServiceAccountToken). 48h expiry (lazy: `effective_status`), statuses issued/redeemed/expired/revoked, multiple active codes per validator (one group per code). Redemption records `redeemed_group_chat_id` + `redeemed_by_telegram_uid` and upserts the issuing user's `social_connections.TelegramConnection` (numeric Telegram uid = identity, username display-only, no OAuth tokens). - **Services**: `validators/grafana_service.py` - GrafanaValidatorStatusService - Polls Grafana Cloud (`/api/ds/query`) Prometheus + Loki datasources and updates `ValidatorWallet.metrics_status` / `logs_status` for `status='active'` wallets, per network. The Prometheus query also reads the `version` label from `genlayer_node_info` — **normalised at ingest** in `parse_response` ('v' prefix stripped, capped to the 50-char column; when a node briefly reports two version series right after an upgrade, the higher parseable one wins). Each run writes a `ValidatorWalletObservation` and latches today's `ValidatorWalletStatusSnapshot` rollup (`_record_history`, best-effort — never breaks the live status sync). Observations are retained forever by explicit decision — no pruning in points. Used by the Wall of Shame cron. - GrafanaValidatorStatusService is also the **source of truth for node versions** (`_sync_node_versions`, best-effort, runs before the active-wallet early return so networks with zero active wallets are still covered): version detection covers **every reporting node on the network regardless of on-chain status** (a quarantined node can still record its upgrade), **except banned wallets**, and only counts versions observed on wallets known to the DB and linked to an operator — the `version` label is self-reported by the node being judged and rewarded, so unknown Prometheus series count for nothing. Only versions that are both semver-valid AND PEP 440-parseable drive comparisons (e.g. `0.6.0-genlayer.1` is excluded — `packaging` can't parse it; in the shame loop an unparseable observed version or an unparseable active target yields `version_status='unknown'`, never a lexicographic fallback verdict). It auto-creates a `TargetNodeVersion` when a STABLE release (bare `x.y.z`, no pre-release/build) higher than the active target is reported by **at least `NODE_VERSION_MIN_OPERATORS_FOR_AUTO_TARGET` (default 1: the first adopter creates the target) distinct operators** (`target_date=now`; an unparseable active target is never blindly superseded; a broadcast notification is emitted via `broadcast_target_node_version`), raises each linked operator's `node_version_` to their highest observed version via a direct `.update()` (**monotonic** — a wallet skipping a scrape cycle can't transiently downgrade the field; genuine downgrades need admin correction), and directly awards an already-approved `node-upgrade` Contribution (`_award_node_upgrade`, early-bonus 4/3/2/1) when a visible operator first reaches the active target. **Removing the node-upgrade multiplier pauses the auto-award** (it is skipped with a warning, not created at 1.0). The per-operator loop is individually fault-isolated — one operator's failure never blocks the rest. Dedup shares the exact `version {v} [{network}]` notes key with the old manual flow so nothing double-awards. A run where a whole datasource comes back empty (no Prometheus series or no Loki counts) still updates live wallet statuses (they self-heal) but **skips the permanent history latch** — a datasource blackout must not shame every validator's recorded day. @@ -306,6 +308,10 @@ backend/ - `/api/v1/validators/wallets/sync/` - POST cron-protected background sync trigger with DB-backed lock (on-chain validator sync) - `/api/v1/validators/wallets/sync-grafana/` - POST cron-protected background sync trigger for Grafana observability cross-check (separate SyncLock row `grafana_status_sync` so it can run alongside the on-chain sync) - `/api/v1/validators/wallets/wall-of-shame/` - Public read-only endpoint listing active validator wallets with `metrics_status` / `logs_status`. SHAME rows sort first. Cached 60s. Optional `?network=asimov|bradbury` filter. Each wallet also carries `clean_streak_days` + `clean_streak_broken_by` (consecutive not-shamed days for that node, from `validators/streaks.py` over the daily rollup). The grouped `validators` output adds `network_streaks` — per-operator-per-network any-node-clean streaks (a network-day is clean if ≥1 of the operator's nodes was clean) — plus per-node `clean_streak_days` on each `networks` entry. Streaks start accumulating at deploy (history wasn't recorded before). Days with no Grafana data while the node was active (sync outage, pre-history) are SKIPPED — they neither count nor break, so an infra failure on our side never resets streaks; days spent non-active per the on-chain sync break the streak with `broken_by: ['status']`. + - `/api/v1/validators/telegram-bind-codes/` - POST (auth, validator profile required, `telegram_bind_issue` throttle 10/hour) issues a bind code and returns the plaintext ONCE + - `/api/v1/validators/telegram-bind-codes/mine/` - GET the current user's codes (metadata only, never raw codes) + - `/api/v1/validators/telegram-bind-codes/{id}/revoke/` - POST owner-only revoke of an unredeemed code (409 if already redeemed) + - `/api/v1/validators/telegram-bind-codes/redeem/` - POST for the Deckard Telegram bot only: service account bearer token with scope `telegram_bind:redeem`. Body `{code, group_chat_id, telegram_uid, telegram_username?}`. Single-use atomic redemption: marks the code, records group + uid, upserts TelegramConnection. Errors carry a machine `code`: `invalid_request` (400), `invalid_code` (404), `already_redeemed`/`revoked` (409), `expired` (410). DM-vs-group is enforced bot-side. - `/api/v1/validators/wallets/grafana/` - Public minimal roster for the Grafana Infinity datasource (`GrafanaValidatorSerializer`). Flat array, one row per wallet across ALL statuses; fields: `network` (Grafana label value e.g. `asimov-phase5`), `node` (on-chain validator address == Prometheus `genlayer_node_info` `node` label, lowercased), `name`, `status`, `operator`, `account`/`account_name` (only for visible operators), `explorer_url`, plus **raw link/identity facts** (verdicts are computed dashboard-side, NOT here): `linked` (bool — wallet attributed to a portal account; a bare fact, safe for non-visible operators), `moniker` and `logo_uri` (raw synced `getIdentity()` values, empty string = unset), `has_description` (presence bool so the roster doesn't ship long texts). Excludes observability/shame fields by design. Cached 60s. Optional `?network=asimov|bradbury` filter. - The roster **also appends one synthetic `status='missing'` row per network** for every graduated portal validator (visible Validator role user) with no wallet linked on that network (`_missing_graduated_rows` in the view) — graduated validators are expected on every testnet, and an absent one otherwise has no row for dashboards to show. On these rows `node` = the account address (unique join key only — matches no metric series), `operator` is null, and the link/identity facts (`linked`/`moniker`/`logo_uri`/`has_description`) are **null** (no wallet to describe — distinct from `false`/empty on a real wallet). A wallet of ANY status (incl. `inactive`) suppresses the missing row; an unlinked wallet (`operator=None`) does not. @@ -487,6 +493,12 @@ GET /api/v1/leaderboard/user_stats/by-address/{address}/ (requires auth) GET /api/v1/multipliers/ (requires auth) GET /api/v1/multiplier-periods/ +# Validators - Telegram group bind codes (Deckard support bot) +POST /api/v1/validators/telegram-bind-codes/ (requires auth + validator profile; returns plaintext code ONCE; throttled 10/hour) +GET /api/v1/validators/telegram-bind-codes/mine/ (requires auth; metadata only) +POST /api/v1/validators/telegram-bind-codes/{id}/revoke/ (requires auth, owner-only) +POST /api/v1/validators/telegram-bind-codes/redeem/ (service account bearer token, scope telegram_bind:redeem) + # Validators - Wall of Shame POST /api/v1/validators/wallets/sync-grafana/ (cron-protected, X-Cron-Token, background) GET /api/v1/validators/wallets/wall-of-shame/ (public, cached 60s, ?network= filter) diff --git a/backend/service_accounts/scopes.py b/backend/service_accounts/scopes.py index f8627c6f..c1c3db73 100644 --- a/backend/service_accounts/scopes.py +++ b/backend/service_accounts/scopes.py @@ -6,4 +6,10 @@ AI_REVIEW_PROPOSE_SCOPE, ) -ALLOWED_SERVICE_ACCOUNT_SCOPES = frozenset(AI_REVIEW_SCOPES) +# Granted to the Deckard Telegram support bot so it can redeem validator +# Telegram group bind codes (validators app). +TELEGRAM_BIND_REDEEM_SCOPE = 'telegram_bind:redeem' + +ALLOWED_SERVICE_ACCOUNT_SCOPES = frozenset( + AI_REVIEW_SCOPES + (TELEGRAM_BIND_REDEEM_SCOPE,) +) diff --git a/backend/social_connections/admin.py b/backend/social_connections/admin.py index 517e558b..0dcd0000 100644 --- a/backend/social_connections/admin.py +++ b/backend/social_connections/admin.py @@ -12,6 +12,7 @@ DiscordRoleSyncLock, GitHubConnection, PendingOAuthState, + TelegramConnection, TwitterConnection, UsedOAuthCode, ) @@ -68,6 +69,13 @@ class TwitterConnectionAdmin(admin.ModelAdmin): readonly_fields = ('platform_user_id', 'access_token', 'refresh_token', 'linked_at', 'created_at', 'updated_at') +@admin.register(TelegramConnection) +class TelegramConnectionAdmin(admin.ModelAdmin): + list_display = ('user', 'platform_username', 'platform_user_id', 'linked_at') + search_fields = ('user__email', 'platform_username', 'platform_user_id') + readonly_fields = ('platform_user_id', 'linked_at', 'created_at', 'updated_at') + + @admin.register(DiscordConnection) class DiscordConnectionAdmin(admin.ModelAdmin): list_display = ( diff --git a/backend/social_connections/migrations/0008_telegramconnection.py b/backend/social_connections/migrations/0008_telegramconnection.py new file mode 100644 index 00000000..5f31fac3 --- /dev/null +++ b/backend/social_connections/migrations/0008_telegramconnection.py @@ -0,0 +1,35 @@ +# Generated by Django 5.2.10 on 2026-07-30 10:18 + +import django.db.models.deletion +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('social_connections', '0007_alter_discordearnedroleassignment_options'), + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name='TelegramConnection', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('platform_user_id', models.CharField(help_text="Platform's unique user ID", max_length=100)), + ('platform_username', models.CharField(help_text='Username on the platform', max_length=100)), + ('access_token', models.TextField(blank=True, help_text='Encrypted OAuth access token')), + ('refresh_token', models.TextField(blank=True, help_text='Encrypted OAuth refresh token')), + ('linked_at', models.DateTimeField(help_text='When the account was linked')), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='%(class)s', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'verbose_name': 'Telegram Connection', + 'verbose_name_plural': 'Telegram Connections', + 'db_table': 'social_connections_telegram', + }, + ), + ] diff --git a/backend/social_connections/models.py b/backend/social_connections/models.py index 43f61266..8801d569 100644 --- a/backend/social_connections/models.py +++ b/backend/social_connections/models.py @@ -48,6 +48,23 @@ class Meta: verbose_name_plural = 'Twitter Connections' +class TelegramConnection(SocialConnection): + """Telegram identity for a portal account. + + Telegram has no OAuth flow here: the connection is created/confirmed when + the Deckard support bot redeems a Telegram group bind code issued by this + user (see validators.TelegramGroupBindCode). `platform_user_id` holds the + numeric Telegram user id (the stable identity); `platform_username` is the + display-only @username, which Telegram users can change at any time. The + inherited OAuth token fields stay empty. + """ + + class Meta: + db_table = 'social_connections_telegram' + verbose_name = 'Telegram Connection' + verbose_name_plural = 'Telegram Connections' + + class DiscordConnection(SocialConnection): discriminator = models.CharField(max_length=10, blank=True, help_text="Discord discriminator (legacy)") avatar_hash = models.CharField(max_length=100, blank=True, help_text="Discord avatar hash for CDN URL") diff --git a/backend/tally/settings.py b/backend/tally/settings.py index 4da3eb72..7c7bd64d 100644 --- a/backend/tally/settings.py +++ b/backend/tally/settings.py @@ -221,6 +221,8 @@ def get_required_env(key): 'public_leaderboard': '300/minute', # Wallet linking is a one-time action per validator 'wallet_link': '10/hour', + # Telegram group bind codes are live secrets for 48h; bound hoarding + 'telegram_bind_issue': '10/hour', 'pending_email_start': '10/hour', 'pending_email_resend': '10/hour', 'pending_email_confirm': '30/hour', diff --git a/backend/utils/throttling.py b/backend/utils/throttling.py index d274590f..2c1d569b 100644 --- a/backend/utils/throttling.py +++ b/backend/utils/throttling.py @@ -18,6 +18,15 @@ class WalletLinkRateThrottle(UserRateThrottle): scope = 'wallet_link' +class TelegramBindCodeIssueRateThrottle(UserRateThrottle): + """ + Per-user throttle for issuing Telegram group bind codes. Codes are cheap + to mint but each is a live secret for 48h; a tight rate bounds hoarding + and keeps the active-code surface small. + """ + scope = 'telegram_bind_issue' + + class PendingEmailStartRateThrottle(AnonRateThrottle): scope = 'pending_email_start' diff --git a/backend/validators/migrations/0018_telegramgroupbindcode.py b/backend/validators/migrations/0018_telegramgroupbindcode.py new file mode 100644 index 00000000..8b853c1f --- /dev/null +++ b/backend/validators/migrations/0018_telegramgroupbindcode.py @@ -0,0 +1,36 @@ +# Generated by Django 5.2.10 on 2026-07-30 10:18 + +import django.db.models.deletion +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('validators', '0017_validatoroperatorwallet'), + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name='TelegramGroupBindCode', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('identifier', models.CharField(help_text='Non-secret lookup id embedded in the code', max_length=16, unique=True)), + ('digest', models.CharField(help_text='SHA-256 digest of the plaintext code (plaintext is never stored)', max_length=64, unique=True)), + ('status', models.CharField(choices=[('issued', 'Issued'), ('redeemed', 'Redeemed'), ('expired', 'Expired'), ('revoked', 'Revoked')], db_index=True, default='issued', max_length=10)), + ('expires_at', models.DateTimeField()), + ('redeemed_at', models.DateTimeField(blank=True, null=True)), + ('redeemed_group_chat_id', models.CharField(blank=True, help_text='Telegram chat id of the bound group (set on redemption)', max_length=32)), + ('redeemed_by_telegram_uid', models.CharField(blank=True, help_text='Numeric Telegram user id that redeemed the code (set on redemption)', max_length=32)), + ('created_by', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='telegram_bind_codes', to=settings.AUTH_USER_MODEL)), + ('validator', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='telegram_bind_codes', to='validators.validator')), + ], + options={ + 'ordering': ['-created_at'], + }, + ), + ] diff --git a/backend/validators/models.py b/backend/validators/models.py index c200fcea..82747ca6 100644 --- a/backend/validators/models.py +++ b/backend/validators/models.py @@ -1,3 +1,7 @@ +import hashlib +import secrets +from datetime import timedelta + from django.conf import settings from django.core.exceptions import FieldDoesNotExist from django.db import IntegrityError, models, transaction @@ -190,6 +194,116 @@ def ensure_validator_profile(user): return validator +class TelegramGroupBindCode(BaseModel): + """One-time code that binds a Telegram group to a validator via the Deckard bot. + + A validator issues a code from the portal, pastes it in their Telegram + group, and the Deckard support bot redeems it server-to-server (service + account scope `telegram_bind:redeem`). One code binds exactly one group; + a validator may hold multiple active codes (and therefore bind multiple + groups). DM bindings are refused bot-side. + + The plaintext code is shown exactly once at issuance. Only its SHA-256 + digest is stored, mirroring service_accounts.ServiceAccountToken: lookup is + by the non-secret `identifier` embedded in the code, then the presented + digest is compared in constant time. + """ + + CODE_PREFIX = 'tgb_' + DEFAULT_TTL_HOURS = 48 + + STATUS_ISSUED = 'issued' + STATUS_REDEEMED = 'redeemed' + STATUS_EXPIRED = 'expired' + STATUS_REVOKED = 'revoked' + STATUS_CHOICES = [ + (STATUS_ISSUED, 'Issued'), + (STATUS_REDEEMED, 'Redeemed'), + (STATUS_EXPIRED, 'Expired'), + (STATUS_REVOKED, 'Revoked'), + ] + + validator = models.ForeignKey( + 'Validator', + on_delete=models.CASCADE, + related_name='telegram_bind_codes', + ) + created_by = models.ForeignKey( + settings.AUTH_USER_MODEL, + on_delete=models.CASCADE, + related_name='telegram_bind_codes', + ) + identifier = models.CharField( + max_length=16, unique=True, + help_text="Non-secret lookup id embedded in the code", + ) + digest = models.CharField( + max_length=64, unique=True, + help_text="SHA-256 digest of the plaintext code (plaintext is never stored)", + ) + status = models.CharField( + max_length=10, choices=STATUS_CHOICES, default=STATUS_ISSUED, db_index=True, + ) + expires_at = models.DateTimeField() + redeemed_at = models.DateTimeField(null=True, blank=True) + redeemed_group_chat_id = models.CharField( + max_length=32, blank=True, + help_text="Telegram chat id of the bound group (set on redemption)", + ) + redeemed_by_telegram_uid = models.CharField( + max_length=32, blank=True, + help_text="Numeric Telegram user id that redeemed the code (set on redemption)", + ) + + class Meta: + ordering = ['-created_at'] + + def __str__(self): + return f"TelegramGroupBindCode {self.identifier} ({self.status})" + + @staticmethod + def hash_code(plaintext): + # Plain SHA-256 (not salted_hmac): the code secret has enough entropy + # for its 48h single-use lifetime, and this survives SECRET_KEY rotation. + return hashlib.sha256(plaintext.encode('utf-8')).hexdigest() + + @classmethod + def identifier_from_plaintext(cls, plaintext): + """Extract the non-secret lookup identifier from `tgb__`.""" + if not plaintext or not plaintext.startswith(cls.CODE_PREFIX): + return None + remainder = plaintext[len(cls.CODE_PREFIX):] + identifier, separator, secret = remainder.partition('_') + if not separator or not identifier or not secret: + return None + return identifier + + @classmethod + def issue(cls, validator, created_by, ttl_hours=DEFAULT_TTL_HOURS): + """Create a bind code and return (instance, plaintext).""" + identifier = secrets.token_hex(6) + plaintext = f'{cls.CODE_PREFIX}{identifier}_{secrets.token_urlsafe(12)}' + bind_code = cls.objects.create( + validator=validator, + created_by=created_by, + identifier=identifier, + digest=cls.hash_code(plaintext), + expires_at=timezone.now() + timedelta(hours=ttl_hours), + ) + return bind_code, plaintext + + @property + def effective_status(self): + """`status` with lazy expiry: an issued code past expires_at reads expired.""" + if self.status == self.STATUS_ISSUED and self.expires_at <= timezone.now(): + return self.STATUS_EXPIRED + return self.status + + def is_redeemable(self, now=None): + now = now or timezone.now() + return self.status == self.STATUS_ISSUED and self.expires_at > now + + class ValidatorWalletStatusSnapshot(BaseModel): """ Daily snapshot of a validator wallet's status. diff --git a/backend/validators/serializers.py b/backend/validators/serializers.py index f4519308..f677339e 100644 --- a/backend/validators/serializers.py +++ b/backend/validators/serializers.py @@ -1,9 +1,34 @@ from rest_framework import serializers from django.conf import settings -from .models import ValidatorOperatorWallet, ValidatorWallet, Validator +from .models import TelegramGroupBindCode, ValidatorOperatorWallet, ValidatorWallet, Validator from users.utils import truncate_address +class TelegramBindCodeSerializer(serializers.ModelSerializer): + """List/detail serializer for Telegram group bind codes. + + Never exposes the plaintext code or its digest — the plaintext is returned + exactly once by the issuance endpoint. `status` is the lazy-expiry + effective status, so an issued code past its expiry reads 'expired'. + """ + + status = serializers.CharField(source='effective_status', read_only=True) + + class Meta: + model = TelegramGroupBindCode + fields = [ + 'id', + 'identifier', + 'status', + 'expires_at', + 'redeemed_at', + 'redeemed_group_chat_id', + 'redeemed_by_telegram_uid', + 'created_at', + ] + read_only_fields = fields + + def grafana_network_label(network): """Grafana `network` label value for a portal network key (e.g. 'asimov-phase5').""" return settings.GRAFANA_NETWORK_LABELS.get(network, network) diff --git a/backend/validators/tests/test_telegram_bind_codes.py b/backend/validators/tests/test_telegram_bind_codes.py new file mode 100644 index 00000000..03ceadeb --- /dev/null +++ b/backend/validators/tests/test_telegram_bind_codes.py @@ -0,0 +1,331 @@ +from datetime import timedelta + +from django.contrib.auth import get_user_model +from django.utils import timezone +from rest_framework import status +from rest_framework.test import APITestCase + +from service_accounts.testing import service_account_auth_headers +from social_connections.models import TelegramConnection +from validators.models import TelegramGroupBindCode, Validator + +User = get_user_model() + +ISSUE_URL = '/api/v1/validators/telegram-bind-codes/' +MINE_URL = '/api/v1/validators/telegram-bind-codes/mine/' +REDEEM_URL = '/api/v1/validators/telegram-bind-codes/redeem/' + +GROUP_CHAT_ID = '-1001234567890' +TELEGRAM_UID = '424242424242' + + +def redeem_payload(code, **overrides): + payload = { + 'code': code, + 'group_chat_id': GROUP_CHAT_ID, + 'telegram_uid': TELEGRAM_UID, + 'telegram_username': 'validator_ops', + } + payload.update(overrides) + return payload + + +class TelegramBindCodeIssuanceTests(APITestCase): + def setUp(self): + self.user = User.objects.create_user( + email='validator@example.com', + password='testpass123', + address='0x1111111111111111111111111111111111111111', + name='Validator One', + ) + self.validator = Validator.objects.create(user=self.user) + + def test_anonymous_cannot_issue(self): + response = self.client.post(ISSUE_URL) + self.assertIn( + response.status_code, + (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN), + ) + self.assertEqual(TelegramGroupBindCode.objects.count(), 0) + + def test_user_without_validator_profile_cannot_issue(self): + plain_user = User.objects.create_user( + email='plain@example.com', + password='testpass123', + ) + self.client.force_authenticate(user=plain_user) + response = self.client.post(ISSUE_URL) + self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN) + self.assertEqual(TelegramGroupBindCode.objects.count(), 0) + + def test_validator_issues_code_and_plaintext_is_returned_once(self): + self.client.force_authenticate(user=self.user) + response = self.client.post(ISSUE_URL) + + self.assertEqual(response.status_code, status.HTTP_201_CREATED) + code = response.data['code'] + self.assertTrue(code.startswith(TelegramGroupBindCode.CODE_PREFIX)) + self.assertEqual(response.data['status'], TelegramGroupBindCode.STATUS_ISSUED) + + bind_code = TelegramGroupBindCode.objects.get(id=response.data['id']) + self.assertEqual(bind_code.validator, self.validator) + self.assertEqual(bind_code.created_by, self.user) + # Only the digest is stored, never the plaintext. + self.assertEqual(bind_code.digest, TelegramGroupBindCode.hash_code(code)) + self.assertNotIn(code, [bind_code.identifier, bind_code.digest]) + # 48h expiry window. + ttl = bind_code.expires_at - timezone.now() + self.assertGreater(ttl, timedelta(hours=47)) + self.assertLessEqual(ttl, timedelta(hours=48)) + + def test_validator_can_hold_multiple_active_codes(self): + self.client.force_authenticate(user=self.user) + first = self.client.post(ISSUE_URL) + second = self.client.post(ISSUE_URL) + self.assertEqual(first.status_code, status.HTTP_201_CREATED) + self.assertEqual(second.status_code, status.HTTP_201_CREATED) + self.assertEqual( + TelegramGroupBindCode.objects.filter( + validator=self.validator, + status=TelegramGroupBindCode.STATUS_ISSUED, + ).count(), + 2, + ) + + def test_mine_lists_codes_without_raw_secrets(self): + self.client.force_authenticate(user=self.user) + issued = self.client.post(ISSUE_URL) + + response = self.client.get(MINE_URL) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(len(response.data), 1) + entry = response.data[0] + self.assertEqual(entry['id'], issued.data['id']) + self.assertNotIn('code', entry) + self.assertNotIn('digest', entry) + + def test_mine_only_returns_own_codes(self): + other = User.objects.create_user( + email='other-validator@example.com', + password='testpass123', + ) + other_validator = Validator.objects.create(user=other) + TelegramGroupBindCode.issue(other_validator, other) + + self.client.force_authenticate(user=self.user) + response = self.client.get(MINE_URL) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data, []) + + def test_mine_reports_lazy_expiry(self): + self.client.force_authenticate(user=self.user) + bind_code, _ = TelegramGroupBindCode.issue(self.validator, self.user) + TelegramGroupBindCode.objects.filter(pk=bind_code.pk).update( + expires_at=timezone.now() - timedelta(minutes=1) + ) + + response = self.client.get(MINE_URL) + self.assertEqual( + response.data[0]['status'], TelegramGroupBindCode.STATUS_EXPIRED + ) + + +class TelegramBindCodeRevokeTests(APITestCase): + def setUp(self): + self.user = User.objects.create_user( + email='validator@example.com', + password='testpass123', + ) + self.validator = Validator.objects.create(user=self.user) + self.bind_code, self.plaintext = TelegramGroupBindCode.issue( + self.validator, self.user + ) + self.client.force_authenticate(user=self.user) + + def _revoke(self, code_id): + return self.client.post( + f'/api/v1/validators/telegram-bind-codes/{code_id}/revoke/' + ) + + def test_owner_can_revoke_issued_code(self): + response = self._revoke(self.bind_code.id) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data['status'], TelegramGroupBindCode.STATUS_REVOKED) + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_REVOKED) + + def test_revoked_code_cannot_be_redeemed(self): + from rest_framework.test import APIClient + + self._revoke(self.bind_code.id) + response = APIClient().post( + REDEEM_URL, + redeem_payload(self.plaintext), + format='json', + **service_account_auth_headers(['telegram_bind:redeem']), + ) + self.assertEqual(response.status_code, status.HTTP_409_CONFLICT) + self.assertEqual(response.data['code'], 'revoked') + + def test_redeemed_code_cannot_be_revoked(self): + TelegramGroupBindCode.objects.filter(pk=self.bind_code.pk).update( + status=TelegramGroupBindCode.STATUS_REDEEMED + ) + response = self._revoke(self.bind_code.id) + self.assertEqual(response.status_code, status.HTTP_409_CONFLICT) + + def test_non_owner_cannot_revoke(self): + other = User.objects.create_user( + email='other@example.com', + password='testpass123', + ) + self.client.force_authenticate(user=other) + response = self._revoke(self.bind_code.id) + self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND) + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_ISSUED) + + +class TelegramBindCodeRedeemTests(APITestCase): + def setUp(self): + self.user = User.objects.create_user( + email='validator@example.com', + password='testpass123', + name='Validator One', + ) + self.validator = Validator.objects.create(user=self.user) + self.bind_code, self.plaintext = TelegramGroupBindCode.issue( + self.validator, self.user + ) + self.auth = service_account_auth_headers(['telegram_bind:redeem']) + + def _redeem(self, payload, auth=None): + return self.client.post( + REDEEM_URL, payload, format='json', **(auth if auth is not None else self.auth) + ) + + def test_redeem_requires_service_account_token(self): + response = self._redeem(redeem_payload(self.plaintext), auth={}) + self.assertIn( + response.status_code, + (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN), + ) + + def test_session_authenticated_user_cannot_redeem(self): + self.client.force_authenticate(user=self.user) + response = self._redeem(redeem_payload(self.plaintext), auth={}) + self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN) + + def test_redeem_requires_matching_scope(self): + wrong_scope = service_account_auth_headers( + ['ai_review:read'], name='wrong-scope-account' + ) + response = self._redeem(redeem_payload(self.plaintext), auth=wrong_scope) + self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN) + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_ISSUED) + + def test_successful_redeem_binds_group_and_upserts_connection(self): + response = self._redeem(redeem_payload(self.plaintext)) + + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertTrue(response.data['success']) + self.assertEqual(response.data['validator_id'], self.validator.id) + self.assertEqual(response.data['group_chat_id'], GROUP_CHAT_ID) + + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_REDEEMED) + self.assertEqual(self.bind_code.redeemed_group_chat_id, GROUP_CHAT_ID) + self.assertEqual(self.bind_code.redeemed_by_telegram_uid, TELEGRAM_UID) + self.assertIsNotNone(self.bind_code.redeemed_at) + + connection = TelegramConnection.objects.get(user=self.user) + self.assertEqual(connection.platform_user_id, TELEGRAM_UID) + self.assertEqual(connection.platform_username, 'validator_ops') + + def test_redeem_is_single_use(self): + first = self._redeem(redeem_payload(self.plaintext)) + second = self._redeem( + redeem_payload(self.plaintext, group_chat_id='-1009999999999') + ) + + self.assertEqual(first.status_code, status.HTTP_200_OK) + self.assertEqual(second.status_code, status.HTTP_409_CONFLICT) + self.assertEqual(second.data['code'], 'already_redeemed') + self.bind_code.refresh_from_db() + # The original binding is untouched. + self.assertEqual(self.bind_code.redeemed_group_chat_id, GROUP_CHAT_ID) + + def test_second_code_binds_second_group_for_same_validator(self): + second_code, second_plaintext = TelegramGroupBindCode.issue( + self.validator, self.user + ) + first = self._redeem(redeem_payload(self.plaintext)) + second = self._redeem( + redeem_payload(second_plaintext, group_chat_id='-1009999999999') + ) + + self.assertEqual(first.status_code, status.HTTP_200_OK) + self.assertEqual(second.status_code, status.HTTP_200_OK) + second_code.refresh_from_db() + self.assertEqual(second_code.redeemed_group_chat_id, '-1009999999999') + + def test_expired_code_cannot_be_redeemed(self): + TelegramGroupBindCode.objects.filter(pk=self.bind_code.pk).update( + expires_at=timezone.now() - timedelta(minutes=1) + ) + response = self._redeem(redeem_payload(self.plaintext)) + + self.assertEqual(response.status_code, status.HTTP_410_GONE) + self.assertEqual(response.data['code'], 'expired') + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_EXPIRED) + self.assertFalse(TelegramConnection.objects.filter(user=self.user).exists()) + + def test_unknown_code_is_rejected(self): + response = self._redeem(redeem_payload('tgb_deadbeefdead_notarealsecret')) + self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND) + self.assertEqual(response.data['code'], 'invalid_code') + + def test_tampered_secret_is_rejected(self): + # Same identifier, wrong secret: the constant-time digest compare fails. + identifier = self.bind_code.identifier + response = self._redeem( + redeem_payload(f'tgb_{identifier}_wrongsecretvalue') + ) + self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND) + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_ISSUED) + + def test_malformed_payload_is_rejected(self): + missing_code = self._redeem({ + 'group_chat_id': GROUP_CHAT_ID, + 'telegram_uid': TELEGRAM_UID, + }) + bad_chat = self._redeem( + redeem_payload(self.plaintext, group_chat_id='not-a-chat-id') + ) + bad_uid = self._redeem( + redeem_payload(self.plaintext, telegram_uid='abc') + ) + for response in (missing_code, bad_chat, bad_uid): + self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) + self.bind_code.refresh_from_db() + self.assertEqual(self.bind_code.status, TelegramGroupBindCode.STATUS_ISSUED) + + def test_redeem_updates_existing_connection_and_keeps_username(self): + TelegramConnection.objects.create( + user=self.user, + platform_user_id='111', + platform_username='old_handle', + linked_at=timezone.now() - timedelta(days=30), + ) + response = self._redeem( + redeem_payload(self.plaintext, telegram_username='') + ) + + self.assertEqual(response.status_code, status.HTTP_200_OK) + connection = TelegramConnection.objects.get(user=self.user) + self.assertEqual(connection.platform_user_id, TELEGRAM_UID) + # No username in the payload never blanks the stored handle. + self.assertEqual(connection.platform_username, 'old_handle') diff --git a/backend/validators/urls.py b/backend/validators/urls.py index 0af9fcde..d5a6f1e2 100644 --- a/backend/validators/urls.py +++ b/backend/validators/urls.py @@ -1,11 +1,24 @@ from django.urls import path, include from rest_framework.routers import DefaultRouter -from .views import ValidatorViewSet, ValidatorWalletViewSet +from .views import ( + TelegramBindCodeRedeemView, + TelegramBindCodeViewSet, + ValidatorViewSet, + ValidatorWalletViewSet, +) router = DefaultRouter() router.register(r'wallets', ValidatorWalletViewSet, basename='validator-wallet') +router.register(r'telegram-bind-codes', TelegramBindCodeViewSet, basename='telegram-bind-code') router.register(r'', ValidatorViewSet, basename='validator') urlpatterns = [ + # Service-account (Deckard bot) redemption; must precede the router so the + # viewset's detail route never swallows 'redeem'. + path( + 'telegram-bind-codes/redeem/', + TelegramBindCodeRedeemView.as_view(), + name='telegram-bind-code-redeem', + ), path('', include(router.urls)), -] \ No newline at end of file +] diff --git a/backend/validators/views.py b/backend/validators/views.py index 08ec6460..31a6d9bf 100644 --- a/backend/validators/views.py +++ b/backend/validators/views.py @@ -1,20 +1,30 @@ import logging import re +import secrets as secrets_lib import uuid from rest_framework import viewsets, status from rest_framework.decorators import action from rest_framework.response import Response from rest_framework.permissions import IsAuthenticated, AllowAny +from rest_framework.views import APIView -from utils.throttling import WalletLinkRateThrottle +from utils.throttling import TelegramBindCodeIssueRateThrottle, WalletLinkRateThrottle from django.db.models import Min, Q, Count from django.db import IntegrityError, transaction from django.conf import settings from django.utils import timezone from django.core.cache import cache -from .models import SyncLock, Validator, ValidatorOperatorWallet, ValidatorWallet +from .models import ( + SyncLock, + TelegramGroupBindCode, + Validator, + ValidatorOperatorWallet, + ValidatorWallet, + get_validator_profile, +) from .serializers import ( GrafanaValidatorSerializer, + TelegramBindCodeSerializer, ValidatorOperatorWalletSerializer, ValidatorWalletSerializer, WallOfShameSerializer, @@ -22,6 +32,9 @@ grafana_network_label, ) from .permissions import IsCronToken +from service_accounts.authentication import ServiceAccountAuthentication +from service_accounts.permissions import HasServiceAccountScope +from service_accounts.scopes import TELEGRAM_BIND_REDEEM_SCOPE from .genlayer_validators_service import GenLayerValidatorsService from .grafana_service import GrafanaValidatorStatusService from .version_status import compute_version_status @@ -1145,3 +1158,202 @@ def wall_of_shame(self, request): } cache.set(cache_key, payload, self.WALL_OF_SHAME_CACHE_TTL_SECONDS) return Response(payload) + + +class TelegramBindCodeViewSet(viewsets.GenericViewSet): + """ + Validator-facing Telegram group bind codes. + + A validator issues a one-time code here, pastes it in their Telegram group + (`/bindcode `), and the Deckard support bot redeems it through the + service-account endpoint below. The plaintext code is returned exactly once + by `create`; every other read only exposes metadata. + """ + permission_classes = [IsAuthenticated] + serializer_class = TelegramBindCodeSerializer + + def get_queryset(self): + return TelegramGroupBindCode.objects.filter(created_by=self.request.user) + + def get_throttles(self): + if self.action == 'create': + return [TelegramBindCodeIssueRateThrottle()] + return super().get_throttles() + + def create(self, request): + """Issue a new bind code. Returns the plaintext code ONCE.""" + validator = get_validator_profile(request.user) + if validator is None: + return Response( + {'error': 'Only validators can issue Telegram group bind codes.'}, + status=status.HTTP_403_FORBIDDEN, + ) + + bind_code, plaintext = TelegramGroupBindCode.issue(validator, request.user) + logger.info( + "Telegram bind code issued: user=%s (id=%s) code_id=%s", + request.user.address, request.user.id, bind_code.id, + ) + data = TelegramBindCodeSerializer(bind_code).data + data['code'] = plaintext + return Response(data, status=status.HTTP_201_CREATED) + + @action(detail=False, methods=['get']) + def mine(self, request): + """List the current user's bind codes (metadata only, no raw codes).""" + codes = self.get_queryset() + serializer = self.get_serializer(codes, many=True) + return Response(serializer.data) + + @action(detail=True, methods=['post']) + def revoke(self, request, pk=None): + """Revoke an unredeemed bind code so it can never bind a group.""" + with transaction.atomic(): + bind_code = ( + self.get_queryset() + .select_for_update() + .filter(pk=pk) + .first() + ) + if bind_code is None: + return Response( + {'error': 'Bind code not found.'}, + status=status.HTTP_404_NOT_FOUND, + ) + if bind_code.status == TelegramGroupBindCode.STATUS_REDEEMED: + return Response( + {'error': 'This code was already redeemed and cannot be revoked.'}, + status=status.HTTP_409_CONFLICT, + ) + if bind_code.status != TelegramGroupBindCode.STATUS_REVOKED: + bind_code.status = TelegramGroupBindCode.STATUS_REVOKED + bind_code.save(update_fields=['status', 'updated_at']) + return Response(self.get_serializer(bind_code).data) + + +class TelegramBindCodeRedeemView(APIView): + """ + Service-account endpoint for the Deckard Telegram bot. + + POST /api/v1/validators/telegram-bind-codes/redeem/ + Auth: `Authorization: Bearer sa__` with scope `telegram_bind:redeem`. + Body: {"code": "...", "group_chat_id": "...", "telegram_uid": "...", + "telegram_username": "optional"} + + Redemption is single-use and atomic: it marks the code redeemed, records + the bound group chat id and the redeeming Telegram uid, and upserts the + issuing user's TelegramConnection. The portal does not judge whether the + chat is a group vs a DM — the bot refuses DMs before ever calling this. + """ + authentication_classes = [ServiceAccountAuthentication] + permission_classes = [HasServiceAccountScope] + required_scopes = {'*': TELEGRAM_BIND_REDEEM_SCOPE} + + CHAT_ID_PATTERN = re.compile(r'^-?\d{1,31}$') + UID_PATTERN = re.compile(r'^\d{1,31}$') + + def post(self, request): + code = request.data.get('code') + group_chat_id = str(request.data.get('group_chat_id', '') or '').strip() + telegram_uid = str(request.data.get('telegram_uid', '') or '').strip() + telegram_username = str(request.data.get('telegram_username', '') or '').strip() + + if not isinstance(code, str) or not code.strip(): + return Response( + {'error': 'code is required.', 'code': 'invalid_request'}, + status=status.HTTP_400_BAD_REQUEST, + ) + code = code.strip() + if not self.CHAT_ID_PATTERN.match(group_chat_id): + return Response( + {'error': 'group_chat_id must be a Telegram chat id.', 'code': 'invalid_request'}, + status=status.HTTP_400_BAD_REQUEST, + ) + if not self.UID_PATTERN.match(telegram_uid): + return Response( + {'error': 'telegram_uid must be a numeric Telegram user id.', 'code': 'invalid_request'}, + status=status.HTTP_400_BAD_REQUEST, + ) + + invalid_response = Response( + {'error': 'Invalid or unknown bind code.', 'code': 'invalid_code'}, + status=status.HTTP_404_NOT_FOUND, + ) + identifier = TelegramGroupBindCode.identifier_from_plaintext(code) + if identifier is None: + return invalid_response + digest = TelegramGroupBindCode.hash_code(code) + + from social_connections.models import TelegramConnection + + now = timezone.now() + with transaction.atomic(): + bind_code = ( + TelegramGroupBindCode.objects + .select_for_update() + .select_related('created_by', 'validator') + .filter(identifier=identifier) + .first() + ) + if bind_code is None or not secrets_lib.compare_digest(bind_code.digest, digest): + return invalid_response + + if bind_code.status == TelegramGroupBindCode.STATUS_REVOKED: + return Response( + {'error': 'This bind code was revoked.', 'code': 'revoked'}, + status=status.HTTP_409_CONFLICT, + ) + if bind_code.status == TelegramGroupBindCode.STATUS_REDEEMED: + return Response( + {'error': 'This bind code was already redeemed.', 'code': 'already_redeemed'}, + status=status.HTTP_409_CONFLICT, + ) + if not bind_code.is_redeemable(now): + if bind_code.status == TelegramGroupBindCode.STATUS_ISSUED: + bind_code.status = TelegramGroupBindCode.STATUS_EXPIRED + bind_code.save(update_fields=['status', 'updated_at']) + return Response( + {'error': 'This bind code has expired.', 'code': 'expired'}, + status=status.HTTP_410_GONE, + ) + + bind_code.status = TelegramGroupBindCode.STATUS_REDEEMED + bind_code.redeemed_at = now + bind_code.redeemed_group_chat_id = group_chat_id + bind_code.redeemed_by_telegram_uid = telegram_uid + bind_code.save(update_fields=[ + 'status', + 'redeemed_at', + 'redeemed_group_chat_id', + 'redeemed_by_telegram_uid', + 'updated_at', + ]) + + # Verifiable Telegram identity for the issuing portal account. The + # numeric uid is the stable identity; the display-only username is + # kept only when the bot supplies one, so a later redemption + # without it never blanks an existing handle. + connection_defaults = { + 'platform_user_id': telegram_uid, + 'linked_at': now, + } + if telegram_username: + connection_defaults['platform_username'] = telegram_username + TelegramConnection.objects.update_or_create( + user=bind_code.created_by, + defaults=connection_defaults, + ) + + logger.info( + "Telegram bind code redeemed: code_id=%s validator_id=%s group_chat_id=%s", + bind_code.id, bind_code.validator_id, group_chat_id, + ) + return Response({ + 'success': True, + 'bind_code_id': bind_code.id, + 'validator_id': bind_code.validator_id, + 'user_id': bind_code.created_by_id, + 'user_name': bind_code.created_by.name, + 'group_chat_id': group_chat_id, + 'redeemed_at': bind_code.redeemed_at, + }) diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 18886390..0e9eb3f4 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -378,6 +378,7 @@ const routes = { '/validators/leaderboard': Leaderboard, '/validators/participants': Validators, '/validators/wall-of-shame': WallOfShame, + '/validators/telegram': ValidatorTelegram, // Link Telegram groups via Deckard bot bind codes '/validators/waitlist': Waitlist, '/validators/waitlist/participants': WaitlistParticipants, '/validators/waitlist/join': ValidatorWaitlist, @@ -477,6 +478,7 @@ const routes = { - `genTvAPI` - Gen TV streams (`list`, `get(slug)`) - `poapsAPI` - POAP list/detail/claims, user POAPs, secret claims, mint-link claims, and recovery wallet verification - `validatorsAPI.getWallOfShame(params)` - Public Wall of Shame list for active validators with Grafana metrics/logs status badges (renders in `routes/WallOfShame.svelte`) + - `validatorsAPI` Telegram group bind codes - `issueTelegramBindCode()` (raw code is ONLY in this response), `getMyTelegramBindCodes()`, `revokeTelegramBindCode(id)` (renders in `routes/ValidatorTelegram.svelte`; bot handle comes from `DECKARD_BOT_USERNAME` in `lib/config.js`, env `VITE_DECKARD_BOT_USERNAME`) - `notificationsAPI` - Portal notifications (`list`, `unreadCount`, `markRead(id)`, `markAllRead`) - `socialTasksAPI` - Social tasks (`list({ status, category })`, `complete(slug)`) - `stewardAPI` - Steward tools: submissions review plus structured AI-review feedback (`getAIFeedback(id)` reads all reviewer records for a submission; `submitAIFeedback(id, data)` creates or versioned-revises the current reviewer's exact-proposal feedback) and Discord XP (`getDiscordXP(params)` lists community XP rows for contributions and social task completions; `recordDiscordXPCopy(stateId)`, `markDiscordXPDistributed(stateId)`, `unsetDiscordXPDistributed(stateId)` act on a row's state id — `row.id`, not the contribution id) diff --git a/frontend/src/App.svelte b/frontend/src/App.svelte index 32cbb621..cd1db744 100644 --- a/frontend/src/App.svelte +++ b/frontend/src/App.svelte @@ -91,6 +91,7 @@ import Waitlist from './routes/Waitlist.svelte'; import WaitlistParticipants from './routes/WaitlistParticipants.svelte'; import WallOfShame from './routes/WallOfShame.svelte'; + import ValidatorTelegram from './routes/ValidatorTelegram.svelte'; import TermsOfUse from './routes/TermsOfUse.svelte'; import PrivacyPolicy from './routes/PrivacyPolicy.svelte'; @@ -287,6 +288,7 @@ '/validators/tasks': roleGatedRoute(SocialTasks, 'validator'), '/validators/participants': protectedRoute(Validators), '/validators/wall-of-shame': roleGatedRoute(WallOfShame, 'validator'), + '/validators/telegram': roleGatedRoute(ValidatorTelegram, 'validator'), '/validators/waitlist': protectedRoute(Waitlist), '/validators/waitlist/participants': protectedRoute(WaitlistParticipants), '/validators/waitlist/join': ValidatorWaitlist, diff --git a/frontend/src/components/Sidebar.svelte b/frontend/src/components/Sidebar.svelte index 5d9c0bf2..88d92d25 100644 --- a/frontend/src/components/Sidebar.svelte +++ b/frontend/src/components/Sidebar.svelte @@ -438,6 +438,19 @@ {/if} + { e.preventDefault(); openRoleSection('/validators/telegram', 'validator'); }} + class="flex items-center justify-between border-l-[1.5px] px-3 py-2 text-[14px] font-medium tracking-[0.28px] { + isActive('/validators/telegram') ? 'border-[#387DE8]' : 'border-[#f5f5f5]' + } {isRoleLocked('validator') ? 'text-gray-400' : 'text-black'}" + title={isRoleLocked('validator') ? 'Become a validator to unlock' : ''} + > + Telegram Support + {#if isRoleLocked('validator')} + + {/if} + {/if} @@ -931,6 +944,19 @@ {/if} + { e.preventDefault(); openRoleSection('/validators/telegram', 'validator'); }} + class="flex items-center justify-between border-l-[1.5px] px-3 py-2 text-[14px] font-medium tracking-[0.28px] { + isActive('/validators/telegram') ? 'border-[#387DE8]' : 'border-[#f5f5f5]' + } {isRoleLocked('validator') ? 'text-gray-400' : 'text-black'}" + title={isRoleLocked('validator') ? 'Become a validator to unlock' : ''} + > + Telegram Support + {#if isRoleLocked('validator')} + + {/if} + {/if} diff --git a/frontend/src/lib/api.js b/frontend/src/lib/api.js index fdac386f..774d88cd 100644 --- a/frontend/src/lib/api.js +++ b/frontend/src/lib/api.js @@ -283,7 +283,12 @@ export const validatorsAPI = { getMyValidatorWallets: () => api.get('/validators/my-wallets/'), linkValidatorWalletsByOperator: (operatorAddress) => api.post('/validators/link-by-operator/', { operator_address: operatorAddress }), getNetworks: () => api.get('/validators/wallets/networks/'), - getWallOfShame: (params = {}) => api.get('/validators/wallets/wall-of-shame/', { params }) + getWallOfShame: (params = {}) => api.get('/validators/wallets/wall-of-shame/', { params }), + // Telegram group bind codes (Deckard support bot). The raw code is only in + // the issue response — list/revoke responses carry metadata only. + issueTelegramBindCode: () => api.post('/validators/telegram-bind-codes/'), + getMyTelegramBindCodes: () => api.get('/validators/telegram-bind-codes/mine/'), + revokeTelegramBindCode: (id) => api.post(`/validators/telegram-bind-codes/${id}/revoke/`) }; // Builders API diff --git a/frontend/src/lib/config.js b/frontend/src/lib/config.js index 485220cb..e624bae8 100644 --- a/frontend/src/lib/config.js +++ b/frontend/src/lib/config.js @@ -13,3 +13,7 @@ export const TURNSTILE_SITE_KEY = import.meta.env.VITE_TURNSTILE_SITE_KEY || ''; // External Links Configuration export const FAUCET_URL = 'https://testnet-faucet.genlayer.foundation/'; + +// Deckard Telegram support bot @username (without the @). When unset, the +// Telegram group linking page falls back to generic wording. +export const DECKARD_BOT_USERNAME = import.meta.env.VITE_DECKARD_BOT_USERNAME || ''; diff --git a/frontend/src/routes/ValidatorTelegram.svelte b/frontend/src/routes/ValidatorTelegram.svelte new file mode 100644 index 00000000..5cef71db --- /dev/null +++ b/frontend/src/routes/ValidatorTelegram.svelte @@ -0,0 +1,249 @@ + + +
+
+

+ Link a Telegram group +

+

+ Connect a Telegram group to your validator so {botLabel} can support you + there. You can link as many groups as you need — each one uses its own + one-time code. Direct messages are not supported: the bot only binds + groups. +

+
+ + +
+

+ How it works +

+
    +
  1. Create a new Telegram group for your validator.
    2. +
  2. Add {botLabel} to that group.
    3. +
  3. Generate a code below, then run /bindcode <code> in the group.
    4. +
+

+ Each code binds one group, works once, and expires after 48 hours. +

+
+ + {#if error} +
+ {error} +
+ {:else} + +
+
+
+

+ Generate a group code +

+

+ The code is shown only once — paste it in your group right away. +

+
+ +
+ + {#if freshCode} +
+

+ Your one-time code +

+
+ + /bindcode {freshCode} + + +
+

+ Run this command in your Telegram group. It will not be shown + again{freshCodeExpiresAt ? ` and expires ${formatDate(freshCodeExpiresAt)}` : ''}. +

+
+ {/if} +
+ + +
+

+ Your codes +

+ {#if loading} +

Loading...

+ {:else if codes.length === 0} +

+ No codes yet. Generate one above to link your first group. +

+ {:else} +
+ {#each codes as code (code.id)} +
+
+
+
+ tgb_{code.identifier}_… + + {STATUS_LABELS[code.status] || code.status} + +
+

+ Created {formatDate(code.created_at)} + {#if code.status === 'issued'} + · Expires {formatDate(code.expires_at)} + {:else if code.status === 'redeemed'} + · Group linked {formatDate(code.redeemed_at)} + {#if code.redeemed_group_chat_id} + (chat {code.redeemed_group_chat_id}) + {/if} + {/if} +

+
+ {#if code.status === 'issued'} + + {/if} +
+
+ {/each} +
+ {/if} +
+ {/if} +
From b01989985b404b85e192bf2feaa7836256acd488 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iva=CC=81n=20Raskovsky?= Date: Thu, 30 Jul 2026 12:37:16 +0200 Subject: [PATCH 15/21] Update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bb8af0e..1652aad9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ All notable user-facing changes to this project will be documented in this file. ## Unreleased +- Validators can link Telegram support groups to their validator: generate a one-time code on the new Telegram Support page, paste it in a Telegram group with the Deckard support bot, and the group is bound to the validator (multiple groups supported, codes expire in 48 hours and can be revoked) (0cd7e5f) + - Marketing can create campaign links like portal.genlayer.foundation/join/builders/ethcc from the admin panel without a deployment; each link tracks visits, signups, and role activations per campaign, campaign traffic reaches Google Analytics with clean final URLs, and new accounts are attributed to the campaign that brought them (810bdc32) - Notification bodies in the navbar dropdown now stop at 120 characters with an ellipsis while keeping their formatting and links (b4815f48) From db37420b173c6fc4f5a15ddef2176179b2caa4d9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iva=CC=81n=20Raskovsky?= Date: Sat, 1 Aug 2026 03:35:30 +0200 Subject: [PATCH 16/21] Make syncing the production database to local development reliable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Getting a copy of production into a local environment now runs as a single command: it dumps production with pg_dump, restores that into a throwaway local Postgres container, brings the copy up to the current schema, and only then converts it to SQLite. The previous approach serialized production over the network row by row and never completed against the current data volume. The two overlapping sync commands are consolidated into one, and the documentation now records why each stage of the pipeline exists — the schema drift, identifier mismatches, seeded rows and unguarded model signals that each caused a failed restore before the shape settled. ## Claude Implementation Notes - backend/scripts/sync_prod_to_sqlite.py: New end-to-end sync. Stages are pg_dump, restore into a local Postgres 17 container, migrate that copy, dumpdata locally, loaddata into a fresh SQLite, rebuild the leaderboard. Pins the Docker platform (cached amd64 images fail on Apple Silicon), keeps contenttypes/auth.permission in the export so permission ids match, clears all tables but django_migrations, and suppresses model signals during the load. Resume flags for each stage. - .claude/commands/sync-db.md: Rewritten around the new script; records the rationale per stage, the unguarded post_save receivers it works around, and why the other two scripts are not the local path. - .claude/commands/migrate-to-sqlite.md: Deleted; folded into sync-db. - backend/scripts/README.md: Adds a which-script-do-I-want table and the same rationale and known-bug notes. - backend/CLAUDE.md: Replaces the RDS-to-SQLite section with the new script. --- .claude/commands/migrate-to-sqlite.md | 18 -- .claude/commands/sync-db.md | 111 ++++++---- backend/CLAUDE.md | 15 +- backend/scripts/README.md | 75 +++++++ backend/scripts/sync_prod_to_sqlite.py | 284 +++++++++++++++++++++++++ 5 files changed, 435 insertions(+), 68 deletions(-) delete mode 100644 .claude/commands/migrate-to-sqlite.md create mode 100755 backend/scripts/sync_prod_to_sqlite.py diff --git a/.claude/commands/migrate-to-sqlite.md b/.claude/commands/migrate-to-sqlite.md deleted file mode 100644 index 93be15f5..00000000 --- a/.claude/commands/migrate-to-sqlite.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -description: Migrate production RDS database to local SQLite ---- - -Migrate the production PostgreSQL database to local SQLite for development. - -Steps: -1. Navigate to backend directory -2. Run the migration script: `python scripts/migrate_rds_to_sqlite.py` -3. Verify the migration completed successfully -4. Note that all user passwords will be reset to 'pass' - -The script will: -- Export data from production RDS using AWS SSM credentials -- Clean and process data (remove leaderboard entries, reset passwords) -- Create fresh SQLite database with migrations -- Import cleaned data -- Backup existing db.sqlite3 before replacing diff --git a/.claude/commands/sync-db.md b/.claude/commands/sync-db.md index 230f7859..8d6b770a 100644 --- a/.claude/commands/sync-db.md +++ b/.claude/commands/sync-db.md @@ -1,55 +1,76 @@ --- -description: Sync production database to local/dev environment +description: Sync production database to local development --- -Run the database migration script to sync production data to local development environment. +Put a copy of the production database in local `db.sqlite3`: -## Prerequisites -- Virtual environment must be activated: `source backend/env/bin/activate` -- AWS CLI configured with Parameter Store access -- Docker installed (for database operations) +```bash +cd backend +source env/bin/activate +python scripts/sync_prod_to_sqlite.py +``` -## Script Location -`backend/scripts/migrate-prod-to-dev.sh` +Roughly 30 minutes. Requires Docker running and AWS credentials for Parameter +Store. Every user password becomes `pass`. The existing `db.sqlite3` is renamed +to `db.sqlite3.backup_` first — those are ~1.6GB each, so prune them. -## Usage Options +Flags for resuming after a failure: -### Download Production Database Only (Safest) -```bash -cd backend/scripts -./migrate-prod-to-dev.sh --download -``` -Downloads production database to `backend/backups/` without making any local changes. +- `--reuse-dump` — skip the pg_dump, use the newest `backups/*.sql` +- `--reuse-postgres` — the `tally-local-pg` container already holds the data +- `--keep-container` — leave Postgres up (`docker start tally-local-pg` to reuse) +- `--keep-json` — keep the intermediate `prod_snapshot.json` +- `--no-leaderboard` — skip the leaderboard rebuild -### Upload Latest Dump to Dev Database -```bash -cd backend/scripts -./migrate-prod-to-dev.sh --upload -``` -Restores the most recent backup file to development database. +Verified end to end on 2026-08-01: 16 minutes with the dump already local, +0 dangling foreign keys, 56,304 users, 106,937 contributions. -### Run Django Migrations and Create Admin User Only -```bash -cd backend/scripts -./migrate-prod-to-dev.sh --setup -``` -Runs migrations and creates/updates admin user (`dev@genlayer.foundation` / `password`). +## Do not use the other two scripts -### Full Migration (Download + Upload + Setup) -```bash -cd backend/scripts -./migrate-prod-to-dev.sh -``` -Complete workflow: download production data, restore to dev, run migrations, and create admin. - -## What It Does -1. Fetches production database credentials from AWS Parameter Store -2. Downloads production PostgreSQL database using Docker (matching version) -3. Restores to development database (local or AWS dev instance) -4. Runs Django migrations -5. Creates/updates admin user with Steward role - -## Notes -- Backups are saved to `backend/backups/` with timestamps -- Uses Docker to avoid PostgreSQL version mismatch issues -- See `backend/scripts/README.md` for detailed documentation and troubleshooting +`scripts/migrate_rds_to_sqlite.py` runs `dumpdata` straight against production +RDS. Django emits a query per row for many-to-many fields, so over a remote link +it manages about 90 user rows per minute — a 3.5 hour run did not finish the +users table, and production has ~56k users. + +`scripts/migrate-prod-to-dev.sh` targets a **PostgreSQL** database (the shared +AWS dev instance), not local SQLite. Local Django uses SQLite unless +`DATABASE_URL` is set, so it is not the local-development path. Its upload step +is untested here. + +## Why the working script is shaped the way it is + +Each stage is scar tissue from a real failure; do not "simplify" them away: + +1. **pg_dump, not dumpdata, against production** — one streamed dump takes + minutes instead of never finishing. +2. **Explicit Docker `--platform`** — a cached amd64 `postgres:17` on Apple + Silicon fails with `exec format error`. +3. **Migrate the local Postgres copy before exporting** — production's schema + lags the code, so `dumpdata` otherwise fails on columns that exist only in + the models (it died on `ethereum_auth_pendingwalletsignup.acquisition_campaign_link_id`). +4. **Do not exclude contenttypes/auth.permission from the export** — the m2m + rows reference production's permission ids; a freshly migrated database + generates different ones, and `loaddata` then fails its foreign-key check at + commit, rolling back the entire load. +5. **Clear every table except `django_migrations` before loading** — data + migrations seed rows that collide with the snapshot on natural keys such as + `projects.Project.slug`. +6. **Suppress model signals during the load** — see the known bug below. +7. **Rebuild the leaderboard afterwards** — leaderboard entries are excluded + from the export, so it is empty until `manage.py update_leaderboard` runs. + +## Known bug this works around + +`contributions/models.py` `sync_contribution_discord_xp_state` and +`sync_social_task_completion_discord_xp_state` do not check +`kwargs.get('raw')`, so a fixture load recreates every +`ContributionDiscordXPState` and collides on `contribution_id` at the first row. +`users/signals.py` `create_referral_code` and `poaps/signals.py` +`attach_legacy_poap_claims` have the same gap on User. + +Neighbouring receivers guard correctly — `ensure_validator_profile_for_graduation_contribution` +in the same file, and `update_leaderboard_on_contribution` in +`leaderboard/models.py`, whose comment reads "Skip during fixture loading +(loaddata) to avoid ordering issues". The real fix is a one-line +`if kwargs.get('raw', False): return` in each of the four. Until that lands, the +sync script suppresses signals for the duration of the load. diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index dac34978..4587b2ce 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -419,11 +419,16 @@ cd backend/scripts - Creates timestamped backups in `backend/backups/` - See `backend/scripts/README.md` for detailed setup and troubleshooting -### RDS to SQLite Migration -- **Script**: `backend/scripts/migrate_rds_to_sqlite.py` -- **Purpose**: Convert production PostgreSQL to local SQLite for development -- **Usage**: `python scripts/migrate_rds_to_sqlite.py` (from backend directory) -- **Notes**: Resets all passwords to 'pass', excludes leaderboard entries, backs up existing db.sqlite3 +### Production to SQLite Sync (local development) +- **Script**: `backend/scripts/sync_prod_to_sqlite.py` +- **Purpose**: Put a copy of production in local `db.sqlite3` (the default local database) +- **Usage**: `python scripts/sync_prod_to_sqlite.py` (from backend directory, venv active, Docker running) +- **Takes**: ~30 min. Flags: `--reuse-dump`, `--reuse-postgres`, `--keep-container`, `--keep-json`, `--no-leaderboard` +- **Notes**: Resets all passwords to `pass`, backs up the existing `db.sqlite3` (~1.6GB per backup — prune them), rebuilds the leaderboard at the end +- **How**: pg_dump production → restore into a local Postgres container → `migrate` that copy → `dumpdata` locally → `loaddata` into a fresh SQLite. It never runs `dumpdata` against production, because Django's per-row m2m queries make that ~90 rows/minute over a remote link (hours, never finishes). + +- **Do NOT use `backend/scripts/migrate_rds_to_sqlite.py`** — it exports directly from production RDS and does not complete. Kept only for reference. Its `loaddata json_file 'exclude' 'leaderboard'` call is also wrong: those trailing strings are parsed as fixture labels, not as an exclude option. +- **Known bug the sync script works around**: `sync_contribution_discord_xp_state` and `sync_social_task_completion_discord_xp_state` (this file's `contributions/models.py`), plus `users/signals.py:create_referral_code` and `poaps/signals.py:attach_legacy_poap_claims`, do not check `kwargs.get('raw')`, so a fixture load recreates rows the fixture already contains and collides on `ContributionDiscordXPState.contribution_id`. Neighbouring receivers guard correctly; the fix is a one-line early return in each. ## API Endpoints Summary diff --git a/backend/scripts/README.md b/backend/scripts/README.md index 7a184ba0..4da08564 100644 --- a/backend/scripts/README.md +++ b/backend/scripts/README.md @@ -2,6 +2,81 @@ Scripts for migrating Tally production database to development environment. +## Which script do I want? + +| Goal | Script | +|---|---| +| Production data in my local `db.sqlite3` | `sync_prod_to_sqlite.py` | +| Production data in the shared dev **Postgres** instance | `migrate-prod-to-dev.sh` | +| Nothing — it does not finish | ~~`migrate_rds_to_sqlite.py`~~ | + +Local Django uses SQLite unless `DATABASE_URL` is set, so day-to-day work wants +the first row. + +## sync_prod_to_sqlite.py (local development) + +```bash +cd backend +source env/bin/activate +python scripts/sync_prod_to_sqlite.py +``` + +Roughly 30 minutes. Needs Docker running and AWS credentials. All passwords +become `pass`. The existing `db.sqlite3` is renamed to +`db.sqlite3.backup_` — those are ~1.6GB each, so prune old ones. + +Flags for resuming after a failure: `--reuse-dump` (skip pg_dump, use the newest +`backups/*.sql`), `--reuse-postgres` (the `tally-local-pg` container already +holds the data), `--keep-container`, `--keep-json`, `--no-leaderboard`. + +It works in five stages: pg_dump production → restore into a local Postgres 17 +container on port 5434 → `manage.py migrate` that copy → `dumpdata` from the +local copy → `loaddata` into a fresh SQLite, then rebuild the leaderboard. + +Each stage exists because of a specific failure; do not "simplify" them away: + +- **pg_dump, never `dumpdata`, against production.** Django issues a query per + row for many-to-many fields. Over a remote link that is ~90 user rows per + minute; a 3.5 hour run never finished the users table (production has ~56k). +- **Explicit Docker `--platform`.** A cached amd64 `postgres:17` on Apple + Silicon fails with `exec format error`. +- **Migrate the local copy before exporting.** Production's schema lags the + code, so `dumpdata` otherwise fails on model-only columns. +- **contenttypes and auth.permission are NOT excluded from the export.** The + m2m rows reference production's permission ids; a fresh database generates + different ones and `loaddata` fails its foreign-key check at commit, rolling + back the whole load. +- **Every table except `django_migrations` is cleared before loading.** Data + migrations seed rows that collide with the snapshot on natural keys such as + `projects.Project.slug`. +- **Model signals are suppressed during the load.** Several `post_save` + receivers ignore Django's `raw` flag and recreate rows the snapshot already + contains — see "Known bug" below. +- **The leaderboard is rebuilt afterwards** (`manage.py update_leaderboard`), + because leaderboard entries are excluded from the export. + +### Known bug it works around + +`contributions/models.py` `sync_contribution_discord_xp_state` and +`sync_social_task_completion_discord_xp_state`, plus +`users/signals.py:create_referral_code` and +`poaps/signals.py:attach_legacy_poap_claims`, do not check +`kwargs.get('raw')`. A fixture load therefore recreates every +`ContributionDiscordXPState` and dies on +`UNIQUE constraint failed: ...contribution_id` at the first row. + +Neighbouring receivers guard correctly — +`ensure_validator_profile_for_graduation_contribution` in the same file and +`update_leaderboard_on_contribution` in `leaderboard/models.py` ("Skip during +fixture loading (loaddata) to avoid ordering issues"). The real fix is a +one-line `if kwargs.get('raw', False): return` in each of the four. + +## migrate_rds_to_sqlite.py — do not use + +Runs `dumpdata` straight against production RDS and does not complete (see +above). It also calls `loaddata json_file 'exclude' 'leaderboard'`, where the +trailing strings are parsed as *fixture labels*, not as an exclude option. + ## Prerequisites 1. **Virtual Environment** must be activated: diff --git a/backend/scripts/sync_prod_to_sqlite.py b/backend/scripts/sync_prod_to_sqlite.py new file mode 100755 index 00000000..c4d25558 --- /dev/null +++ b/backend/scripts/sync_prod_to_sqlite.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python +"""Sync the production database into local db.sqlite3. + +Production is dumped with pg_dump, restored into a local Postgres container, +brought up to the current schema, and only then converted to SQLite. The +conversion never talks to production: `dumpdata` issues a query per row for +many-to-many fields, which over a remote link runs at roughly 90 rows/minute +and never finishes against a ~56k-user database. + +Usage: + python scripts/sync_prod_to_sqlite.py # full sync + python scripts/sync_prod_to_sqlite.py --reuse-dump # skip pg_dump, use newest backup + python scripts/sync_prod_to_sqlite.py --reuse-postgres # skip download+restore entirely + python scripts/sync_prod_to_sqlite.py --keep-container # leave Postgres running afterwards + +All user passwords become 'pass'. +""" +import argparse +import os +import platform +import subprocess +import sys +import time +from datetime import datetime +from pathlib import Path + +BACKEND_DIR = Path(__file__).resolve().parent.parent +BACKUP_DIR = BACKEND_DIR / 'backups' +SNAPSHOT = BACKEND_DIR / 'prod_snapshot.json' + +PROD_PARAM = '/tally/prod/database_url' +PG_IMAGE = 'postgres:17' +CONTAINER = 'tally-local-pg' +PG_PORT = '5434' +PG_PASSWORD = 'localpass' +LOCAL_DB_URL = f'postgresql://postgres:{PG_PASSWORD}@localhost:{PG_PORT}/postgres' + +# contenttypes and auth.permission are deliberately NOT excluded: the m2m rows +# in users_user_user_permissions reference production's permission ids, and a +# freshly migrated database generates different ones, which fails loaddata's +# foreign key check at commit time. +DUMPDATA_EXCLUDES = ['sessions', 'admin.logentry', 'leaderboard.leaderboardentry'] + +# Docker platform must be explicit: a cached amd64 postgres image on Apple +# Silicon dies with "exec format error". +DOCKER_PLATFORM = 'linux/arm64' if platform.machine() == 'arm64' else 'linux/amd64' + + +def log(msg): + print(f'\n=== {msg}', flush=True) + + +def run(cmd, **kwargs): + kwargs.setdefault('check', True) + return subprocess.run(cmd, **kwargs) + + +def capture(cmd): + return subprocess.run(cmd, check=True, capture_output=True, text=True).stdout.strip() + + +def dump_production(): + log('Dumping production with pg_dump') + url = capture([ + 'aws', 'ssm', 'get-parameter', '--name', PROD_PARAM, + '--with-decryption', '--query', 'Parameter.Value', '--output', 'text', + ]) + rest = url.split('://', 1)[1] + creds, hostpart = rest.split('@', 1) + user, password = creds.split(':', 1) + hostport, dbname = hostpart.split('/', 1) + host, _, port = hostport.partition(':') + port = port or '5432' + + BACKUP_DIR.mkdir(exist_ok=True) + out = BACKUP_DIR / f'tally_prod_{datetime.now():%Y%m%d_%H%M%S}.sql' + print(f'{host}:{port}/{dbname} -> {out.name}', flush=True) + run([ + 'docker', 'run', '--rm', '--platform', DOCKER_PLATFORM, + '-v', f'{BACKUP_DIR}:/backup', + '-e', f'PGPASSWORD={password}', + PG_IMAGE, 'pg_dump', + '-h', host, '-p', port, '-U', user, '-d', dbname, + '--no-owner', '--no-acl', '--clean', '--if-exists', + '--format=plain', f'--file=/backup/{out.name}', + ]) + return out + + +def latest_dump(): + dumps = sorted(BACKUP_DIR.glob('tally_prod_*.sql'), key=lambda p: p.stat().st_mtime) + if not dumps: + sys.exit('No dump found in backups/. Run without --reuse-dump.') + return dumps[-1] + + +def start_postgres(): + log('Starting local Postgres container') + existing = capture(['docker', 'ps', '-aq', '-f', f'name=^{CONTAINER}$']) + if existing: + run(['docker', 'start', CONTAINER], stdout=subprocess.DEVNULL) + else: + run([ + 'docker', 'run', '-d', '--name', CONTAINER, '--platform', DOCKER_PLATFORM, + '-e', f'POSTGRES_PASSWORD={PG_PASSWORD}', '-e', 'POSTGRES_DB=postgres', + '-p', f'{PG_PORT}:5432', PG_IMAGE, + ], stdout=subprocess.DEVNULL) + + for _ in range(60): + ready = subprocess.run( + ['docker', 'exec', CONTAINER, 'pg_isready', '-U', 'postgres'], + capture_output=True, + ) + if ready.returncode == 0: + print(f'Postgres ready on localhost:{PG_PORT}', flush=True) + return + time.sleep(2) + sys.exit('Postgres container did not become ready.') + + +def restore(dump_path): + log(f'Restoring {dump_path.name} into local Postgres') + run(['docker', 'cp', str(dump_path), f'{CONTAINER}:/tmp/dump.sql']) + # The dump carries --clean --if-exists, so restoring over an existing copy + # is fine; ON_ERROR_STOP=0 tolerates the drop statements on a fresh volume. + run([ + 'docker', 'exec', '-e', f'PGPASSWORD={PG_PASSWORD}', CONTAINER, + 'psql', '-q', '-U', 'postgres', '-d', 'postgres', + '-v', 'ON_ERROR_STOP=0', '-f', '/tmp/dump.sql', + ], stdout=subprocess.DEVNULL) + + +def manage(args, db_url=None): + env = os.environ.copy() + if db_url: + env['DATABASE_URL'] = db_url + else: + env.pop('DATABASE_URL', None) + run([sys.executable, '-u', 'manage.py'] + args, cwd=BACKEND_DIR, env=env) + + +def export_snapshot(): + # Production's schema lags the code, so migrate the local copy first or + # dumpdata fails on columns that only exist in the models. + log('Migrating local Postgres copy up to current schema') + manage(['migrate'], db_url=LOCAL_DB_URL) + + log('Exporting snapshot from local Postgres') + args = ['dumpdata', '--indent', '2'] + for label in DUMPDATA_EXCLUDES: + args += ['--exclude', label] + args += ['--output', str(SNAPSHOT)] + manage(args, db_url=LOCAL_DB_URL) + print(f'{SNAPSHOT.name}: {SNAPSHOT.stat().st_size / 1e9:.1f} GB', flush=True) + + +def rebuild_sqlite(): + log('Creating fresh SQLite database') + db = BACKEND_DIR / 'db.sqlite3' + if db.exists(): + backup = BACKEND_DIR / f'db.sqlite3.backup_{datetime.now():%Y%m%d_%H%M%S}' + db.replace(backup) + print(f'Existing database saved to {backup.name}', flush=True) + manage(['migrate']) + + log('Loading snapshot into SQLite') + run([sys.executable, '-u', __file__, '--_load', str(SNAPSHOT)], cwd=BACKEND_DIR) + + +def load_into_sqlite(snapshot): + """Child step: runs with SQLite settings, signals off, tables cleared.""" + os.environ.pop('DATABASE_URL', None) + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tally.settings') + sys.path.insert(0, str(BACKEND_DIR)) + + import django + + django.setup() + + from django.contrib.auth import get_user_model + from django.contrib.auth.hashers import make_password + from django.contrib.contenttypes.models import ContentType + from django.core.management import call_command + from django.db import connection + from django.db.models import signals + + # Several post_save receivers ignore Django's `raw` flag and recreate rows + # the snapshot already contains -- contributions.sync_contribution_discord_xp_state + # collides on ContributionDiscordXPState.contribution_id, and the User + # receivers in users/signals.py and poaps/signals.py fire once per restored + # user. A full snapshot needs no derived writes, so suppress all of them. + all_signals = [ + signals.pre_save, signals.post_save, + signals.pre_delete, signals.post_delete, + signals.m2m_changed, + ] + saved = {} + for sig in all_signals: + saved[sig] = sig.receivers + sig.receivers = [] + sig.sender_receivers_cache.clear() + + # Data migrations seed rows (default projects, contribution types) that + # collide with the snapshot on natural keys such as slug. + with connection.cursor() as cur: + cur.execute( + "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'" + ) + tables = [r[0] for r in cur.fetchall() if r[0] != 'django_migrations'] + cur.execute('PRAGMA foreign_keys = OFF') + for table in tables: + cur.execute(f'DELETE FROM "{table}"') + cur.execute('PRAGMA foreign_keys = ON') + print(f'Cleared seeded rows from {len(tables)} tables', flush=True) + + try: + call_command('loaddata', snapshot, verbosity=1) + ContentType.objects.clear_cache() + n = get_user_model().objects.update(password=make_password('pass')) + print(f"Reset password to 'pass' for {n} users", flush=True) + finally: + for sig, receivers in saved.items(): + sig.receivers = receivers + sig.sender_receivers_cache.clear() + + +def verify(): + log('Verifying') + import sqlite3 + + con = sqlite3.connect(BACKEND_DIR / 'db.sqlite3') + dangling = con.execute('PRAGMA foreign_key_check').fetchall() + print(f'Dangling foreign keys: {len(dangling)}', flush=True) + for table in ('users_user', 'contributions_contribution', 'leaderboard_leaderboardentry'): + n = con.execute(f'SELECT count(*) FROM {table}').fetchone()[0] + print(f'{table}: {n}', flush=True) + con.close() + return not dangling + + +def main(): + p = argparse.ArgumentParser(description=__doc__) + p.add_argument('--reuse-dump', action='store_true', help='use newest backups/*.sql') + p.add_argument('--reuse-postgres', action='store_true', help='container already holds the data') + p.add_argument('--keep-container', action='store_true', help='leave Postgres running') + p.add_argument('--keep-json', action='store_true', help='keep the intermediate snapshot') + p.add_argument('--no-leaderboard', action='store_true', help='skip leaderboard rebuild') + p.add_argument('--_load', help=argparse.SUPPRESS) + args = p.parse_args() + + if args._load: + load_into_sqlite(args._load) + return + + started = time.time() + if not args.reuse_postgres: + dump = latest_dump() if args.reuse_dump else dump_production() + start_postgres() + restore(dump) + else: + start_postgres() + + export_snapshot() + rebuild_sqlite() + + if not args.no_leaderboard: + # Leaderboard entries are excluded from the snapshot; rebuild them. + log('Rebuilding leaderboard') + manage(['update_leaderboard']) + + ok = verify() + + if not args.keep_json: + SNAPSHOT.unlink(missing_ok=True) + if not args.keep_container: + run(['docker', 'stop', CONTAINER], stdout=subprocess.DEVNULL) + print(f'Stopped {CONTAINER} (docker start {CONTAINER} to reuse)', flush=True) + + log(f'Done in {(time.time() - started) / 60:.0f} min. All passwords are pass.') + sys.exit(0 if ok else 1) + + +if __name__ == '__main__': + main() From 0c3028cc764cff1fff6af648d42c3a38f08844ef Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Sun, 2 Aug 2026 17:40:18 +0200 Subject: [PATCH 17/21] Make the production database sync fail loudly instead of silently The sync now recreates the local Postgres container before every restore, so tables created by a previous run's migrations can no longer survive the dump's --clean pass and corrupt the copy. The restore aborts on the first failed statement instead of tolerating errors, the production URL is handed to pg_dump whole instead of being hand-parsed (percent-encoded passwords and sslmode query params now work), and --reuse-postgres exits with an error when there is no container to reuse rather than exporting a freshly created empty database over the local SQLite. ## Claude Implementation Notes - backend/scripts/sync_prod_to_sqlite.py: dump_production passes the SSM value directly as pg_dump --dbname and drops the URL parsing plus the PGPASSWORD env; start_postgres gains fresh=True (docker rm -f before starting) used by the restore path, with pg_container_exists extracted as a helper; restore runs psql with ON_ERROR_STOP=1 and an updated rationale comment; main fails fast on --reuse-postgres when pg_container_exists() is false. --- backend/scripts/sync_prod_to_sqlite.py | 46 ++++++++++++++------------ 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/backend/scripts/sync_prod_to_sqlite.py b/backend/scripts/sync_prod_to_sqlite.py index c4d25558..4751d8fb 100755 --- a/backend/scripts/sync_prod_to_sqlite.py +++ b/backend/scripts/sync_prod_to_sqlite.py @@ -65,22 +65,15 @@ def dump_production(): 'aws', 'ssm', 'get-parameter', '--name', PROD_PARAM, '--with-decryption', '--query', 'Parameter.Value', '--output', 'text', ]) - rest = url.split('://', 1)[1] - creds, hostpart = rest.split('@', 1) - user, password = creds.split(':', 1) - hostport, dbname = hostpart.split('/', 1) - host, _, port = hostport.partition(':') - port = port or '5432' - BACKUP_DIR.mkdir(exist_ok=True) out = BACKUP_DIR / f'tally_prod_{datetime.now():%Y%m%d_%H%M%S}.sql' - print(f'{host}:{port}/{dbname} -> {out.name}', flush=True) + print(f'-> {out.name}', flush=True) + # The URI goes to pg_dump whole: parsing it here breaks percent-encoded + # passwords and drops query params such as ?sslmode=require. run([ 'docker', 'run', '--rm', '--platform', DOCKER_PLATFORM, '-v', f'{BACKUP_DIR}:/backup', - '-e', f'PGPASSWORD={password}', - PG_IMAGE, 'pg_dump', - '-h', host, '-p', port, '-U', user, '-d', dbname, + PG_IMAGE, 'pg_dump', '--dbname', url, '--no-owner', '--no-acl', '--clean', '--if-exists', '--format=plain', f'--file=/backup/{out.name}', ]) @@ -94,10 +87,17 @@ def latest_dump(): return dumps[-1] -def start_postgres(): +def pg_container_exists(): + return bool(capture(['docker', 'ps', '-aq', '-f', f'name=^{CONTAINER}$'])) + + +def start_postgres(fresh=False): log('Starting local Postgres container') - existing = capture(['docker', 'ps', '-aq', '-f', f'name=^{CONTAINER}$']) - if existing: + if fresh and pg_container_exists(): + # A previous run migrated this copy past the prod schema; tables the + # new dump does not know about survive --clean and poison the restore. + run(['docker', 'rm', '-f', CONTAINER], stdout=subprocess.DEVNULL) + if pg_container_exists(): run(['docker', 'start', CONTAINER], stdout=subprocess.DEVNULL) else: run([ @@ -121,12 +121,13 @@ def start_postgres(): def restore(dump_path): log(f'Restoring {dump_path.name} into local Postgres') run(['docker', 'cp', str(dump_path), f'{CONTAINER}:/tmp/dump.sql']) - # The dump carries --clean --if-exists, so restoring over an existing copy - # is fine; ON_ERROR_STOP=0 tolerates the drop statements on a fresh volume. + # The volume is fresh and the dump carries --clean --if-exists, so no + # statement may fail; abort loudly rather than flow a partial restore + # into the SQLite conversion. run([ 'docker', 'exec', '-e', f'PGPASSWORD={PG_PASSWORD}', CONTAINER, 'psql', '-q', '-U', 'postgres', '-d', 'postgres', - '-v', 'ON_ERROR_STOP=0', '-f', '/tmp/dump.sql', + '-v', 'ON_ERROR_STOP=1', '-f', '/tmp/dump.sql', ], stdout=subprocess.DEVNULL) @@ -253,12 +254,15 @@ def main(): return started = time.time() - if not args.reuse_postgres: - dump = latest_dump() if args.reuse_dump else dump_production() + if args.reuse_postgres: + if not pg_container_exists(): + sys.exit(f'--reuse-postgres: no {CONTAINER} container to reuse. ' + 'Run a full sync (or --reuse-dump) first.') start_postgres() - restore(dump) else: - start_postgres() + dump = latest_dump() if args.reuse_dump else dump_production() + start_postgres(fresh=True) + restore(dump) export_snapshot() rebuild_sqlite() From 276271668ef3f9fd525f56500fde1a7070dc99d7 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Sun, 2 Aug 2026 17:41:27 +0200 Subject: [PATCH 18/21] Keep wallet addresses out of Telegram bind code logs, document bot env var The bind code issuance log now identifies the issuing user by portal id only, keeping full wallet addresses out of application log sinks in line with the portal's address-privacy stance. The Deckard bot username setting is now part of the documented frontend environment so deployments configure it instead of silently falling back to generic wording. ## Claude Implementation Notes - backend/validators/views.py: issuance log drops request.user.address; user_id + code_id remain as identifiers - frontend/CLAUDE.md: add VITE_DECKARD_BOT_USERNAME to the Environment Variables section - frontend/.env.example: add VITE_DECKARD_BOT_USERNAME with a note that unset falls back to generic wording --- backend/validators/views.py | 4 ++-- frontend/.env.example | 4 ++++ frontend/CLAUDE.md | 1 + 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/backend/validators/views.py b/backend/validators/views.py index 31a6d9bf..1bd8d6bb 100644 --- a/backend/validators/views.py +++ b/backend/validators/views.py @@ -1191,8 +1191,8 @@ def create(self, request): bind_code, plaintext = TelegramGroupBindCode.issue(validator, request.user) logger.info( - "Telegram bind code issued: user=%s (id=%s) code_id=%s", - request.user.address, request.user.id, bind_code.id, + "Telegram bind code issued: user_id=%s code_id=%s", + request.user.id, bind_code.id, ) data = TelegramBindCodeSerializer(bind_code).data data['code'] = plaintext diff --git a/frontend/.env.example b/frontend/.env.example index 1fde14bd..86b05c5c 100644 --- a/frontend/.env.example +++ b/frontend/.env.example @@ -32,6 +32,10 @@ VITE_RECAPTCHA_SITE_KEY=6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI # Cloudflare Turnstile for email verification VITE_TURNSTILE_SITE_KEY=your_turnstile_site_key +# Deckard Telegram support bot @username without the @ (optional - the +# Telegram group linking page falls back to generic wording when unset) +VITE_DECKARD_BOT_USERNAME= + # Production values (to be set in Amplify environment variables) # VITE_API_URL=https://your-app-runner-url.amazonaws.com # VITE_GOOGLE_ANALYTICS_ID=G-5X6PRBXWJ7 diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 0e9eb3f4..40719439 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -766,6 +766,7 @@ Set in `.env` file: - `VITE_VALIDATOR_RPC_URL` - Blockchain RPC endpoint for Asimov testnet - `VITE_EXPLORER_URL` - Blockchain explorer URL - `VITE_RECAPTCHA_SITE_KEY` - Google reCAPTCHA site key (required - use test key from .env.example for development) +- `VITE_DECKARD_BOT_USERNAME` - Deckard Telegram support bot @username without the @ (optional - the Telegram group linking page falls back to generic wording) ## Common Commands ```bash From c01603b8bb0ee126b64ddd38859aac6b11cf8eea Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Sun, 2 Aug 2026 18:24:18 +0200 Subject: [PATCH 19/21] Protect the prod credential and discard partial restores in sync MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pg_dump step now hands the production password to libpq through the environment and passes a password-free URI on the command line, so the credential no longer appears in process listings while the dump runs. A restore that fails or is interrupted removes the staging container before the script exits, so a later --reuse-postgres resume can only ever see a completely restored copy. ## Claude Implementation Notes - backend/scripts/sync_prod_to_sqlite.py: dump_production splits userinfo off the URI (rpartition on @, partition on : — correct per RFC 3986, query params and encoding untouched), passes the password via bare `-e PGPASSWORD` with a subprocess env, and unquotes it for libpq; main() wraps restore() so any BaseException (incl. KeyboardInterrupt) triggers docker rm -f of the container before re-raising. Two CodeRabbit findings were skipped deliberately: `docker start` on a running container exits 0 (no already-running failure exists), and the --reuse-postgres existence check has only a millisecond TOCTOU window on a single-user laptop. --- backend/scripts/sync_prod_to_sqlite.py | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/backend/scripts/sync_prod_to_sqlite.py b/backend/scripts/sync_prod_to_sqlite.py index 4751d8fb..2cc1f91a 100755 --- a/backend/scripts/sync_prod_to_sqlite.py +++ b/backend/scripts/sync_prod_to_sqlite.py @@ -23,6 +23,7 @@ import time from datetime import datetime from pathlib import Path +from urllib.parse import unquote BACKEND_DIR = Path(__file__).resolve().parent.parent BACKUP_DIR = BACKEND_DIR / 'backups' @@ -65,18 +66,26 @@ def dump_production(): 'aws', 'ssm', 'get-parameter', '--name', PROD_PARAM, '--with-decryption', '--query', 'Parameter.Value', '--output', 'text', ]) + # The password may not appear in argv (visible in ps for the whole dump), + # so split it off the userinfo and hand it to libpq via the environment. + # Everything else stays untouched: rebuilding the URI wholesale breaks + # percent-encoded values and query params such as ?sslmode=require. + scheme, _, rest = url.partition('://') + userinfo, _, hostpart = rest.rpartition('@') + user, _, password = userinfo.partition(':') + safe_url = f'{scheme}://{user}@{hostpart}' if userinfo else url + BACKUP_DIR.mkdir(exist_ok=True) out = BACKUP_DIR / f'tally_prod_{datetime.now():%Y%m%d_%H%M%S}.sql' print(f'-> {out.name}', flush=True) - # The URI goes to pg_dump whole: parsing it here breaks percent-encoded - # passwords and drops query params such as ?sslmode=require. run([ 'docker', 'run', '--rm', '--platform', DOCKER_PLATFORM, '-v', f'{BACKUP_DIR}:/backup', - PG_IMAGE, 'pg_dump', '--dbname', url, + '-e', 'PGPASSWORD', # bare -e: docker reads the value from our env + PG_IMAGE, 'pg_dump', '--dbname', safe_url, '--no-owner', '--no-acl', '--clean', '--if-exists', '--format=plain', f'--file=/backup/{out.name}', - ]) + ], env=os.environ | {'PGPASSWORD': unquote(password)}) return out @@ -262,7 +271,13 @@ def main(): else: dump = latest_dump() if args.reuse_dump else dump_production() start_postgres(fresh=True) - restore(dump) + try: + restore(dump) + except BaseException: + # A partial restore (failed statement, Ctrl-C) must not linger + # where a later --reuse-postgres run would export it as if complete. + run(['docker', 'rm', '-f', CONTAINER], stdout=subprocess.DEVNULL) + raise export_snapshot() rebuild_sqlite() From 3261a831aeea81b0abaff5ccaaa39a8031180d4c Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Sun, 2 Aug 2026 19:15:15 +0200 Subject: [PATCH 20/21] Stop the sync from ever touching a dotenv-configured database The SQLite steps now pass an explicit empty DATABASE_URL instead of removing the variable. Removing it let settings reload the value from backend/.env, which pointed the migrate and destructive table-clearing steps at whatever database that file names. An empty value stays present, survives dotenv, and resolves to SQLite. The staging Postgres now binds to loopback only, since it holds unredacted production data behind a fixed password. The production snapshot moved into the gitignored backups/ directory so it can never be staged, and pg_dump writes to a .partial name that is renamed only on success, so --reuse-dump can never resume from an interrupted download. ## Claude Implementation Notes - backend/scripts/sync_prod_to_sqlite.py: manage() and load_into_sqlite() set DATABASE_URL='' instead of popping it (load_dotenv only fills ABSENT vars; verified on this machine that '' survives .env and settings picks SQLite); SNAPSHOT constant moved to BACKUP_DIR with a mkdir in export_snapshot() for --reuse-postgres runs; container publish changed to 127.0.0.1:5434 and LOCAL_DB_URL to 127.0.0.1 to skip localhost/::1 resolution; dump_production() writes tally_prod_*.sql.partial and renames after pg_dump succeeds, which latest_dump()'s *.sql glob never matches. - .claude/commands/sync-db.md: --keep-json flag doc now names backups/prod_snapshot.json. --- .claude/commands/sync-db.md | 2 +- backend/scripts/sync_prod_to_sqlite.py | 30 ++++++++++++++++++-------- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/.claude/commands/sync-db.md b/.claude/commands/sync-db.md index 8d6b770a..9e3ba94d 100644 --- a/.claude/commands/sync-db.md +++ b/.claude/commands/sync-db.md @@ -19,7 +19,7 @@ Flags for resuming after a failure: - `--reuse-dump` — skip the pg_dump, use the newest `backups/*.sql` - `--reuse-postgres` — the `tally-local-pg` container already holds the data - `--keep-container` — leave Postgres up (`docker start tally-local-pg` to reuse) -- `--keep-json` — keep the intermediate `prod_snapshot.json` +- `--keep-json` — keep the intermediate `backups/prod_snapshot.json` - `--no-leaderboard` — skip the leaderboard rebuild Verified end to end on 2026-08-01: 16 minutes with the dump already local, diff --git a/backend/scripts/sync_prod_to_sqlite.py b/backend/scripts/sync_prod_to_sqlite.py index 2cc1f91a..3b5bba8b 100755 --- a/backend/scripts/sync_prod_to_sqlite.py +++ b/backend/scripts/sync_prod_to_sqlite.py @@ -27,14 +27,16 @@ BACKEND_DIR = Path(__file__).resolve().parent.parent BACKUP_DIR = BACKEND_DIR / 'backups' -SNAPSHOT = BACKEND_DIR / 'prod_snapshot.json' +# The snapshot is a full production fixture; it lives in the gitignored +# backups/ so it can never be staged. +SNAPSHOT = BACKUP_DIR / 'prod_snapshot.json' PROD_PARAM = '/tally/prod/database_url' PG_IMAGE = 'postgres:17' CONTAINER = 'tally-local-pg' PG_PORT = '5434' PG_PASSWORD = 'localpass' -LOCAL_DB_URL = f'postgresql://postgres:{PG_PASSWORD}@localhost:{PG_PORT}/postgres' +LOCAL_DB_URL = f'postgresql://postgres:{PG_PASSWORD}@127.0.0.1:{PG_PORT}/postgres' # contenttypes and auth.permission are deliberately NOT excluded: the m2m rows # in users_user_user_permissions reference production's permission ids, and a @@ -77,6 +79,9 @@ def dump_production(): BACKUP_DIR.mkdir(exist_ok=True) out = BACKUP_DIR / f'tally_prod_{datetime.now():%Y%m%d_%H%M%S}.sql' + # Dump to a .partial name and rename only on success, so latest_dump()'s + # *.sql glob can never resume from an interrupted download. + tmp = out.with_name(out.name + '.partial') print(f'-> {out.name}', flush=True) run([ 'docker', 'run', '--rm', '--platform', DOCKER_PLATFORM, @@ -84,8 +89,9 @@ def dump_production(): '-e', 'PGPASSWORD', # bare -e: docker reads the value from our env PG_IMAGE, 'pg_dump', '--dbname', safe_url, '--no-owner', '--no-acl', '--clean', '--if-exists', - '--format=plain', f'--file=/backup/{out.name}', + '--format=plain', f'--file=/backup/{tmp.name}', ], env=os.environ | {'PGPASSWORD': unquote(password)}) + tmp.rename(out) return out @@ -112,7 +118,9 @@ def start_postgres(fresh=False): run([ 'docker', 'run', '-d', '--name', CONTAINER, '--platform', DOCKER_PLATFORM, '-e', f'POSTGRES_PASSWORD={PG_PASSWORD}', '-e', 'POSTGRES_DB=postgres', - '-p', f'{PG_PORT}:5432', PG_IMAGE, + # Loopback only: this container holds unredacted production data + # behind a fixed password. + '-p', f'127.0.0.1:{PG_PORT}:5432', PG_IMAGE, ], stdout=subprocess.DEVNULL) for _ in range(60): @@ -142,10 +150,11 @@ def restore(dump_path): def manage(args, db_url=None): env = os.environ.copy() - if db_url: - env['DATABASE_URL'] = db_url - else: - env.pop('DATABASE_URL', None) + # '' rather than pop: settings.py runs load_dotenv(), which fills in any + # ABSENT variable from backend/.env -- a popped DATABASE_URL would come + # back and point the SQLite steps (and their table clearing) at that + # database. An empty value stays present, and settings treats it as unset. + env['DATABASE_URL'] = db_url or '' run([sys.executable, '-u', 'manage.py'] + args, cwd=BACKEND_DIR, env=env) @@ -156,6 +165,7 @@ def export_snapshot(): manage(['migrate'], db_url=LOCAL_DB_URL) log('Exporting snapshot from local Postgres') + BACKUP_DIR.mkdir(exist_ok=True) # --reuse-postgres runs never created it args = ['dumpdata', '--indent', '2'] for label in DUMPDATA_EXCLUDES: args += ['--exclude', label] @@ -179,7 +189,9 @@ def rebuild_sqlite(): def load_into_sqlite(snapshot): """Child step: runs with SQLite settings, signals off, tables cleared.""" - os.environ.pop('DATABASE_URL', None) + # '' rather than pop -- see manage(): load_dotenv() refills absent vars, + # and this step deletes every table in whatever database it connects to. + os.environ['DATABASE_URL'] = '' os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tally.settings') sys.path.insert(0, str(BACKEND_DIR)) From b1a330583bbb9f4d4027a878509939b464de1611 Mon Sep 17 00:00:00 2001 From: JoaquinBN Date: Mon, 3 Aug 2026 10:12:46 +0200 Subject: [PATCH 21/21] Keep retained production data owner-only on disk The backups directory, which holds the unredacted SQL dumps and the intermediate JSON snapshot, is now forced to owner-only permissions every time the sync touches it, so a permissive umask or a pre-existing loose directory can no longer expose production data to other local users. The sync documentation now states that --keep-json retains a full production fixture that must be deleted manually. ## Claude Implementation Notes - backend/scripts/sync_prod_to_sqlite.py: ensure_backup_dir() helper does mkdir + chmod 0o700 (chmod unconditionally, since mkdir's mode is umask-masked and skipped for existing dirs); replaces the two bare BACKUP_DIR.mkdir sites in dump_production and export_snapshot. Directory perms cover every file inside, so CodeRabbit's per-file chmod and run-container-as-host-user suggestions were skipped as redundant. Also skipped: UUID dump names (concurrent runs are structurally unsupported -- shared container name/port/snapshot; the second run's docker rm -f kills the first mid-restore) and a generated PG_PASSWORD (loopback-only bind; any same-user process can read backups/*.sql directly, and an owner-only password file is readable by the same principal). - .claude/commands/sync-db.md: --keep-json flag now notes the snapshot is a full production fixture to delete manually. --- .claude/commands/sync-db.md | 3 ++- backend/scripts/sync_prod_to_sqlite.py | 11 +++++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/.claude/commands/sync-db.md b/.claude/commands/sync-db.md index 9e3ba94d..15576c1b 100644 --- a/.claude/commands/sync-db.md +++ b/.claude/commands/sync-db.md @@ -19,7 +19,8 @@ Flags for resuming after a failure: - `--reuse-dump` — skip the pg_dump, use the newest `backups/*.sql` - `--reuse-postgres` — the `tally-local-pg` container already holds the data - `--keep-container` — leave Postgres up (`docker start tally-local-pg` to reuse) -- `--keep-json` — keep the intermediate `backups/prod_snapshot.json` +- `--keep-json` — keep the intermediate `backups/prod_snapshot.json` (a full + production fixture; delete it manually when done) - `--no-leaderboard` — skip the leaderboard rebuild Verified end to end on 2026-08-01: 16 minutes with the dump already local, diff --git a/backend/scripts/sync_prod_to_sqlite.py b/backend/scripts/sync_prod_to_sqlite.py index 3b5bba8b..f5bca8af 100755 --- a/backend/scripts/sync_prod_to_sqlite.py +++ b/backend/scripts/sync_prod_to_sqlite.py @@ -62,6 +62,13 @@ def capture(cmd): return subprocess.run(cmd, check=True, capture_output=True, text=True).stdout.strip() +def ensure_backup_dir(): + BACKUP_DIR.mkdir(exist_ok=True) + # Everything in here is unredacted production data; keep it owner-only + # whatever the umask says, including for pre-existing directories. + BACKUP_DIR.chmod(0o700) + + def dump_production(): log('Dumping production with pg_dump') url = capture([ @@ -77,7 +84,7 @@ def dump_production(): user, _, password = userinfo.partition(':') safe_url = f'{scheme}://{user}@{hostpart}' if userinfo else url - BACKUP_DIR.mkdir(exist_ok=True) + ensure_backup_dir() out = BACKUP_DIR / f'tally_prod_{datetime.now():%Y%m%d_%H%M%S}.sql' # Dump to a .partial name and rename only on success, so latest_dump()'s # *.sql glob can never resume from an interrupted download. @@ -165,7 +172,7 @@ def export_snapshot(): manage(['migrate'], db_url=LOCAL_DB_URL) log('Exporting snapshot from local Postgres') - BACKUP_DIR.mkdir(exist_ok=True) # --reuse-postgres runs never created it + ensure_backup_dir() # --reuse-postgres runs never created it args = ['dumpdata', '--indent', '2'] for label in DUMPDATA_EXCLUDES: args += ['--exclude', label]