Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
61c2e06
Add marketing campaign vanity links with funnel attribution (#958)
JoaquinBN Jul 29, 2026
d30649f
Stop re-scanning every user account on each authenticated request
JoaquinBN Jul 29, 2026
ccd23f7
Index case-insensitive wallet address lookups
JoaquinBN Jul 29, 2026
f9ad80d
Cut redundant community leaderboard scoring work
JoaquinBN Jul 29, 2026
3677db3
Stop the portal shell from multiplying requests per page view
JoaquinBN Jul 29, 2026
f5692c5
Make slow requests visible in production logs
JoaquinBN Jul 29, 2026
8634e08
Silence the Gunicorn control socket warning at startup
JoaquinBN Jul 29, 2026
459db6e
Document the request-volume and query-shape constraints
JoaquinBN Jul 29, 2026
1fa1498
Repair an interrupted address index build and honour forced profile r…
JoaquinBN Jul 29, 2026
dd60f92
Discard profile responses that arrive after sign-out
JoaquinBN Jul 29, 2026
91da6b6
Stop a late profile read from reverting a just-saved profile
JoaquinBN Jul 29, 2026
cf61862
Keep signup-in-progress sessions from reading as signed in
JoaquinBN Jul 29, 2026
45090ec
Assert the pending-signup response and catch best-effort profile refr…
JoaquinBN Jul 29, 2026
0cd7e5f
Let validators link Telegram support groups through the Deckard bot
rasca Jul 30, 2026
b019899
Update changelog
rasca Jul 30, 2026
1db693d
Fix the July 27 database CPU exhaustion (#959)
JoaquinBN Jul 30, 2026
db37420
Make syncing the production database to local development reliable
rasca Aug 1, 2026
0c3028c
Make the production database sync fail loudly instead of silently
JoaquinBN Aug 2, 2026
2762716
Keep wallet addresses out of Telegram bind code logs, document bot en…
JoaquinBN Aug 2, 2026
c01603b
Protect the prod credential and discard partial restores in sync
JoaquinBN Aug 2, 2026
c253b43
Merge pull request #960 from genlayer-foundation/feature/telegram-gro…
JoaquinBN Aug 2, 2026
3261a83
Stop the sync from ever touching a dotenv-configured database
JoaquinBN Aug 2, 2026
b1a3305
Keep retained production data owner-only on disk
JoaquinBN Aug 3, 2026
79299aa
Merge pull request #962 from genlayer-foundation/prod-db-sync
JoaquinBN Aug 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 0 additions & 18 deletions .claude/commands/migrate-to-sqlite.md

This file was deleted.

112 changes: 67 additions & 45 deletions .claude/commands/sync-db.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,77 @@
---
description: Sync production database to local/dev environment
description: Sync production database to local development
---

Run the database migration script to sync production data to local development environment.
Put a copy of the production database in local `db.sqlite3`:

## Prerequisites
- Virtual environment must be activated: `source backend/env/bin/activate`
- AWS CLI configured with Parameter Store access
- Docker installed (for database operations)
```bash
cd backend
source env/bin/activate
python scripts/sync_prod_to_sqlite.py
```

## Script Location
`backend/scripts/migrate-prod-to-dev.sh`
Roughly 30 minutes. Requires Docker running and AWS credentials for Parameter
Store. Every user password becomes `pass`. The existing `db.sqlite3` is renamed
to `db.sqlite3.backup_<timestamp>` first — those are ~1.6GB each, so prune them.

## Usage Options
Flags for resuming after a failure:

### Download Production Database Only (Safest)
```bash
cd backend/scripts
./migrate-prod-to-dev.sh --download
```
Downloads production database to `backend/backups/` without making any local changes.
- `--reuse-dump` — skip the pg_dump, use the newest `backups/*.sql`
- `--reuse-postgres` — the `tally-local-pg` container already holds the data
- `--keep-container` — leave Postgres up (`docker start tally-local-pg` to reuse)
- `--keep-json` — keep the intermediate `backups/prod_snapshot.json` (a full
production fixture; delete it manually when done)
- `--no-leaderboard` — skip the leaderboard rebuild

### Upload Latest Dump to Dev Database
```bash
cd backend/scripts
./migrate-prod-to-dev.sh --upload
```
Restores the most recent backup file to development database.
Verified end to end on 2026-08-01: 16 minutes with the dump already local,
0 dangling foreign keys, 56,304 users, 106,937 contributions.

### Run Django Migrations and Create Admin User Only
```bash
cd backend/scripts
./migrate-prod-to-dev.sh --setup
```
Runs migrations and creates/updates admin user (`dev@genlayer.foundation` / `password`).
## Do not use the other two scripts

### Full Migration (Download + Upload + Setup)
```bash
cd backend/scripts
./migrate-prod-to-dev.sh
```
Complete workflow: download production data, restore to dev, run migrations, and create admin.

## What It Does
1. Fetches production database credentials from AWS Parameter Store
2. Downloads production PostgreSQL database using Docker (matching version)
3. Restores to development database (local or AWS dev instance)
4. Runs Django migrations
5. Creates/updates admin user with Steward role

## Notes
- Backups are saved to `backend/backups/` with timestamps
- Uses Docker to avoid PostgreSQL version mismatch issues
- See `backend/scripts/README.md` for detailed documentation and troubleshooting
`scripts/migrate_rds_to_sqlite.py` runs `dumpdata` straight against production
RDS. Django emits a query per row for many-to-many fields, so over a remote link
it manages about 90 user rows per minute — a 3.5 hour run did not finish the
users table, and production has ~56k users.

`scripts/migrate-prod-to-dev.sh` targets a **PostgreSQL** database (the shared
AWS dev instance), not local SQLite. Local Django uses SQLite unless
`DATABASE_URL` is set, so it is not the local-development path. Its upload step
is untested here.

## Why the working script is shaped the way it is

Each stage is scar tissue from a real failure; do not "simplify" them away:

1. **pg_dump, not dumpdata, against production** — one streamed dump takes
minutes instead of never finishing.
2. **Explicit Docker `--platform`** — a cached amd64 `postgres:17` on Apple
Silicon fails with `exec format error`.
3. **Migrate the local Postgres copy before exporting** — production's schema
lags the code, so `dumpdata` otherwise fails on columns that exist only in
the models (it died on `ethereum_auth_pendingwalletsignup.acquisition_campaign_link_id`).
4. **Do not exclude contenttypes/auth.permission from the export** — the m2m
rows reference production's permission ids; a freshly migrated database
generates different ones, and `loaddata` then fails its foreign-key check at
commit, rolling back the entire load.
5. **Clear every table except `django_migrations` before loading** — data
migrations seed rows that collide with the snapshot on natural keys such as
`projects.Project.slug`.
6. **Suppress model signals during the load** — see the known bug below.
7. **Rebuild the leaderboard afterwards** — leaderboard entries are excluded
from the export, so it is empty until `manage.py update_leaderboard` runs.

## Known bug this works around

`contributions/models.py` `sync_contribution_discord_xp_state` and
`sync_social_task_completion_discord_xp_state` do not check
`kwargs.get('raw')`, so a fixture load recreates every
`ContributionDiscordXPState` and collides on `contribution_id` at the first row.
`users/signals.py` `create_referral_code` and `poaps/signals.py`
`attach_legacy_poap_claims` have the same gap on User.

Neighbouring receivers guard correctly — `ensure_validator_profile_for_graduation_contribution`
in the same file, and `update_leaderboard_on_contribution` in
`leaderboard/models.py`, whose comment reads "Skip during fixture loading
(loaddata) to avoid ordering issues". The real fix is a one-line
`if kwargs.get('raw', False): return` in each of the four. Until that lands, the
sync script suppresses signals for the duration of the load.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ All notable user-facing changes to this project will be documented in this file.

## Unreleased

- Validators can link Telegram support groups to their validator: generate a one-time code on the new Telegram Support page, paste it in a Telegram group with the Deckard support bot, and the group is bound to the validator (multiple groups supported, codes expire in 48 hours and can be revoked) (0cd7e5f)

- Marketing can create campaign links like portal.genlayer.foundation/join/builders/ethcc from the admin panel without a deployment; each link tracks visits, signups, and role activations per campaign, campaign traffic reaches Google Analytics with clean final URLs, and new accounts are attributed to the campaign that brought them (810bdc32)

- Notification bodies in the navbar dropdown now stop at 120 characters with an ellipsis while keeping their formatting and links (b4815f48)

- Builder submissions now pass through AI review before standard steward review, high-point acceptances are escalated as proposals to top-level stewards, and apex stewards have a focused queue for accepted contributions marked interesting. Appeals and more-information resubmissions remain visible to both AI review stages.
Expand Down
7 changes: 7 additions & 0 deletions amplify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ applications:
VITE_APP_NAME: Tally
VITE_VALIDATOR_RPC_URL: https://rpc.testnet-chain.genlayer.com
customRules:
# Campaign vanity links: reverse-proxy the reserved /join/ namespace to
# the Django resolver. Must stay BEFORE the SPA catch-all (rules apply
# in order). One dynamic rule for all campaigns; never add per-campaign
# rules here.
- source: '/join/<*>'
target: 'https://tally-backend.33qpgck0g28d0.us-east-1.cs.amazonlightsail.com/campaigns/redirect/<*>'
status: '200'
- source: '</^[^.]+$|\.(?!(css|gif|ico|jpg|js|png|txt|svg|woff|woff2|ttf|map|json|webp)$)([^.]+$)/>'
target: '/index.html'
status: '200'
Expand Down
Loading
Loading