Skip to content

fix: init card database before accepting connections#144

Merged
Pyronewbic merged 3 commits into
mainfrom
dev
May 25, 2026
Merged

fix: init card database before accepting connections#144
Pyronewbic merged 3 commits into
mainfrom
dev

Conversation

@Pyronewbic
Copy link
Copy Markdown
Owner

Summary

  • Move eBay token warmup + card database init before app.listen()
  • Cloud Run only routes traffic after the card DB (29K cards) is loaded
  • Eliminates the 503/empty response window on cold start
  • requireCardDb middleware stays as a safety net if init fails

Test plan

  • 310 unit tests pass
  • Cold start serves full sets data on first request
  • Health endpoint shows cardDatabase: true immediately

Error monitoring: type filter on /api/errors, 30-day TTL via Firestore
expiresAt field, composite index for type+createdAt, process-level
unhandledRejection/uncaughtException handlers.

Analytics dashboard: tabbed admin panel with daily volume bar chart,
by-tier/status/path horizontal bars, top queries table, unique users
and avg latency stats. /api/analytics now returns daily, byStatus,
uniqueUsers fields.

Funnel tracking: user-milestones collection records signup, firstSearch,
firstGrade, firstPortfolioAdd per userId. GET /api/funnel (owner-only)
returns stage counts with 5-min cache. Admin funnel tab shows
proportional bars with conversion percentages.
Prevents caching empty responses when /api/sets, /api/sets/:setCode,
or /api/autocomplete are hit before initCardDatabase() completes.
Move eBay token + card DB init before app.listen so Cloud Run only
routes traffic after data is ready. Eliminates the 503 window on
cold start.
@Pyronewbic Pyronewbic merged commit b8ae989 into main May 25, 2026
9 checks passed
@github-actions
Copy link
Copy Markdown

Terraform Plan

Acquiring state lock. This may take a few moments...
google_project_service.artifactregistry: Refreshing state... [id=casecomp-495718/artifactregistry.googleapis.com]
google_logging_metric.api_errors: Refreshing state... [id=cardscrapebot-errors]
google_project_service.cloudkms: Refreshing state... [id=casecomp-495718/cloudkms.googleapis.com]
google_project_service.monitoring: Refreshing state... [id=casecomp-495718/monitoring.googleapis.com]
data.google_secret_manager_secret_version.api_key: Reading...
google_project_service.cloudbuild: Refreshing state... [id=casecomp-495718/cloudbuild.googleapis.com]
data.google_project.current: Reading...
google_storage_bucket.site: Refreshing state... [id=casecomp-site]
google_project_iam_member.deploy_sa_occurrence_editor: Refreshing state... [id=casecomp-495718/roles/containeranalysis.occurrences.editor/serviceAccount:casecomp-deploy@casecomp-495718.iam.gserviceaccount.com]
google_project_service.compute: Refreshing state... [id=casecomp-495718/compute.googleapis.com]
google_compute_managed_ssl_certificate.api_cert: Refreshing state... [id=projects/casecomp-495718/global/sslCertificates/cardscrapebot-cert-v2]
google_project_service.binaryauthorization: Refreshing state... [id=casecomp-495718/binaryauthorization.googleapis.com]
data.google_secret_manager_secret_version.api_key: Read complete after 1s [id=projects/129850122606/secrets/CASECOMP_API_KEY/versions/1]
google_project_service.run: Refreshing state... [id=casecomp-495718/run.googleapis.com]
google_project_iam_member.deploy_sa_note_attacher: Refreshing state... [id=casecomp-495718/roles/containeranalysis.notes.attacher/serviceAccount:casecomp-deploy@casecomp-495718.iam.gserviceaccount.com]
data.google_project.current: Read complete after 1s [id=projects/casecomp-495718]
google_project_service.firestore: Refreshing state... [id=casecomp-495718/firestore.googleapis.com]
google_project_service.containeranalysis: Refreshing state... [id=casecomp-495718/containeranalysis.googleapis.com]
google_compute_managed_ssl_certificate.site_cert: Refreshing state... [id=projects/casecomp-495718/global/sslCertificates/casecomp-site-cert]
google_project_service.secretmanager: Refreshing state... [id=casecomp-495718/secretmanager.googleapis.com]
google_artifact_registry_repository.casecomp_api: Refreshing state... [id=projects/casecomp-495718/locations/us/repositories/casecomp-api]
google_monitoring_notification_channel.email: Refreshing state... [id=projects/casecomp-495718/notificationChannels/3431772178774051140]
google_artifact_registry_repository.casecomp_node24: Refreshing state... [id=projects/casecomp-495718/locations/us/repositories/casecomp-node24]
google_monitoring_uptime_check_config.api_uptime: Refreshing state... [id=projects/casecomp-495718/uptimeCheckConfigs/casecomp-api-health-lQkUaC0Vzb8]
google_project_service.scheduler: Refreshing state... [id=casecomp-495718/cloudscheduler.googleapis.com]
google_kms_key_ring.binary_auth: Refreshing state... [id=projects/casecomp-495718/locations/global/keyRings/binary-auth]
google_storage_bucket_iam_member.site_public: Refreshing state... [id=b/casecomp-site/roles/storage.objectViewer/allUsers]
google_firestore_database.default: Refreshing state... [id=projects/casecomp-495718/databases/(default)]
google_secret_manager_secret.api_secrets["PSA_AUTH_TOKEN"]: Refreshing state... [id=projects/casecomp-495718/secrets/PSA_AUTH_TOKEN]
google_secret_manager_secret.api_secrets["TOGETHER_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/TOGETHER_API_KEY]
google_secret_manager_secret.api_secrets["CASECOMP_JWT_SECRET"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_JWT_SECRET]
google_secret_manager_secret.api_secrets["CASECOMP_ADMIN_SUB"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_ADMIN_SUB]
google_secret_manager_secret.api_secrets["CASECOMP_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_API_KEY]
google_secret_manager_secret.api_secrets["EBAY_CLIENT_SECRET"]: Refreshing state... [id=projects/casecomp-495718/secrets/EBAY_CLIENT_SECRET]
google_secret_manager_secret.api_secrets["ANTHROPIC_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/ANTHROPIC_API_KEY]
google_secret_manager_secret.api_secrets["GOOGLE_OAUTH_CLIENT_ID"]: Refreshing state... [id=projects/casecomp-495718/secrets/GOOGLE_OAUTH_CLIENT_ID]
google_secret_manager_secret.api_secrets["CASECOMP_SANDBOX_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_SANDBOX_KEY]
google_secret_manager_secret.api_secrets["RESEND_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/RESEND_API_KEY]
google_secret_manager_secret.api_secrets["EBAY_CLIENT_ID"]: Refreshing state... [id=projects/casecomp-495718/secrets/EBAY_CLIENT_ID]
google_compute_global_address.api_ip: Refreshing state... [id=projects/casecomp-495718/global/addresses/cardscrapebot-ip]
google_container_analysis_note.deploy_attestor: Refreshing state... [id=projects/casecomp-495718/notes/deploy-attestor]
google_monitoring_alert_policy.api_error_alert: Refreshing state... [id=projects/casecomp-495718/alertPolicies/16365448047387079183]
google_kms_crypto_key.attestor_key: Refreshing state... [id=projects/casecomp-495718/locations/global/keyRings/binary-auth/cryptoKeys/attestor-key]
google_monitoring_alert_policy.api_uptime_alert: Refreshing state... [id=projects/casecomp-495718/alertPolicies/14098674883088940398]
google_secret_manager_secret_iam_member.cloud_run_access["ANTHROPIC_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/ANTHROPIC_API_KEY/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["CASECOMP_SANDBOX_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_SANDBOX_KEY/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["CASECOMP_ADMIN_SUB"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_ADMIN_SUB/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["GOOGLE_OAUTH_CLIENT_ID"]: Refreshing state... [id=projects/casecomp-495718/secrets/GOOGLE_OAUTH_CLIENT_ID/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["PSA_AUTH_TOKEN"]: Refreshing state... [id=projects/casecomp-495718/secrets/PSA_AUTH_TOKEN/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["TOGETHER_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/TOGETHER_API_KEY/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["RESEND_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/RESEND_API_KEY/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["EBAY_CLIENT_SECRET"]: Refreshing state... [id=projects/casecomp-495718/secrets/EBAY_CLIENT_SECRET/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["EBAY_CLIENT_ID"]: Refreshing state... [id=projects/casecomp-495718/secrets/EBAY_CLIENT_ID/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["CASECOMP_API_KEY"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_API_KEY/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_secret_manager_secret_iam_member.cloud_run_access["CASECOMP_JWT_SECRET"]: Refreshing state... [id=projects/casecomp-495718/secrets/CASECOMP_JWT_SECRET/roles/secretmanager.secretAccessor/serviceAccount:129850122606-compute@developer.gserviceaccount.com]
google_artifact_registry_repository_iam_member.api_deploy: Refreshing state... [id=projects/casecomp-495718/locations/us/repositories/casecomp-api/roles/artifactregistry.writer/serviceAccount:casecomp-deploy@casecomp-495718.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.api_cloudbuild: Refreshing state... [id=projects/casecomp-495718/locations/us/repositories/casecomp-api/roles/artifactregistry.writer/serviceAccount:129850122606@cloudbuild.gserviceaccount.com]
google_kms_crypto_key_iam_member.deploy_sa_signer: Refreshing state... [id=projects/casecomp-495718/locations/global/keyRings/binary-auth/cryptoKeys/attestor-key/roles/cloudkms.signerVerifier/serviceAccount:casecomp-deploy@casecomp-495718.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.node24_deploy: Refreshing state... [id=projects/casecomp-495718/locations/us/repositories/casecomp-node24/roles/artifactregistry.writer/serviceAccount:casecomp-deploy@casecomp-495718.iam.gserviceaccount.com]
google_binary_authorization_attestor.deploy: Refreshing state... [id=projects/casecomp-495718/attestors/deploy-attestor]
google_binary_authorization_policy.default: Refreshing state... [id=projects/casecomp-495718]
google_cloud_run_v2_service.site["us-central1"]: Refreshing state... [id=projects/casecomp-495718/locations/us-central1/services/casecomp-site]
google_cloud_run_v2_service.site["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/services/casecomp-site]
google_cloud_run_v2_service.api["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/services/casecomp-api]
google_cloud_run_v2_service.api["us-central1"]: Refreshing state... [id=projects/casecomp-495718/locations/us-central1/services/casecomp-api]
google_firestore_field.error_logs_ttl: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/error-logs/fields/expiresAt]
google_firestore_index.composite["grade-logs_userId_createdAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/grade-logs/indexes/CICAgJim14AK]
google_firestore_index.composite["api-keys_ownerId_createdAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/api-keys/indexes/CICAgJiUpoMK]
google_firestore_index.composite["error-logs_type_createdAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/error-logs/indexes/CICAgNi4-ZIK]
google_firestore_index.composite["api-analytics_userId_ts"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/api-analytics/indexes/CICAgJjF9oIK]
google_firestore_index.composite["grade-logs_source_createdAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/grade-logs/indexes/CICAgJj7z4EJ]
google_firestore_index.composite["price-history_cardId_recordedAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/price-history/indexes/CICAgNi47oMK]
google_firestore_index.composite["price-history_cardKey_recordedAt"]: Refreshing state... [id=projects/casecomp-495718/databases/(default)/collectionGroups/price-history/indexes/CICAgOjXh4EK]
google_cloud_run_v2_service_iam_member.site_public["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/services/casecomp-site/roles/run.invoker/allUsers]
google_cloud_run_v2_service_iam_member.site_public["us-central1"]: Refreshing state... [id=projects/casecomp-495718/locations/us-central1/services/casecomp-site/roles/run.invoker/allUsers]
google_compute_region_network_endpoint_group.site_neg["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/regions/asia-south1/networkEndpointGroups/casecomp-site-neg]
google_compute_region_network_endpoint_group.site_neg["us-central1"]: Refreshing state... [id=projects/casecomp-495718/regions/us-central1/networkEndpointGroups/casecomp-site-neg-us-central1]
google_cloud_run_v2_service_iam_member.public["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/services/casecomp-api/roles/run.invoker/allUsers]
google_compute_region_network_endpoint_group.api_neg["us-central1"]: Refreshing state... [id=projects/casecomp-495718/regions/us-central1/networkEndpointGroups/casecomp-api-neg-us-central1]
google_compute_region_network_endpoint_group.api_neg["asia-south1"]: Refreshing state... [id=projects/casecomp-495718/regions/asia-south1/networkEndpointGroups/casecomp-api-neg]
google_cloud_run_v2_service_iam_member.public["us-central1"]: Refreshing state... [id=projects/casecomp-495718/locations/us-central1/services/casecomp-api/roles/run.invoker/allUsers]
google_compute_backend_service.site_backend: Refreshing state... [id=projects/casecomp-495718/global/backendServices/casecomp-site-backend]
google_cloud_scheduler_job.track_prices: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/jobs/casecomp-track-prices]
google_cloud_scheduler_job.check_alerts: Refreshing state... [id=projects/casecomp-495718/locations/asia-south1/jobs/casecomp-check-alerts]
google_compute_backend_service.api_backend: Refreshing state... [id=projects/casecomp-495718/global/backendServices/cardscrapebot-backend]
google_compute_url_map.api_urlmap: Refreshing state... [id=projects/casecomp-495718/global/urlMaps/cardscrapebot-urlmap]
google_compute_target_https_proxy.api_proxy: Refreshing state... [id=projects/casecomp-495718/global/targetHttpsProxies/cardscrapebot-https-proxy]
google_compute_global_forwarding_rule.api_https: Refreshing state... [id=projects/casecomp-495718/global/forwardingRules/cardscrapebot-https-rule]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_cloud_run_v2_service.site["asia-south1"] will be updated in-place
  ~ resource "google_cloud_run_v2_service" "site" {
        id                      = "projects/casecomp-495718/locations/asia-south1/services/casecomp-site"
        name                    = "casecomp-site"
        # (30 unchanged attributes hidden)

      ~ template {
            # (9 unchanged attributes hidden)

          ~ containers {
                name        = null
                # (5 unchanged attributes hidden)

              ~ resources {
                  ~ limits            = {
                      ~ "cpu"    = "0.5" -> "1000m"
                        # (1 unchanged element hidden)
                    }
                    # (2 unchanged attributes hidden)
                }

                # (2 unchanged blocks hidden)
            }

          ~ scaling {
              ~ max_instance_count = 3 -> 10
                # (1 unchanged attribute hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # google_cloud_run_v2_service.site["us-central1"] will be updated in-place
  ~ resource "google_cloud_run_v2_service" "site" {
        id                      = "projects/casecomp-495718/locations/us-central1/services/casecomp-site"
        name                    = "casecomp-site"
        # (30 unchanged attributes hidden)

      ~ template {
            # (9 unchanged attributes hidden)

          ~ containers {
                name        = null
                # (5 unchanged attributes hidden)

              ~ resources {
                  ~ limits            = {
                      ~ "cpu"    = "0.5" -> "1000m"
                        # (1 unchanged element hidden)
                    }
                    # (2 unchanged attributes hidden)
                }

                # (2 unchanged blocks hidden)
            }

          ~ scaling {
              ~ max_instance_count = 3 -> 10
                # (1 unchanged attribute hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

Merge to main to apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant