refactor(backend): restructure config to support both networks by silent-cipher · Pull Request #570 · FilOzone/dealbot

silent-cipher · 2026-05-25T12:34:52Z

Built on top of #569

Major changes

Environment variables have been split into two categories:
- global variables
- per-network variables
Per-network variables are defined using network-specific prefixes (MAINNET_* and CALIBRATION_*).
Job scheduling now schedules provider jobs for both networks.
Workers now receive the network through job data, and the network is propagated throughout the call stack where necessary.
WalletSDKService now maintains a per-network cache.
SynapseViemClient / Synapse instances are accessed through the WalletSDKService cache. If an instance is unavailable, it is created on demand using the synapse factory commons. The cache behavior remains unchanged and is not updated after on-demand creation.
Network information is also propagated through logContext across checks.
Pull pieces continue to be hosted using pieceCid, making the process network-agnostic.

…ork-config

Copilot

Pull request overview

Refactors the backend configuration system to support running against multiple Filecoin networks (e.g., calibration + mainnet) from a single deployment, and threads network through chain- and provider-dependent services so jobs/metrics/DB operations are correctly network-scoped.

Changes:

Replaces the legacy single-network config (app.config.ts + validationSchema) with a new multi-network config loader + Joi env validation + legacy env compatibility shim.
Updates wallet/chain integration and check services (deal, retrieval, pull-check, piece-cleanup, data-retention, metrics collectors, dev-tools) to accept and use an explicit network parameter and per-network config.
Extends provider API responses and filtering to support network scoping (controller/service/DTO + blocklists per network).

Reviewed changes

Copilot reviewed 58 out of 58 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
apps/backend/src/worker.module.ts	Switches Nest config wiring to new `loadConfig`/`validateConfig`.
apps/backend/src/wallet-sdk/wallet-sdk.service.ts	Refactors wallet SDK to maintain per-network Synapse + provider caches/state.
apps/backend/src/wallet-sdk/wallet-sdk.service.spec.ts	Updates wallet-sdk tests for multi-network state and config shape.
apps/backend/src/retrieval/retrieval.service.ts	Threads `network` through retrieval flow and deal selection/liveness checks.
apps/backend/src/retrieval/retrieval.service.spec.ts	Updates retrieval tests for network-scoped config/labels/entities.
apps/backend/src/retrieval-addons/strategies/ipfs-block.strategy.ts	Looks up provider info using the deal’s network.
apps/backend/src/pull-check/pull-piece-stream-tracker.service.ts	Updates config typing/imports for new config module.
apps/backend/src/pull-check/pull-piece-stream-tracker.service.spec.ts	Aligns pull-piece stream tracker tests with new config types.
apps/backend/src/pull-check/pull-check.service.ts	Makes pull-check execution explicitly network-scoped (config + synapse client).
apps/backend/src/pull-check/pull-check.service.spec.ts	Updates pull-check tests to pass network and validate new labels/context.
apps/backend/src/providers/providers.service.ts	Adds optional network filter + per-network blocklist application in queries.
apps/backend/src/providers/providers.service.spec.ts	Updates providers service tests for per-network blocklist query clauses.
apps/backend/src/providers/providers.controller.ts	Adds `network` query param with validation for providers listing endpoint.
apps/backend/src/providers/dto/provider-list-response.dto.ts	Adds `network` to provider DTO for API responses/docs.
apps/backend/src/piece-cleanup/piece-cleanup.service.ts	Scopes cleanup quotas, DB queries, provider lookups, and Synapse access by network.
apps/backend/src/piece-cleanup/piece-cleanup.service.spec.ts	Updates piece-cleanup tests for network-scoped methods and Synapse sourcing.
apps/backend/src/metrics-prometheus/wallet-balance.collector.ts	Collects wallet balances per active network and labels metrics with `network`.
apps/backend/src/metrics-prometheus/wallet-balance.collector.spec.ts	Updates wallet-balance collector tests for per-network metric labels.
apps/backend/src/jobs/data-set-creation.handler.ts	Threads `network` into dataset provisioning/repair calls.
apps/backend/src/http-client/http-client.service.ts	Updates config import to new module.
apps/backend/src/dev-tools/dto/trigger-retrieval.dto.ts	Adds optional `network` query param validation for retrieval trigger.
apps/backend/src/dev-tools/dto/trigger-deal.dto.ts	Adds optional `network` query param validation for deal trigger.
apps/backend/src/dev-tools/dev-tools.service.ts	Adds default network behavior and passes network through deal/retrieval operations.
apps/backend/src/dev-tools/dev-tools.controller.ts	Exposes optional `network` query params for dev-tools endpoints.
apps/backend/src/deal/deal.service.ts	Makes deal creation, dataset liveness/repair, and wallet address/config network-scoped.
apps/backend/src/deal/deal.service.spec.ts	Updates deal service tests for new signatures and per-network config.
apps/backend/src/deal-addons/strategies/ipni.strategy.ts	Updates config import to new module.
apps/backend/src/dataSource/dataSource.service.ts	Updates config import to new module.
apps/backend/src/dataSource/dataSource.service.spec.ts	Updates config import to new module.
apps/backend/src/dataset-liveness/dataset-liveness.service.ts	Makes liveness probes network-aware (synapse client + provider registry access).
apps/backend/src/dataset-liveness/dataset-liveness.service.spec.ts	Updates liveness tests to pass network argument.
apps/backend/src/database/database.module.ts	Updates config import to new module.
apps/backend/src/data-retention/data-retention.service.ts	Makes data-retention polling network-scoped and uses per-network blocklists/providers.
apps/backend/src/config/types.ts	Introduces new typed config model including per-network slices and active networks.
apps/backend/src/config/loader.ts	Implements env-based config loader, including per-network prefixed vars.
apps/backend/src/config/loader.spec.ts	Adds tests for config loader behavior across networks and env layouts.
apps/backend/src/config/legacy-env-compat.ts	Adds compatibility shim translating legacy single-network env vars to prefixed form.
apps/backend/src/config/legacy-env-compat.spec.ts	Adds tests for legacy env translation behavior.
apps/backend/src/config/index.ts	Exposes config entry points (`loadConfig`, `validateConfig`) and types.
apps/backend/src/config/env.schema.ts	Adds Joi env validation including dynamic per-network requirements + legacy shim hook.
apps/backend/src/config/env.schema.spec.ts	Adds extensive tests for env schema validation rules.
apps/backend/src/config/env.parsers.ts	Adds domain-level env parsing helpers (networks, run mode, lists).
apps/backend/src/config/env.helpers.ts	Adds low-level env getters/coercion helpers.
apps/backend/src/config/constants.ts	Defines per-network default values and supported env prefixes.
apps/backend/src/config/app.config.ts	Removes legacy single-network config loader and schema.
apps/backend/src/common/synapse-factory.ts	Updates Synapse factory to accept `INetworkConfig` (wallet-key vs session-key variants).
apps/backend/src/common/sp-blocklist.ts	Updates blocklist helper to accept per-network blocklist fields.
apps/backend/src/common/sp-blocklist.spec.ts	Updates blocklist tests for new config shape.
apps/backend/src/common/logging.ts	Extends job/provider log context types to include `network`.
apps/backend/src/clickhouse/clickhouse.service.ts	Updates config import to new module.
apps/backend/src/app.module.ts	Switches Nest config wiring to new `loadConfig`/`validateConfig`.
apps/backend/src/app.controller.ts	Updates config endpoint to return per-active-network rates/intervals.
apps/backend/.env.example	Documents and demonstrates multi-network env layout and per-network overrides.
.env.example	Updates secrets-only example to use per-network prefixed wallet/session keys.

…ork-config

…iew findings Per-network env vars now resolve `<NETWORK>_<VAR>` -> unprefixed `<VAR>` -> default, so shared tuning is set once and overridden per network only where it differs. A single source of truth (network-fields.ts) classifies chain-specific vars (credentials, endpoints, blocklists) that never inherit, and drives both the loader resolver and the Joi schema. Validation: shared unprefixed keys are validated against the same rules as their prefixed overrides (not waved through by allowUnknown). Per-network defaults moved out of Joi into the loader so NestJS-injected prefixed defaults can't shadow a shared value. Also resolves prior-PR review findings on this branch: - TARGET<MAX is asserted post-resolution in the loader (the Joi .custom read the wrong, unprefixed key and never fired) - legacy var renames (DEALBOT_MAINTENANCE_WINDOWS_UTC, DEALBOT_MAINTENANCE_WINDOW_MINUTES, JOB_PIECE_CLEANUP_PER_SP_PER_HOUR) are promoted to their current names - empty/degenerate NETWORKS is rejected instead of booting idle - getBooleanEnv no longer treats "0"/"no"/"False" as true - tryGetSynapse() replaces the dead getSynapse() ?? createSynapseInstance fallback - retrieval provider lookup filters by (address, network) - shutdown drain timeout includes maxPieceCleanupRuntimeSeconds - dropped the import-time process.env.NETWORK default Prometheus label

SgtPooki

This is a huge PR.. I threw a few AIs at it to pick it apart and they came up with some flaws in this approach that I think are made simpler by #621. This works as is, but I think #621 makes the env loading story more consistent and understandable.

Approving under the assumption that #621 will also get merged.

Copilot

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 3 comments.

…ork-config

silent-cipher added 17 commits May 25, 2026 12:47

refactor(config): introduce multi network config

6180549

chore: migrate to multi network

274a932

migrate to multi-network

407c76d

migrate data-retention service

d379d34

migrate deal service

a60a542

migrate dev-tools

5bfe206

test: fix retrieval service specs

9681d73

chore: remove log

5bdbd42

migrate job service to multi network

ebed65e

test: fix env schema tests

490abf5

chore: add missing legacy per-network environment variables

60dc67e

add network to dataset liveness service

50f4fde

migrate pull-check service

a0b9beb

fix: keep clickhouse chain agnostic

e72921e

test: fix

2f84dd4

migrate providers service to multi network

145519d

migrate backend .env.example to multi-network configuration

77c9b98

FilOzzy added this to FOC May 25, 2026

github-project-automation Bot moved this to 📌 Triage in FOC May 25, 2026

silent-cipher mentioned this pull request May 25, 2026

refactor: restructure config to support both networks #468

Closed

silent-cipher added 3 commits May 25, 2026 18:27

docs: update environment-variables.md

a29872a

fix: job timeouts

bb97328

Merge branch 'feat/prometheum-network-label' into refactor/multi-netw…

9de0d75

…ork-config

BigLep moved this from 📌 Triage to 🔎 Awaiting review in FOC May 25, 2026

BigLep assigned silent-cipher May 25, 2026

BigLep requested a review from Copilot May 25, 2026 18:24

BigLep added this to the M4.5: GA Fast Follows milestone May 25, 2026

Copilot started reviewing on behalf of BigLep May 25, 2026 18:24 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Comment thread apps/backend/src/config/loader.ts Outdated

Comment thread apps/backend/src/config/loader.ts Outdated

Comment thread apps/backend/src/config/loader.spec.ts Outdated

Comment thread apps/backend/src/config/constants.ts Outdated

chore: address pr comments

6f45bda

silent-cipher mentioned this pull request May 26, 2026

feat(web): add multi-network support with network switcher #576

Open

silent-cipher requested a review from SgtPooki May 26, 2026 17:09

silent-cipher added 2 commits June 17, 2026 17:14

Merge branch 'feat/prometheum-network-label' into refactor/multi-netw…

4b90429

…ork-config

chore: add trailing newline to .env.example

4a9c35a

SgtPooki mentioned this pull request Jun 24, 2026

refactor(config): per-network shared-value inheritance + resolve #570 review findings #621

Open

SgtPooki approved these changes Jun 24, 2026

View reviewed changes

github-project-automation Bot moved this from 🔎 Awaiting review to ✔️ Approved by reviewer in FOC Jun 24, 2026

SgtPooki requested a review from Copilot June 24, 2026 18:24

Copilot started reviewing on behalf of SgtPooki June 24, 2026 18:25 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread apps/backend/src/wallet-sdk/wallet-sdk.service.ts

Comment thread apps/backend/src/providers/providers.service.ts

Comment thread apps/backend/src/config/loader.ts Outdated

silent-cipher mentioned this pull request Jun 26, 2026

feat: add network column to schema for multi-network support #463

Open

silent-cipher added 4 commits June 26, 2026 19:15

Merge branch 'feat/prometheum-network-label' into refactor/multi-netw…

72bff00

…ork-config

feat: add network field to sampled-retrieval logging

564403e

chore: address copilo review comments

ff9f9c6

Merge branch 'feat/prometheum-network-label' into refactor/multi-netw…

5833525

…ork-config

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(backend): restructure config to support both networks#570

refactor(backend): restructure config to support both networks#570
silent-cipher wants to merge 27 commits into
feat/prometheum-network-labelfrom
refactor/multi-network-config

silent-cipher commented May 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

silent-cipher commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Major changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

silent-cipher commented May 25, 2026 •

edited

Loading