Historical Market Data Collector

An asynchronous, provider-isolated collector for auditable Gold/TMN observations. It stores scheduled cycles, every HTTP retry, optionally retained raw bodies, parser outcomes, normalized prices, and provider health transitions in SQLite.

Collection remains isolated from analysis and trading. The optional analyze command performs a read-only, historical, paper-only lead/lag study; it never creates orders or changes the database.

Quick start

uv sync
uv run market-collector validate-config --config config/collector.example.yaml
uv run market-collector db-upgrade --config config/collector.example.yaml
uv run market-collector run --config config/collector.example.yaml

Historical lead/lag analysis

After enough observations have been collected, create a reproducible report without starting the collector or requiring its credentials:

uv run market-collector analyze \
  --database data/market_data.db \
  --output-dir analysis-output \
  --price-fields-json '{"milli":"last_price","talasea":"last_price"}' \
  --start-at 2026-08-01T00:00:00Z \
  --end-at 2026-09-01T00:00:00Z \
  --max-lag-seconds 3600

The command writes analysis.json (auditable statistics and event-level evidence) and a self-contained report.html with heatmaps, distributions, cross-correlation, timing, stability, price-difference, return, ranking, and paper-signal views. When a field is omitted it is selected automatically and marked in the report; set each field explicitly before comparing tradeable prices. It accounts for quote age and incomplete horizons, reports lag uncertainty from polling, uses a circular-shift permutation test, and applies cost gates only to paper signals. It does not establish causality or verify that a provider label is an executable bid/ask. A per-provider observation safety cap prevents accidentally loading an unbounded history; analyze long histories in explicit time windows. Cost-adjusted follower movement is not executable PnL, and no paper trade candidate is emitted until a verified execution model is added.

uv run market-collector --config config/collector.example.yaml is equivalent to run.

Quality checks:

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright

Timestamp and price semantics

• All application datetimes are timezone-aware and normalized to UTC.
• observation_timestamp is the local UTC time at which the successful response was received.
• Provider timestamps are preserved separately, including their original representation.
• Durations use a monotonic clock and are not derived from wall-clock timestamps.
• Prices are parsed with Decimal and stored as exact decimal strings. Missing prices are NULL.
• Returned buy and sell labels are preserved as supplied by each provider; the collector does not reinterpret them as bid or ask.

Milli's documented price18 scale factor is 100. Talasea's documented price scale factor is 1000. No undocumented price conversion is performed.

The detailed field decisions are recorded in docs/provider-analysis.md. The original providers.txt capture contains session-like values, is ignored by Git, and was not copied into configuration or fixtures. Rotate those values if they are still valid.

Provider configuration

Providers are loaded from the configured module:Class adapter path. A new provider therefore does not require scheduler, HTTP, or storage changes. Request strings may use these instrument placeholders: {instrument}, {market_symbol}, {base_asset}, and {quote_asset}.

Any YAML string may reference an environment variable as ${NAME}. A missing variable is a startup error. Authentication values belong in environment variables, never in configuration committed to source control.

Raw response retention supports all, errors, and none. errors retains non-success HTTP bodies and successful bodies that later fail parsing or validation.

Providers may opt specific otherwise-non-retryable HTTP statuses into retry via retry.retryable_status_codes. This is intended for documented provider-edge quirks and should be scoped to the affected provider.

Operations

The process runs in the foreground and handles SIGINT and SIGTERM. Shutdown stops new polls, cancels or completes current work, drains the SQLite writer, and closes HTTP/database resources. Missed schedules are skipped after downtime; no catch-up requests or synthetic data are created.

SQLite is configured with WAL mode, foreign keys, and a busy timeout. Keep the database on a local filesystem and back it up using a SQLite-aware snapshot procedure.

Clock accuracy is part of data integrity. Production hosts must run NTP, chrony, or an equivalent time-synchronization service because later lead/lag analysis may depend on differences of seconds.

The systemd unit in deploy/systemd/market-collector.service expects a synchronized project virtual environment and secrets supplied through a protected environment file; the deployment script below sets both up.

Ubuntu 24.04 deployment

For the production host at /home/ubuntu/XUbritrage, use the versioned deployment script. It requires sudo, runs the collector as the ubuntu user, installs missing system prerequisites and uv, creates a non-versioned config/collector.yaml from the example once, creates a root-only environment file once, synchronizes the locked production dependencies, and installs the systemd service. Re-running deploy preserves both configuration and secrets.

cd /home/ubuntu/XUbritrage
sudo bash deploy/market-collector.sh deploy

Before deploying a configuration containing ${VARIABLE_NAME}, define every referenced value in /etc/market-collector/environment using systemd environment-file syntax, for example PROVIDER_API_TOKEN=.... The script checks that all variables referenced by the active configuration have non-empty values without printing the values. That environment file is owned by root and mode 0600; it is read by systemd and never stored in the repository.

Operational commands use the system journal, so they include both historical and live structured application logs:

sudo bash deploy/market-collector.sh status
sudo bash deploy/market-collector.sh stop
sudo bash deploy/market-collector.sh start
sudo bash deploy/market-collector.sh logs --lines 500
sudo bash deploy/market-collector.sh logs --follow

If deployment or startup fails, the script prints the service status and the most recent 100 journal entries. The collector is enabled at boot and restarted by systemd after failures.