Skip to content

[Bug] Pagination defaults inconsistent + unbounded countDocuments + cron uses env not Settings #34

@chronoai-shining

Description

@chronoai-shining

Severity: Medium

Three related API-surface hygiene issues.

1. Pagination defaults vary

Endpoint Default pageSize Max
/audit (admin/controllers/AuditController.ts:91-101) 50 200
/ingest/history (ingest/controllers/IngestController.ts:169-173) 20 100
/schemas (schemas/controllers/SchemaController.ts:108-112) 20 100
/workflows (workflows/controllers/WorkflowController.ts:97-101) 20 100
/connectors (workflows/controllers/ConnectorController.ts:102-106) 20 100
/runs/history (runner/controllers/HistoryController.ts:36-44) 20 100

No lower-bound clamp: pageSize=0 or negative bypasses pagination logic.

2. countDocuments(filter) on every list, no upper bound

Every list endpoint runs countDocuments(filter) in parallel with the find. At scale (100k+ docs) this is slower than the find itself. ConnectorController.syncConnectors (workflows/controllers/ConnectorController.ts:272-285) does find().toArray() with no limit — loads the entire catalog into memory.

3. Cron uses env, not Settings

runner/session-manager.ts:24-27 reads VERIFY_CRON_INTERVAL_HOURS once at module load. The Settings document (admin/models/settings.ts:11) stores verifyCronIntervalHours and lets operators update it via PUT /settings. Updates are silent no-ops until pod restart.

Same for Settings.eventRetentionDays — wired into the TTL index at boot, never reapplied.

Remediation

  • Add a parsePagination(page, pageSize) util in shared/crud.ts (depends on [Feature] Centralize HTTP client + CRUD scaffolding (kill duplication) #33). Clamp 1 <= pageSize <= 100, page >= 1. Apply uniformly.
  • Use estimatedDocumentCount() when no filter is applied; for filtered queries, allow a ?countMode=skip|estimate|exact query param so callers can opt out.
  • For syncConnectors, paginate the upload batch or stream.
  • Re-read Settings.verifyCronIntervalHours on boot; subscribe to settings changes (Mongo change stream OR a periodic poll on a settingsVersion field) and call stopVerifyCron(); startVerifyCron();. For expireAfterSeconds on event_logs, run db.runCommand({collMod: ...}) on settings change.

Coupled with #32 (config) and #33 (CRUD helpers).

Metadata

Metadata

Labels

apiAPI design & endpointsbugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions