Skip to content

fix(common-services): add connector.log retention to curator - #64

Open
gnanirahulnutakki wants to merge 1 commit into
masterfrom
fix/curator-connector-log-retention
Open

fix(common-services): add connector.log retention to curator#64
gnanirahulnutakki wants to merge 1 commit into
masterfrom
fix/curator-connector-log-retention

Conversation

@gnanirahulnutakki

Copy link
Copy Markdown
Member

Problem

connector.log is shipped to Elasticsearch by the fid-exporter sidecar — the fid chart defines the source in charts/fid/values.yaml:

connector:
  path: "/opt/radiantone/vds/logs/sync_agents/*/connector.log"
  index: connector.log

…but curator.logs in this chart has no entry for it. Every other prefix the exporter produces is pruned at 7 days; connector.log is the one exception, so its daily indices accumulate indefinitely.

It is also the highest-volume index in the stack — one document per connector event, across every sync agent — so it becomes the dominant consumer of Elasticsearch disk within a few months.

Evidence

Measured on the BSWH clusters (before a manual cleanup):

Cluster connector.log indices Size Oldest index Share of all index data
bswh-use1 90 55.8 GB 2026-05-13 78%
bswh-use2 348 167.6 GB 2025-08-28 85%

On bswh-use2 that is ~12 months of unpruned data. Curator was running nightly and healthy the whole time — it simply had no action matching this prefix, so every run logged Skipping action "delete_indices" due to empty list for the prefixes that had already aged out and never touched connector.log.

Change

One entry added to curator.logs. It inherits the chart's existing defaults (action: delete_indices, unit: days, unit_count: 7, direction: older), so it behaves exactly like the other 24 prefixes.

Verification

Rendered templates/elasticcurator/configmap.yaml before and after and diffed the result:

  • actions: 24 → 25
  • prefix set delta: added connector.log, removed none
  • retention across all 25 actions: uniformly unit_count: 7
  • action_file.yml parses as valid YAML in both renders

The only other diff is the positional renumbering of actions after index 11, which is inherent to range $index in the template — curator treats these as opaque action IDs executed in order, so the renumbering is behaviourally inert.

Rollout note

This is a values-only change with no chart version bump, to avoid colliding with the in-flight common-services/2.0.3 release (#63). It needs to be picked up by the next published chart version.

Consuming tenants pin the dependency explicitly, so they will each need a bump before this reaches a cluster — e.g. radiantlogic-saas/bswh-use1 is on common-services 2.0.1 and radiantlogic-saas/bswh-use2 on 2.0.0, both in app/charts/common-services/Chart.yaml.

Curator runs once per cluster in the -svc namespace, so this single change covers every -svc namespace that consumes the chart. The fid chart ships its own curator, but it defaults to enabled: false and is not deployed on these clusters, so no corresponding change is needed there.

connector.log is shipped to Elasticsearch by the fid-exporter sidecar
(fid chart source 'connector' -> index connector.log) but has no
delete_indices action in curator.logs, so its daily indices are never
reclaimed while every other shipped log is pruned at 7 days.

Observed on the BSWH clusters before cleanup:
  bswh-use1   90 indices   55.8 GB   oldest 2026-05-13   (78% of index data)
  bswh-use2  348 indices  167.6 GB   oldest 2025-08-28   (85% of index data)

Adds connector.log to curator.logs so it inherits the same 7-day
delete_indices action as the other 24 prefixes.
@gnanirahulnutakki gnanirahulnutakki self-assigned this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant