Skip to content

fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output - #527

Open
mayankpande88 wants to merge 4 commits into
mainfrom
fix/runner-gcp-logs-and-bq
Open

fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output#527
mayankpande88 wants to merge 4 commits into
mainfrom
fix/runner-gcp-logs-and-bq

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Two parity fixes for the GCP primitives:

  • gke_logs: the filter matched resource.type="gce_instance" by zone,
    which returns VM syslog rather than node-pool scaling events. Query
    the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
    resource, project_id + location labels, the autoscaler-visibility
    logName, severity>=DEFAULT), with a zone->region fallback since those
    logs are keyed by the cluster's location (zonal vs regional).

  • gke_traces: pass the dataset location (explicit location, else
    derived from zone) in the BigQuery job, and reshape the raw BQ REST
    response ({schema.fields[], rows[].f[].v}) into the
    {data, columns, column_types} envelope the backend warehouse consumer
    expects (mirroring the legacy run_bigquery).

Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j

…ery output

Two parity fixes for the GCP primitives:

- gke_logs: the filter matched resource.type="gce_instance" by zone,
  which returns VM syslog rather than node-pool scaling events. Query
  the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
  resource, project_id + location labels, the autoscaler-visibility
  logName, severity>=DEFAULT), with a zone->region fallback since those
  logs are keyed by the cluster's location (zonal vs regional).

- gke_traces: pass the dataset location (explicit `location`, else
  derived from `zone`) in the BigQuery job, and reshape the raw BQ REST
  response ({schema.fields[], rows[].f[].v}) into the
  {data, columns, column_types} envelope the backend warehouse consumer
  expects (mirroring the legacy run_bigquery).

Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j
@mayankpande88
mayankpande88 requested a review from a team as a code owner July 14, 2026 19:28

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the GCP client to query GKE cluster-autoscaler visibility logs instead of GCE instance logs, implementing a fallback from zone to region when no logs are found. It also updates QueryBigQuery to accept a location parameter and reshape responses into a {data, columns, column_types} envelope. Feedback on these changes includes a correction to zoneToRegion to prevent incorrect truncation of region names (e.g., converting us-central1 to us-central), and a performance optimization to check if cell values are non-empty before unmarshaling them.

Comment thread runner/pkg/observability/gcp/client.go
Comment on lines +205 to +209
var v any
// BigQuery cell values are JSON scalars (usually strings); keep the
// decoded value, or nil on an unexpected shape.
_ = json.Unmarshal(cell.V, &v)
vals[i] = v

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid unnecessary CPU overhead and potential errors from parsing empty or nil json.RawMessage values, we should check if cell.V is non-empty before calling json.Unmarshal.

Suggested change
var v any
// BigQuery cell values are JSON scalars (usually strings); keep the
// decoded value, or nil on an unexpected shape.
_ = json.Unmarshal(cell.V, &v)
vals[i] = v
var v any
if len(cell.V) > 0 {
_ = json.Unmarshal(cell.V, &v)
}
vals[i] = v

RamanKharchee
RamanKharchee previously approved these changes Jul 15, 2026
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants