fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output - #527
fix(runner/gcp): correct gke_logs filter and reshape gke_traces BigQuery output#527mayankpande88 wants to merge 4 commits into
Conversation
…ery output
Two parity fixes for the GCP primitives:
- gke_logs: the filter matched resource.type="gce_instance" by zone,
which returns VM syslog rather than node-pool scaling events. Query
the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
resource, project_id + location labels, the autoscaler-visibility
logName, severity>=DEFAULT), with a zone->region fallback since those
logs are keyed by the cluster's location (zonal vs regional).
- gke_traces: pass the dataset location (explicit `location`, else
derived from `zone`) in the BigQuery job, and reshape the raw BQ REST
response ({schema.fields[], rows[].f[].v}) into the
{data, columns, column_types} envelope the backend warehouse consumer
expects (mirroring the legacy run_bigquery).
Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j
There was a problem hiding this comment.
Code Review
This pull request updates the GCP client to query GKE cluster-autoscaler visibility logs instead of GCE instance logs, implementing a fallback from zone to region when no logs are found. It also updates QueryBigQuery to accept a location parameter and reshape responses into a {data, columns, column_types} envelope. Feedback on these changes includes a correction to zoneToRegion to prevent incorrect truncation of region names (e.g., converting us-central1 to us-central), and a performance optimization to check if cell values are non-empty before unmarshaling them.
| var v any | ||
| // BigQuery cell values are JSON scalars (usually strings); keep the | ||
| // decoded value, or nil on an unexpected shape. | ||
| _ = json.Unmarshal(cell.V, &v) | ||
| vals[i] = v |
There was a problem hiding this comment.
To avoid unnecessary CPU overhead and potential errors from parsing empty or nil json.RawMessage values, we should check if cell.V is non-empty before calling json.Unmarshal.
| var v any | |
| // BigQuery cell values are JSON scalars (usually strings); keep the | |
| // decoded value, or nil on an unexpected shape. | |
| _ = json.Unmarshal(cell.V, &v) | |
| vals[i] = v | |
| var v any | |
| if len(cell.V) > 0 { | |
| _ = json.Unmarshal(cell.V, &v) | |
| } | |
| vals[i] = v |
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
📦 Image Tags Updated |
Two parity fixes for the GCP primitives:
gke_logs: the filter matched resource.type="gce_instance" by zone,
which returns VM syslog rather than node-pool scaling events. Query
the GKE cluster-autoscaler-visibility logs instead (k8s_cluster
resource, project_id + location labels, the autoscaler-visibility
logName, severity>=DEFAULT), with a zone->region fallback since those
logs are keyed by the cluster's location (zonal vs regional).
gke_traces: pass the dataset location (explicit
location, elsederived from
zone) in the BigQuery job, and reshape the raw BQ RESTresponse ({schema.fields[], rows[].f[].v}) into the
{data, columns, column_types} envelope the backend warehouse consumer
expects (mirroring the legacy run_bigquery).
Note: the BigQuery reshape key names mirror the legacy agent; the
agent_warehouse consumer contract should be confirmed against the
backend repo before release.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01L2HAXQH1gEtX7h4sUJsi9j