Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6f642ce
Otel SDK span loss (#4286)
DavidS-ovm Mar 16, 2026
ae22b3a
[ENG-3154] Add self-review step to Azure adapter skill and validate w…
Lionel-Wilson Mar 17, 2026
44a2680
[ENG-3222] Add GCP Dataflow Job adapter with comprehensive link rules…
dylanratcliffe Mar 17, 2026
0f318a3
feat: Implement all-locations LIST support for 8 GCP adapters (#4305)
DavidS-ovm Mar 18, 2026
be4be2d
[ENG-3227] Add Dataflow aggregated LIST support (#4306)
DavidS-ovm Mar 18, 2026
bdaec67
feat: rename SDP proto fields for naming consistency (non-persisted) …
DavidS-ovm Mar 18, 2026
598393f
[ENG-3233] Phase 1: GitHub App Installation Requests — Schema, Config…
DavidS-ovm Mar 18, 2026
27a4eca
fix(deps): update module google.golang.org/grpc to v1.79.3 [security]…
renovate[bot] Mar 19, 2026
b9fb953
feat: add Azure NetworkFlowLog adapter (#4319)
Lionel-Wilson Mar 19, 2026
80fed94
Pin GoReleaser version to v2.14.3 in all workflows (#4333)
DavidS-ovm Mar 19, 2026
967bf2f
Eng 3205 phase 3 cursor cloud agent execution with credentials (#4339)
Lionel-Wilson Mar 19, 2026
2f36f73
feat: add Azure DBforPostgreSQL Flexible Server Backup adapter (#4318)
Lionel-Wilson Mar 19, 2026
14ebd71
feat: Add Dockerfile for snapshot source (ENG-3240) (#4331)
carabasdaniel Mar 19, 2026
b0c4810
feat: add --no-ping flag to skip health check in inject-secrets (#4342)
Lionel-Wilson Mar 19, 2026
71f57e1
feat: add Azure NetworkLoadBalancerFrontendIPConfiguration adapter (#…
Lionel-Wilson Mar 19, 2026
3e7a1ef
refactor(sdpcache): restructure cache package into focused, testable …
DavidS-ovm Mar 20, 2026
2d9f65b
[ENG-3281] feat(api-server): DB-backed state parameter for GitHub App…
DavidS-ovm Mar 20, 2026
f761018
fix(deps): update module github.com/modelcontextprotocol/go-sdk to v1…
renovate[bot] Mar 20, 2026
ba3ceec
fix(deps): update go (#4353)
DavidS-ovm Mar 20, 2026
149ed2d
fix(deps): update google.golang.org/genproto/googleapis/rpc digest to…
DavidS-ovm Mar 20, 2026
6d73dfe
chore(deps): update terraform (#4352)
DavidS-ovm Mar 20, 2026
fa3b39f
Fix all azure integration tests (#4367)
DavidS-ovm Mar 20, 2026
a3d534c
Run go mod tidy
actions-user Mar 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ jobs:
- name: Run GoReleaser (publish)
uses: goreleaser/goreleaser-action@v7
with:
version: latest
# renovate: datasource=github-releases depName=goreleaser/goreleaser
version: "v2.14.3"
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
58 changes: 29 additions & 29 deletions .terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion aws-source/module/provider/.github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ jobs:
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v7
with:
version: latest
# renovate: datasource=github-releases depName=goreleaser/goreleaser
version: "v2.14.3"
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
59 changes: 59 additions & 0 deletions docs.overmind.tech/docs/sources/gcp/Types/gcp-dataflow-job.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: GCP Dataflow Job
sidebar_label: gcp-dataflow-job
---

A **Google Cloud Dataflow Job** is a managed Apache Beam pipeline that processes streaming or batch data at scale. Dataflow handles resource provisioning, autoscaling, and fault tolerance, allowing you to run data processing workloads without managing the underlying infrastructure. Jobs can read from and write to Pub/Sub, BigQuery, Spanner, Bigtable, and other GCP services. See the official documentation for full details: https://cloud.google.com/dataflow/docs.

**Terraform Mappings:**

- `google_dataflow_job.job_id`
- `google_dataflow_flex_template_job.job_id`

## Supported Methods

- `GET`: Get a gcp-dataflow-job by its "locations|jobs"
- ~~`LIST`~~
- `SEARCH`: Search for gcp-dataflow-job by location

## Possible Links

### [`gcp-big-query-dataset`](/sources/gcp/Types/gcp-big-query-dataset)

Dataflow jobs that read from or write to BigQuery reference the dataset containing the tables they use. If the dataset is deleted or misconfigured, the job may fail to access data.

### [`gcp-big-query-table`](/sources/gcp/Types/gcp-big-query-table)

Dataflow jobs can read from or write to specific BigQuery tables. If a table is deleted or its schema changes, the job may fail.

### [`gcp-big-table-admin-instance`](/sources/gcp/Types/gcp-big-table-admin-instance)

Dataflow jobs that use Bigtable as a source or sink reference the Bigtable instance. If the instance is deleted or misconfigured, the job may fail.

### [`gcp-cloud-kms-crypto-key`](/sources/gcp/Types/gcp-cloud-kms-crypto-key)

When customer-managed encryption keys (CMEK) are enabled for the Dataflow job environment, the job references the Cloud KMS Crypto Key used for encryption.

### [`gcp-compute-network`](/sources/gcp/Types/gcp-compute-network)

Dataflow worker VMs are attached to a VPC network. If the network is deleted or misconfigured, workers may lose connectivity or fail to start.

### [`gcp-compute-subnetwork`](/sources/gcp/Types/gcp-compute-subnetwork)

Dataflow workers run in a specific subnetwork. If the subnetwork is deleted or misconfigured, workers may lose connectivity or fail to start.

### [`gcp-iam-service-account`](/sources/gcp/Types/gcp-iam-service-account)

Dataflow workers run under a service account that grants them permissions to access other GCP services. If the service account is deleted or its permissions change, the job may fail.

### [`gcp-pub-sub-subscription`](/sources/gcp/Types/gcp-pub-sub-subscription)

Dataflow jobs that consume messages from Pub/Sub reference the subscription. If the subscription is deleted or misconfigured, the job may fail to consume messages.

### [`gcp-pub-sub-topic`](/sources/gcp/Types/gcp-pub-sub-topic)

Dataflow jobs that publish to or consume from Pub/Sub reference the topic. If the topic is deleted or misconfigured, the job may fail to read or write messages.

### [`gcp-spanner-instance`](/sources/gcp/Types/gcp-spanner-instance)

Dataflow jobs that use Spanner reference the Spanner instance. If the instance is deleted or misconfigured, the job may fail.
3 changes: 2 additions & 1 deletion docs.overmind.tech/docs/sources/gcp/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ Overmind requires read-only access to discover and map your GCP infrastructure.

**Read-only viewer roles** for GCP services including:

- Compute Engine, GKE, Cloud Run, Cloud Functions
- Compute Engine, GKE, Cloud Run, Cloud Functions, Dataflow
- Cloud SQL, BigQuery, Spanner, Cloud Storage
- IAM, networking, monitoring, and logging resources
- And other GCP services
Expand Down Expand Up @@ -408,6 +408,7 @@ Here are all the predefined GCP roles that Overmind requires, plus the custom ro
| `roles/dataform.viewer` | Dataform resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataform#dataform.viewer) |
| `roles/dataplex.catalogViewer` | Dataplex catalog resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.catalogViewer) |
| `roles/dataplex.viewer` | Dataplex resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataplex#dataplex.viewer) |
| `roles/dataflow.viewer` | Dataflow job discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataflow#dataflow.viewer) |
| `roles/dataproc.viewer` | Dataproc cluster discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dataproc#dataproc.viewer) |
| `roles/dns.reader` | Cloud DNS resource discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/dns#dns.reader) |
| `roles/essentialcontacts.viewer` | Essential Contacts discovery [GCP Docs](https://cloud.google.com/iam/docs/roles-permissions/essentialcontacts#essentialcontacts.viewer) |
Expand Down
31 changes: 31 additions & 0 deletions docs.overmind.tech/docs/sources/gcp/data/gcp-dataflow-job.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"type": "gcp-dataflow-job",
"category": 7,
"potentialLinks": [
"gcp-big-query-dataset",
"gcp-big-query-table",
"gcp-big-table-admin-instance",
"gcp-cloud-kms-crypto-key",
"gcp-compute-network",
"gcp-compute-subnetwork",
"gcp-iam-service-account",
"gcp-pub-sub-subscription",
"gcp-pub-sub-topic",
"gcp-spanner-instance"
],
"descriptiveName": "GCP Dataflow Job",
"supportedQueryMethods": {
"get": true,
"getDescription": "Get a gcp-dataflow-job by its \"locations|jobs\"",
"search": true,
"searchDescription": "Search for gcp-dataflow-job by location"
},
"terraformMappings": [
{
"terraformQueryMap": "google_dataflow_job.job_id"
},
{
"terraformQueryMap": "google_dataflow_flex_template_job.job_id"
}
]
}
105 changes: 53 additions & 52 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,14 @@ require (
cloud.google.com/go/auth v0.18.2
cloud.google.com/go/auth/oauth2adapt v0.2.8
cloud.google.com/go/bigquery v1.74.0
cloud.google.com/go/bigtable v1.42.0
cloud.google.com/go/bigtable v1.43.0
cloud.google.com/go/certificatemanager v1.9.6
cloud.google.com/go/compute v1.57.0
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/container v1.46.0
cloud.google.com/go/dataplex v1.28.0
cloud.google.com/go/dataproc/v2 v2.16.0
cloud.google.com/go/eventarc v1.18.0
cloud.google.com/go/filestore v1.10.3
cloud.google.com/go/functions v1.19.7
cloud.google.com/go/iam v1.5.3
Expand Down Expand Up @@ -60,34 +61,34 @@ require (
github.com/Masterminds/semver/v3 v3.4.0
github.com/MrAlias/otel-schema-utils v0.4.0-alpha
github.com/auth0/go-jwt-middleware/v3 v3.0.0
github.com/aws/aws-sdk-go-v2 v1.41.3
github.com/aws/aws-sdk-go-v2/config v1.32.11
github.com/aws/aws-sdk-go-v2/credentials v1.19.11
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19
github.com/aws/aws-sdk-go-v2/service/apigateway v1.38.6
github.com/aws/aws-sdk-go-v2/service/autoscaling v1.64.2
github.com/aws/aws-sdk-go-v2/service/cloudfront v1.60.2
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.55.1
github.com/aws/aws-sdk-go-v2/service/directconnect v1.38.13
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.56.1
github.com/aws/aws-sdk-go-v2/service/ec2 v1.294.0
github.com/aws/aws-sdk-go-v2/service/ecs v1.73.1
github.com/aws/aws-sdk-go-v2/service/efs v1.41.12
github.com/aws/aws-sdk-go-v2/service/eks v1.81.0
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing v1.33.21
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.54.8
github.com/aws/aws-sdk-go-v2/service/iam v1.53.5
github.com/aws/aws-sdk-go-v2/service/kms v1.50.2
github.com/aws/aws-sdk-go-v2/service/lambda v1.88.2
github.com/aws/aws-sdk-go-v2/service/networkfirewall v1.59.5
github.com/aws/aws-sdk-go-v2/service/networkmanager v1.41.6
github.com/aws/aws-sdk-go-v2/service/rds v1.116.2
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.3
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.0
github.com/aws/aws-sdk-go-v2/service/sns v1.39.13
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.23
github.com/aws/aws-sdk-go-v2/service/ssm v1.68.2
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8
github.com/aws/aws-sdk-go-v2 v1.41.4
github.com/aws/aws-sdk-go-v2/config v1.32.12
github.com/aws/aws-sdk-go-v2/credentials v1.19.12
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.20
github.com/aws/aws-sdk-go-v2/service/apigateway v1.39.0
github.com/aws/aws-sdk-go-v2/service/autoscaling v1.64.3
github.com/aws/aws-sdk-go-v2/service/cloudfront v1.60.3
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.55.2
github.com/aws/aws-sdk-go-v2/service/directconnect v1.38.14
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.56.2
github.com/aws/aws-sdk-go-v2/service/ec2 v1.296.0
github.com/aws/aws-sdk-go-v2/service/ecs v1.74.0
github.com/aws/aws-sdk-go-v2/service/efs v1.41.13
github.com/aws/aws-sdk-go-v2/service/eks v1.81.1
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing v1.33.22
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.54.9
github.com/aws/aws-sdk-go-v2/service/iam v1.53.6
github.com/aws/aws-sdk-go-v2/service/kms v1.50.3
github.com/aws/aws-sdk-go-v2/service/lambda v1.88.3
github.com/aws/aws-sdk-go-v2/service/networkfirewall v1.59.6
github.com/aws/aws-sdk-go-v2/service/networkmanager v1.41.7
github.com/aws/aws-sdk-go-v2/service/rds v1.116.3
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.4
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.1
github.com/aws/aws-sdk-go-v2/service/sns v1.39.14
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24
github.com/aws/aws-sdk-go-v2/service/ssm v1.68.3
github.com/aws/aws-sdk-go-v2/service/sts v1.41.9
github.com/aws/smithy-go v1.24.2
github.com/cenkalti/backoff/v5 v5.0.3
github.com/charmbracelet/glamour v0.10.0
Expand All @@ -97,7 +98,7 @@ require (
github.com/go-jose/go-jose/v4 v4.1.3
github.com/google/btree v1.1.3
github.com/google/uuid v1.6.0
github.com/googleapis/gax-go/v2 v2.18.0
github.com/googleapis/gax-go/v2 v2.19.0
github.com/goombaio/namegenerator v0.0.0-20181006234301-989e774b106e
github.com/hashicorp/go-retryablehttp v0.7.8
github.com/hashicorp/hcl/v2 v2.24.0
Expand All @@ -111,7 +112,7 @@ require (
github.com/miekg/dns v1.1.72
github.com/mitchellh/go-homedir v1.1.0
github.com/muesli/reflow v0.3.0
github.com/nats-io/jwt/v2 v2.8.0
github.com/nats-io/jwt/v2 v2.8.1
github.com/nats-io/nats-server/v2 v2.12.5
github.com/nats-io/nats.go v1.49.0
github.com/nats-io/nkeys v0.4.15
Expand Down Expand Up @@ -147,15 +148,15 @@ require (
golang.org/x/sync v0.20.0
golang.org/x/text v0.35.0
gonum.org/v1/gonum v0.17.0
google.golang.org/api v0.271.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20260311181403-84a4fc48630c
google.golang.org/grpc v1.79.2
google.golang.org/api v0.272.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20260319201613-d00831a3d3e7
google.golang.org/grpc v1.79.3
google.golang.org/protobuf v1.36.11
gopkg.in/ini.v1 v1.67.1
gopkg.in/yaml.v3 v3.0.1
k8s.io/api v0.35.2
k8s.io/apimachinery v0.35.2
k8s.io/client-go v0.35.2
k8s.io/api v0.35.3
k8s.io/apimachinery v0.35.3
k8s.io/client-go v0.35.3
sigs.k8s.io/kind v0.31.0
sigs.k8s.io/structured-merge-diff/v6 v6.3.2 // indirect
)
Expand Down Expand Up @@ -184,19 +185,19 @@ require (
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/apache/arrow/go/v15 v15.0.2 // indirect
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.11.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.7 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.20 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.20 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.21 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.12 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.11.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.20 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.8 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.13 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.17 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
Expand Down Expand Up @@ -340,8 +341,8 @@ require (
golang.org/x/tools v0.42.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20260217215200-42d3e9bedb6d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260226221140-a57be14db171 // indirect
google.golang.org/genproto v0.0.0-20260316180232-0b37fe3546d5 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260316180232-0b37fe3546d5 // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
Expand Down
Loading