Skip to content

Add Application Signals logs pipeline translator#2111

Open
jj22ee wants to merge 9 commits intoaws:mainfrom
jj22ee:appsignals-logs-translator-pr
Open

Add Application Signals logs pipeline translator#2111
jj22ee wants to merge 9 commits intoaws:mainfrom
jj22ee:appsignals-logs-translator-pr

Conversation

@jj22ee
Copy link
Copy Markdown
Contributor

@jj22ee jj22ee commented May 8, 2026

Description of the issue

Application Signals currently collects metrics and traces but not logs. This PR adds a CWAgent translator that generates an OTel logs pipeline to route OTLP logs to CloudWatch via the CW OTLP endpoint, with automatic per-service log group creation.

Customers configure logs.logs_collected.application_signals with optional log_group_name (supporting placeholders like {service.name}) and log_stream_name. The provisioner extension creates log groups/streams on first encounter.

Description of changes

Adds a new pipeline translator at translator/translate/otel/pipeline/applicationsignalslogs/ that produces two pipeline shapes:

Dynamic (placeholders in log_group_name/log_stream_name):

  • Auth chain: otlphttp → headers_setter → awscloudwatchlogsprovisioner → sigv4auth
  • Processors: transform → attributestocontext → batch(metadata_keys)

Static (no placeholders):

  • Auth chain: otlphttp → awscloudwatchlogsprovisioner → sigv4auth
  • Processors: batch
  • Static x-aws-log-group/x-aws-log-stream headers on the exporter

Notable behaviors:

  • Multiple placeholders supported in both group and stream (e.g. /{deployment.environment}/prefix/{service.name})
  • Customer-set resource attributes take priority over template (OTTL where guard)
  • AutoEnableIfNeeded auto-opts-in the logs pipeline when metrics_collected.application_signals is configured

New/Updated shared translator packages:

  • extension/awscloudwatchlogsprovisioner — new
  • extension/headerssetter — new
  • processor/attributestocontext — new
  • exporter/otlphttp — new (generic, signal-aware EndpointConfig)
  • extension/sigv4auth — Updated with WithService option
  • processor/batchprocessor — Updated with WithMetadataKeys option
  • processor/transformprocessor — Updated with NewTranslatorWithLogStatements

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Unit tests
  • Integration tests in amazon-cloudwatch-agent-test: <TODO: Make PR>

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@jj22ee jj22ee requested a review from a team as a code owner May 8, 2026 00:09
@jj22ee jj22ee force-pushed the appsignals-logs-translator-pr branch from 83b5680 to 9027c02 Compare May 8, 2026 00:12
@jj22ee jj22ee changed the title Add Translation logic for logs.logs_collected.application_signals Add Application Signals logs pipeline translator with dynamic log group routing May 8, 2026
@jj22ee jj22ee changed the title Add Application Signals logs pipeline translator with dynamic log group routing Add Application Signals logs pipeline translator May 8, 2026
Comment thread go.mod
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go.mod will be updated since this PR depends on:

Comment on lines +53 to +54
// TODO: Update default log group prefix before PR is merged.
defaultLogGroupPrefix = "/aws/telemetry/"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still pending and not final.

defaultLogStreamName = "default"

metadataKeyLogGroup = "aws.cloudwatch.log_group.destination"
metadataKeyLogStream = "aws.cloudwatch.log_stream.destination"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These key values are not following any otel spec, they're just arbitrary attributes created by Transform Processor which are used by AttributesToContext Processor, so we don't expect customers to know about these keys or even set them in their exported logs.

However, if customer does know and set these these attributes in their logs, we'll respect it and won't override them.

@jj22ee jj22ee force-pushed the appsignals-logs-translator-pr branch from 51f3c5d to d454f8f Compare May 8, 2026 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant