feat: migrated to updated transport interface

.gitignore

@@ -2,3 +2,6 @@
 /.php-cs-fixer.cache
 /vendor
 /composer.lock
+/.phpunit.result.cache
+/coverage.xml
+/docker/.env

README.md

@@ -13,9 +13,17 @@ This is not an official OpenTelemetry project. This repository contains a Shopwa
 composer require shopware/opentelemetry
 ```
 
+You also need to install the OpenTelemetry SDK and selected exporter and transport. Below is an example with OTLP over gRPC:
+
+```bash
+composer require open-telemetry/sdk
+composer require open-telemetry/exporter-otlp
+composer require open-telemetry/transport-grpc
+```
+
 ## Configuration
 
-Enable open telemetry with the following environment variables:
+Enable OpenTelemetry SDK with the following environment variables:
 
 ```bash
 OTEL_PHP_AUTOLOAD_ENABLED=true
@@ -37,8 +45,6 @@ OTEL_EXPORTER_OTLP_PROTOCOL=grpc
 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 ```
 
-You will need to install also composer packages: `open-telemetry/transport-grpc` and `open-telemetry/exporter-otlp`.
-
 ### Enabling Shopware custom tracing
 
 To enable tracing for Shopware, you need to add the following config:
@@ -83,18 +89,20 @@ monolog:
 
 ## Transport metrics to OpenTelemetry
 
-You can enable OpenTelemetry metrics transport with the following configuration:
+Shopware core collects metrics through its telemetry abstraction. This bundle
+registers a transport that sends those metrics to OpenTelemetry.
+
+Enable the transport with the following configuration:
 
 ```yaml
 # config/packages/opentelemetry.yaml
 open_telemetry_shopware:
-  metrics:
-    enabled: true
-    namespace: 'io.opentelemetry.contrib.php.shopware' # or your custom namespace
+    metrics:
+        enabled: true
 ```
 
-Please note that OpenTelemetry SDK has to be configured to send metrics to the collector.
-It is configured using the same environment variables. Example configuration could look like this:
+The OpenTelemetry SDK must be configured to send metrics to the collector. It
+uses the same environment variables as tracing. Example:
 
 ```bash
 OTEL_SERVICE_NAME=shopware
@@ -105,6 +113,41 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
 ```
 
+Metrics are only emitted when the `TELEMETRY_METRICS` feature flag is enabled in
+Shopware. Since Shopware 6.7.12, `shopware.telemetry.metrics.enabled` must be
+`true` as well.
+
+### Metric names
+
+Metric names are prefixed with the namespace configured in Shopware core (`shopware.telemetry.metrics.namespace`). 
+Shopware core passes it to transports since version 6.7.12. On older versions the prefix
+`io.opentelemetry.contrib.php.shopware` is used.
+
+This bundle has no namespace option of its own. The option `open_telemetry_shopware.metrics.namespace`
+from `0.2.0-alpha` is deprecated and ignored. It will be removed in 1.0.
+
+### Flushing
+
+Since Shopware 6.7.12, core calls `flush()` on every transport at the end of each request
+and console command, and about every 60 seconds inside a running Messenger worker. This transport
+then exports the collected metrics immediately, so metrics from long-running workers
+do not wait until the worker stops.
+
+On older Shopware versions the metrics are exported once, when the PHP process shuts down.
+
+The method `OpenTelemetryMetricTransport::forceFlush()` is deprecated. Use `flush()` instead.
+
+### Shopware compatibility
+
+| Shopware version   | Behavior                                                                 |
+|--------------------|--------------------------------------------------------------------------|
+| below 6.6.7        | No metrics support in core. Tracing and log forwarding still work.        |
+| 6.6.7 to 6.7.11    | Metrics are exported when the PHP process shuts down. Metric names are prefixed with `io.opentelemetry.contrib.php.shopware`. |
+| 6.7.12 and later   | Metrics are flushed by core (see above). Metric names are prefixed with the namespace from Shopware core. |
+
+The telemetry abstraction in Shopware core is experimental until Shopware 6.8.
+This bundle stays at version `0.x` until then.
+
 ### Temporality configuration
 OpenTelemetry PHP SDK does not support storage for accumulation of metrics. As PHP processes are short-lived, 
 it's best to emit metrics with delta temporality and aggregate them on receiving side. Unfortunately at the moment of writing this

docker/.env.example

@@ -0,0 +1,3 @@
+# Copy to docker/.env and fill in your values
+DD_SITE=datadoghq.eu
+DD_API_KEY=

docker/compose.yml

@@ -1,4 +1,3 @@
-version: "3"
 services:
   # And put them in an OTEL collector pipeline...
   otel-collector:
@@ -16,7 +15,9 @@ services:
 
   # To eventually offload to Tempo...
   tempo:
-    image: grafana/tempo:latest
+    # Pin 2.x.x. Docker Hub has no rolling :2 or :2.10 tag — only full versions.
+    # Bump this patch when Grafana ships a new 2.x release.
+    image: grafana/tempo:2.10.4
     command: [ "-config.file=/etc/tempo.yaml" ]
     stop_signal: SIGKILL
     volumes:
@@ -29,6 +30,8 @@ services:
 
   prometheus:
     image: prom/prometheus:latest
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     command:
       - --config.file=/etc/prometheus.yaml
       - --web.enable-remote-write-receiver

docker/loki.yaml

@@ -23,12 +23,16 @@ query_range:
         enabled: true
         max_size_mb: 100
 
+limits_config:
+  # Required for OTLP log ingestion (see Loki OTEL docs).
+  allow_structured_metadata: true
+
 schema_config:
   configs:
     - from: 2020-10-24
-      store: boltdb-shipper
+      store: tsdb
       object_store: filesystem
-      schema: v11
+      schema: v13
       index:
         prefix: index_
         period: 24h

docker/otel-collector.yaml

@@ -6,9 +6,8 @@ receivers:
       http:
         endpoint: 0.0.0.0:4318
 exporters:
-  logging:
   debug:
-  otlp/tempo:
+  otlp_grpc/tempo:
     endpoint: tempo:4317
     tls:
       insecure: true
@@ -17,8 +16,10 @@ exporters:
   #   endpoint: dd-agent:4317
   #   tls:
   #     insecure: true
-  loki:
-    endpoint: http://loki:3100/loki/api/v1/push
+  # Loki removed the legacy push API exporter from the collector; use native OTLP.
+  # https://grafana.com/docs/loki/latest/send-data/otel/
+  otlp_http/loki:
+    endpoint: http://loki:3100/otlp
   prometheus:
     endpoint: 0.0.0.0:8889
     # adding labels to the metrics
@@ -30,23 +31,22 @@ exporters:
     metric_expiration: 180m
   # exporter to datadog using datadog api
   # for configuration example see https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/collector.yaml
-  datadog:
-    api:
-      site: ${env:DD_SITE}
-      key: ${env:DD_API_KEY}
-    metrics:
-      resource_attributes_as_tags: true
-      histograms:
-        mode: counters
-        send_aggregation_metrics: true
-    traces:
-      # trace_buffer: 10
+  # datadog:
+  #   api:
+  #     site: ${env:DD_SITE}
+  #     key: ${env:DD_API_KEY}
+  #   metrics:
+  #     resource_attributes_as_tags: true
+  #     histograms:
+  #       mode: counters
+  #       send_aggregation_metrics: true
+  #   traces:
+  #     # trace_buffer: 10
 
 processors:
   batch:
   deltatocumulative:
     max_stale: 60m
-    max_streams: 0
 
 
 service:
@@ -58,15 +58,15 @@ service:
       receivers: [otlp]
       # if you want to export through datadog agent, not to api directly, use otlp/datadog instead of datadog
       # in such case otlp/datadog exporter have to be enabled in exporters section
-      exporters: [otlp/tempo, debug, datadog]
+      exporters: [otlp_grpc/tempo, debug]
     logs:
       receivers: [otlp]
-      exporters: [loki, logging]
-    metrics/delta:
-      receivers: [otlp]
-      # if you want to export through datadog agent, not to api directly, use otlp/datadog instead of datadog
-      # in such case otlp/datadog exporter have to be enabled in exporters section
-      exporters: [debug, datadog]
+      exporters: [otlp_http/loki, debug]
+    # metrics/delta:
+    #   receivers: [otlp]
+    #   # if you want to export through datadog agent, not to api directly, use otlp/datadog instead of datadog
+    #   # in such case otlp/datadog exporter have to be enabled in exporters section
+    #   exporters: [debug, datadog]
     metrics/cumulative:
       receivers: [otlp]
       processors: [deltatocumulative]

docker/prometheus.yaml

@@ -12,3 +12,8 @@ scrape_configs:
   - job_name: 'otel-collector'
     static_configs:
       - targets: [ 'otel-collector:8889' ]
+  - job_name: 'shopware_prom_exp'
+    metrics_path: '/api/_internal/prometheus'
+    static_configs:
+      - targets: [ 'host.docker.internal:8000' ]
+

docker/tempo.yaml

@@ -23,7 +23,9 @@ distributor:
     otlp:
       protocols:
         http:
+          endpoint: 0.0.0.0:4318
         grpc:
+          endpoint: 0.0.0.0:4317
     opencensus:
 
 ingester:

phpstan.neon.dist

@@ -7,3 +7,10 @@ parameters:
     reportUnmatchedIgnoredErrors: true
     excludePaths:
     ignoreErrors:
+        # Feature::metricNamespaceSupported() asserts TransportConfig::$namespace, which exists in
+        # shopware/core >= 6.7.12 only. Against a newer core the assertion is redundant, against an
+        # older core it is required, so the report is ignored without being mandatory.
+        -
+            message: '#^Asserted type .*TransportConfig.* does not narrow down the type\.$#'
+            path: src/Feature.php
+            reportUnmatched: false

src/Feature.php

@@ -4,10 +4,23 @@
 
 namespace Shopware\OpenTelemetry;
 
+use Shopware\Core\Framework\Telemetry\Metrics\Config\TransportConfig;
+
 class Feature
 {
     public static function metricsSupported(): bool
     {
         return class_exists('Shopware\Core\Framework\Telemetry\Metrics\Metric\Metric');
     }
+
+    /**
+     * shopware/core >= 6.7.12 passes `shopware.telemetry.metrics.namespace` to transports via TransportConfig.
+     * The assertion tells PHPStan that the property exists when analysing against an older core.
+     *
+     * @phpstan-assert-if-true TransportConfig&object{namespace: string|null} $transportConfig
+     */
+    public static function metricNamespaceSupported(TransportConfig $transportConfig): bool
+    {
+        return property_exists($transportConfig, 'namespace');
+    }
 }

src/Metrics/Transports/OpenTelemetryMetricTransport.php

@@ -20,9 +20,9 @@ public function __construct(
         // careful: if link to meterProvider is removed, SDK will not transmit metrics (it uses weak references)
         private MeterProviderInterface $meterProvider,
         private MetricNameFormatter $formatter,
-        private string $namespace,
+        private string $instrumentationScope,
     ) {
-        $this->meter = $this->meterProvider->getMeter($this->namespace);
+        $this->meter = $this->meterProvider->getMeter($this->instrumentationScope);
     }
 
 
@@ -54,10 +54,22 @@ public function emit(Metric $metric): void
         }
     }
 
+    public function flush(): void
+    {
+        if ($this->meterProvider instanceof \OpenTelemetry\SDK\Metrics\MeterProviderInterface) {
+            $this->meterProvider->forceFlush();
+        }
+    }
+
+    /**
+     * @deprecated - use flush instead
+     */
     public function forceFlush(): bool
     {
         if ($this->meterProvider instanceof \OpenTelemetry\SDK\Metrics\MeterProviderInterface) {
-            return $this->meterProvider->forceFlush();
+            $this->flush();
+
+            return true;
         }
 
         return false;

src/Metrics/Transports/OpenTelemetryMetricTransportFactory.php

@@ -8,17 +8,32 @@
 use Shopware\Core\Framework\Telemetry\Metrics\Factory\MetricTransportFactoryInterface;
 use Shopware\Core\Framework\Telemetry\Metrics\Metric\Type;
 use Shopware\Core\Framework\Telemetry\Metrics\MetricTransportInterface;
+use Shopware\OpenTelemetry\Feature;
 use Shopware\OpenTelemetry\Metrics\MetricNameFormatter;
 
 class OpenTelemetryMetricTransportFactory implements MetricTransportFactoryInterface
 {
+    /**
+     * Name of the OpenTelemetry instrumentation scope that owns the metrics.
+     * Same convention as the tracing instrumentations shipped in this bundle.
+     */
+    public const INSTRUMENTATION_SCOPE = 'io.opentelemetry.contrib.php.shopware';
+
+    /**
+     * Metric name prefix used with shopware/core < 6.7.12, where core does not pass a namespace to transports.
+     * Intentionally equal to the old default prefix name to limit prefix changes on updates at least for
+     * installations with default values.
+     */
+    public const LEGACY_NAMESPACE = 'io.opentelemetry.contrib.php.shopware';
+
     public function __construct(
-        private readonly string $namespace,
-        private MetricNameFormatter $formatter,
-        private OpenTelemetryMeterProviderFactory $meterProviderFactory,
+        private readonly OpenTelemetryMeterProviderFactory $meterProviderFactory,
     ) {}
+
     public function create(TransportConfig $transportConfig): MetricTransportInterface
     {
+        $formatter = new MetricNameFormatter($this->resolveNamespace($transportConfig));
+
         $buckets = [];
         foreach ($transportConfig->metricsConfig as $metric) {
             if (!$metric->enabled) {
@@ -30,7 +45,7 @@ public function create(TransportConfig $transportConfig): MetricTransportInterfa
                 && \is_array($metric->parameters['buckets'])
                 && \count($metric->parameters['buckets']) > 0
             ) {
-                $metricName = $this->formatter->format($metric->name);
+                $metricName = $formatter->format($metric->name);
 
                 assert(!empty($metricName));
                 $buckets[$metricName] = $metric->parameters['buckets'];
@@ -39,6 +54,20 @@ public function create(TransportConfig $transportConfig): MetricTransportInterfa
 
         $meterProvider = $this->meterProviderFactory->createMeterProvider($buckets);
 
-        return new OpenTelemetryMetricTransport($meterProvider, $this->formatter, $this->namespace);
+        return new OpenTelemetryMetricTransport($meterProvider, $formatter, self::INSTRUMENTATION_SCOPE);
+    }
+
+    /**
+     * The metric name prefix is `shopware.telemetry.metrics.namespace`, passed by Shopware core since 6.7.12.
+     * A null value from core means "no prefix". Older core versions do not pass a namespace at all,
+     * they get self::LEGACY_NAMESPACE.
+     */
+    private function resolveNamespace(TransportConfig $transportConfig): string
+    {
+        if (!Feature::metricNamespaceSupported($transportConfig)) {
+            return self::LEGACY_NAMESPACE;
+        }
+
+        return \is_string($transportConfig->namespace) ? $transportConfig->namespace : '';
     }
 }