Some help would be appreciated

[BlowaXD]

Hi,

Thanks for your plugin. I'm trying to use it over Kubernetes.
We have fluentd forwarders that will push the logs to our logging backends. (Sentry, ElasticSearch)

What I would like to do is something like that :

     <match sentry.**>
        @type sentry
        # Set endpoint API URL
        endpoint_url       http://${record['kubernetes']['annotations']['fluentd.sentry.apikey']}@${record['kubernetes']['annotations']['fluentd.sentry.host']}/${record['kubernetes']['annotations']['fluentd.sentry.projectid']}
  
        # Set default events value of 'server_name'
        # To set short hostname, set like below.
        hostname_command   hostname -s
  
        # rewrite shown tag name for Sentry dashboard
        remove_tag_prefix  sentry.
      </match>

but it looks like endpoint_url can not be built dynamically.

Is it possible to do something like that ?
I'm a complete beginner to fluentd and no one else is able to help me on that project.

Thanks by advance,
Blowa.

[michalszakala]

Hello @BlowaXD , not sure if this is still relevant but you could use environment variables to build the URL dynamically, something like:
endpoint_url "http://#{ENV['SENTRY_KUBE_DSN_KEYS']}@#{ENV['SENTRY_KUBE_HOST']}:#{ENV['SENTRY_KUBE_PORT']}/#{ENV['SENTRY_KUBE_PROJECT_ID']}"

[BlowaXD]

Hello @svakavatott, actually, our fluentd forwarder runs on 1 instance per machine (we are running on Kubernetes), while the dsn needs to be build 1 per service, a machine can run several services...

Thanks for your help

[michalszakala]

I'm not sure I understand your setup.
Let's say you have 3 machines. So you have 3 FluentD forwarders, right? Is it one FluentD service and 3 pods on different machines? Do you want to forward to 3 different Sentry instances/projects from each forwarder?

[BlowaXD]

I'm not sure I understand your setup.
Let's say you have 3 machines. So you have 3 FluentD forwarders, right? Is it one FluentD service and 3 pods on different machines? Do you want to forward to 3 different Sentry instances/projects from each forwarder?

Here is a short description:
3 machines (1 fluentd service per machine)
10 microservices (different projects, not just 10x instances of the same microservice)
1 sentry server

The 10 microservices are orchestrated to run on any machine by Kubernetes

So, as my logs from any microservices are forwarded to their own machine fluent service, I need my fluentd service to forward it to sentry with the right information.
The information about sentry are put in kubernetes service's deployment as labels

Some things I can't do (voluntarily) :

• I can't restrict 1 microservice to 1 kind of microservice
• I can't run multiple fluentd services per machine

[michalszakala]

You can still use FluentD environment variables and reference Sentry endpoint configuration from labels using fieldRef in your Kubernetes Deployment definition.

FluentD config example:
endpoint_url "http://#{ENV['SENTRY_KUBE_DSN_KEYS']}@#{ENV['SENTRY_KUBE_HOST']}:#{ENV['SENTRY_KUBE_PORT']}/#{ENV['SENTRY_KUBE_PROJECT_ID']}"

And your Kubernetes Deployment config:

env:
  - name: SENTRY_KUBE_DSN_KEYS
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['sentry-apikey']
  - name: SENTRY_KUBE_HOST
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['sentry-host']
  - name: SENTRY_KUBE_PORT
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['sentry-port']
  - name: SENTRY_KUBE_PROJECT_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['sentry-projectid']