logstash pipelines 定制化调整

[moooofly]

• Updating Logstash's configuration

[moooofly]

Updating Logstash's configuration

此文内容需要结合 https://github.com/spujadas/elk-docker 一起看

• Logstash 的目录布局
• Logstash 服务自身使用的配置在 /opt/logstash/config 目录下（通过 docker exec -it <elk_container_id> /bin/bash 进入到容器中查看），有
  • jvm.options
  • log4j2.properties
  • pipelines.yml
  • startup.options

其中 pipelines.yml 的内容如下，指明真正定义 pipelines 的配置文件所在位置；

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

• 在 /etc/logstash/conf.d/ 下可以看到如下4个文件
  • 02-beats-input.conf
  • 10-syslog.conf
  • 11-nginx.conf
  • 30-output.conf

进一步确认，可以看到

root@059e43685d39:/etc/logstash/conf.d# cat 02-beats-input.conf
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
  }
}
root@059e43685d39:/etc/logstash/conf.d#


root@059e43685d39:/etc/logstash/conf.d# cat 10-syslog.conf
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
root@059e43685d39:/etc/logstash/conf.d#


root@059e43685d39:/etc/logstash/conf.d# cat 11-nginx.conf
filter {
  if [type] == "nginx-access" {
    grok {
      match => { "message" => "%{NGINXACCESS}" }
    }
  }
}
root@059e43685d39:/etc/logstash/conf.d#


root@059e43685d39:/etc/logstash/conf.d# cat 30-output.conf
output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
root@059e43685d39:/etc/logstash/conf.d#

即 pipelines 是由 input/filter/output 组合在一起构成的；

而上述 pipelines 实现的功能就是：通过 filebeat 读取 syslog 和 nginx-access 类型的日志内容，并进行相应的转换，最终输出到 es 中；

调整 logstash 配置

• 方法一：直接通过 bind-mount 启动容器时，引用新配置

To modify an existing configuration file (be it a high-level Logstash configuration file, or a pipeline configuration file), you can bind-mount a local configuration file to a configuration file within the container at runtime. For instance, if you want to replace the image's 30-output.conf configuration file with your local file /path/to/your-30-output.conf, then you would add the following -v option to your docker command line:

$ sudo docker run ... \
    -v /path/to/your-30-output.conf:/etc/logstash/conf.d/30-output.conf \
    ...

• 方法二：创建包含新配置的 images

To create your own image with updated or additional configuration files, you can create a Dockerfile that extends the original image, with contents such as the following:

FROM sebp/elk

# overwrite existing file
ADD /path/to/your-30-output.conf /etc/logstash/conf.d/30-output.conf

# add new file
ADD /path/to/new-12-some-filter.conf /etc/logstash/conf.d/12-some-filter.conf

之后通过 docker build 构建新的镜像并使用；