开发 filebeat http output 插件

[moooofly]

ref: https://www.elastic.co/guide/en/beats/devguide/6.6/new-beat.html

[moooofly]

Overview

At the high level, a simple Beat has two main components:

• a component that collects the actual data, and
• a publisher that sends the data to the specified output, such as Elasticsearch or Logstash.

The publisher is already implemented in libbeat, so you typically only have to worry about the logic specific to your Beat (the code that creates the event and sends it to the publisher). Libbeat also offers common services like configuration management, logging, daemonzing, and Windows service handling, and data processing modules.

The event that you create is a JSON-like object (Go type map[string]interface{}) that contains the collected data to send to the publisher. At a minimum, the event object must contain a @timestamp field and a type field. Beyond that, events can contain any additional fields, and they can be created as often as necessary.

Generating Your Beat

To generate your own Beat, you use the Beat generator available in the beats repo on GitHub.

Python 2 is required (Python 3 will not work).

Run python and specify the path to the Beat generator:

[#529#root@ubuntu-1604 /go/src/github.com/moooofly]$python $GOPATH/src/github.com/elastic/beats/script/generate.py
Beat Name [Examplebeat]: Httpbeat
Your Github Name [your-github-name]: moooofly
Beat Path [github.com/moooofly/httpbeat]:
Firstname Lastname: Fei Sun
[#530#root@ubuntu-1604 /go/src/github.com/moooofly]$

自动生成如下内容

[#531#root@ubuntu-1604 /go/src/github.com/moooofly]$ll httpbeat/
total 76
drwxr-xr-x  8 root root 4096 Mar 13 15:32 ./
drwxr-xr-x 19 root root 4096 Mar 13 15:32 ../
drwxr-xr-x  2 root root 4096 Mar 13 15:32 beater/
drwxr-xr-x  2 root root 4096 Mar 13 15:32 cmd/
drwxr-xr-x  2 root root 4096 Mar 13 15:32 config/
-rw-r--r--  1 root root    0 Mar 13 15:32 CONTRIBUTING.md
drwxr-xr-x  2 root root 4096 Mar 13 15:32 docs/
-rw-r--r--  1 root root  362 Mar 13 15:32 .editorconfig
-rw-r--r--  1 root root   56 Mar 13 15:32 .gitignore
-rw-r--r--  1 root root  552 Mar 13 15:32 LICENSE.txt
-rw-r--r--  1 root root 3228 Mar 13 15:32 magefile.go
-rw-r--r--  1 root root  191 Mar 13 15:32 main.go
-rw-r--r--  1 root root  599 Mar 13 15:32 main_test.go
-rw-r--r--  1 root root  326 Mar 13 15:32 make.bat
-rw-r--r--  1 root root 1268 Mar 13 15:32 Makefile
drwxr-xr-x  2 root root 4096 Mar 13 15:32 _meta/
-rw-r--r--  1 root root  134 Mar 13 15:32 NOTICE.txt
-rw-r--r--  1 root root 2484 Mar 13 15:32 README.md
drwxr-xr-x  3 root root 4096 Mar 13 15:32 tests/
-rw-r--r--  1 root root  862 Mar 13 15:32 .travis.yml
[#532#root@ubuntu-1604 /go/src/github.com/moooofly]$

You now have a raw template of the Beat, but you still need to fetch dependencies and set up the Beat.

Fetching Dependencies and Setting up the Beat

First you need to install the following tools:

• Python
• virtualenv -- apt install virtualenv

To fetch dependencies and set up the Beat, run:

cd ${GOPATH}/src/github.com/moooofly/httpbeat
make setup

错误一：Makefile 编写的问题，执行第一次时，环境变量 ES_BEATS 为空值，第二次执行时就有值了

[#536#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$make setup
mkdir -p vendor/github.com/elastic
cp -R /go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git vendor/github.com/elastic/beats/x-pack
mkdir -p vendor/github.com/magefile
cp -R /go/src/github.com/elastic/beats/vendor/github.com/magefile/mage vendor/github.com/magefile
git init
Initialized empty Git repository in /go/src/github.com/moooofly/httpbeat/.git/
make: *** No rule to make target 'update', needed by 'setup'.  Stop.
[#537#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$

错误二：实现没有安装 virtualenv

[#543#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$make setup
mkdir -p vendor/github.com/elastic
cp -R /go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git vendor/github.com/elastic/beats/x-pack
mkdir -p vendor/github.com/magefile
cp -R /go/src/github.com/elastic/beats/vendor/github.com/magefile/mage vendor/github.com/magefile
git init
Reinitialized existing Git repository in /go/src/github.com/moooofly/httpbeat/.git/
bash: virtualenv: command not found
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:247: recipe for target 'python-env' failed
make: *** [python-env] Error 127
[#544#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$

The Beat now contains the basic config file, httpbeat.yml, and template files. The Beat is "complete" in the sense that you can compile and run it. However, to make it functionally complete, you need to add your custom logic (see The Beater Interface), along with any additional configuration parameters that your Beat requires.

到此，已经得到了一个可以编译运行的完整模板文件，接下来只要编写逻辑代码就可以了；

Building and Running the Beat

To compile the Beat, make sure you are in the Beat directory and run:

make

Now run the Beat:

./httpbeat -e -d "*"

The command automatically loads the default config file, httpbeat.yml, and sends debug output to the console.

You can stop the Beat by pressing Ctrl+C.

[#565#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$./httpbeat -e -d "*"
2019-03-13T17:02:47.387+0800	INFO	instance/beat.go:616	Home path: [/go/src/github.com/moooofly/httpbeat] Config path: [/go/src/github.com/moooofly/httpbeat] Data path: [/go/src/github.com/moooofly/httpbeat/data] Logs path: [/go/src/github.com/moooofly/httpbeat/logs]
2019-03-13T17:02:47.387+0800	DEBUG	[beat]	instance/beat.go:653	Beat metadata path: /go/src/github.com/moooofly/httpbeat/data/meta.json
2019-03-13T17:02:47.387+0800	INFO	instance/beat.go:623	Beat UUID: ac05795d-8f29-4321-8fd4-3210da68b2db
2019-03-13T17:02:47.387+0800	DEBUG	[seccomp]	seccomp/seccomp.go:109	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2019-03-13T17:02:47.387+0800	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-03-13T17:02:47.387+0800	INFO	[beat]	instance/beat.go:936	Beat info	{"system_info": {"beat": {"path": {"config": "/go/src/github.com/moooofly/httpbeat", "data": "/go/src/github.com/moooofly/httpbeat/data", "home": "/go/src/github.com/moooofly/httpbeat", "logs": "/go/src/github.com/moooofly/httpbeat/logs"}, "type": "httpbeat", "uuid": "ac05795d-8f29-4321-8fd4-3210da68b2db"}}}
2019-03-13T17:02:47.387+0800	INFO	[beat]	instance/beat.go:945	Build info	{"system_info": {"build": {"commit": "58c49c40f3c770b5e5afe820cb69b041cbb51241", "libbeat": "6.6.3", "time": "2019-03-13T08:53:13.000Z", "version": "6.6.3"}}}
2019-03-13T17:02:47.387+0800	INFO	[beat]	instance/beat.go:948	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.11.1"}}}
2019-03-13T17:02:47.388+0800	INFO	[beat]	instance/beat.go:952	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-03T07:50:26+08:00","containerized":false,"name":"ubuntu-1604","ip":["127.0.0.1/8","::1/128","10.0.2.15/24","fe80::14:68ff:fe3e:8447/64","11.11.11.12/24","172.17.0.1/16"],"kernel_version":"4.4.0-141-generic","mac":["02:14:68:3e:84:47","08:00:27:bf:25:a3","02:42:8a:36:f2:54"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.3 LTS (Xenial Xerus)","major":16,"minor":4,"patch":3,"codename":"xenial"},"timezone":"CST","timezone_offset_sec":28800,"id":"36048e2ecb0f4be19d57697a529c5d3e"}}}
2019-03-13T17:02:47.389+0800	INFO	[beat]	instance/beat.go:981	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/go/src/github.com/moooofly/httpbeat", "exe": "/go/src/github.com/moooofly/httpbeat/httpbeat", "name": "httpbeat", "pid": 28551, "ppid": 3472, "seccomp": {"mode":"filter"}, "start_time": "2019-03-13T17:02:46.570+0800"}}}
2019-03-13T17:02:47.389+0800	INFO	instance/beat.go:281	Setup Beat: httpbeat; Version: 6.6.3
2019-03-13T17:02:47.389+0800	DEBUG	[beat]	instance/beat.go:302	Initializing output plugins
2019-03-13T17:02:47.390+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:160	add_cloud_metadata: starting to fetch metadata, timeout=3s
2019-03-13T17:02:47.422+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for qcloud after 31.618035ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 10.0.2.3:53: no such host, metadata={}]
2019-03-13T17:02:50.391+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for ec2 after 3.00082132s. result=[provider:ec2, error=failed requesting ec2 metadata: Get http://169.254.169.254/2014-02-25/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for gce after 3.000963834s. result=[provider:gce, error=failed requesting gce metadata: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for az after 3.001104536s. result=[provider:az, error=failed requesting az metadata: Get http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for digitalocean after 3.001183266s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:192	add_cloud_metadata: received disposition for openstack after 3.001291458s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/instance-id: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.392+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:199	add_cloud_metadata: timed-out waiting for all responses
2019-03-13T17:02:50.392+0800	DEBUG	[filters]	add_cloud_metadata/add_cloud_metadata.go:163	add_cloud_metadata: fetchMetadata ran for 3.00138628s
2019-03-13T17:02:50.392+0800	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-03-13T17:02:50.392+0800	DEBUG	[processors]	processors/processor.go:66	Processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2019-03-13T17:02:50.392+0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-03-13T17:02:50.393+0800	DEBUG	[publish]	pipeline/consumer.go:137	start pipeline event consumer
2019-03-13T17:02:50.393+0800	INFO	[publisher]	pipeline/module.go:110	Beat name: ubuntu-1604
2019-03-13T17:02:50.393+0800	INFO	instance/beat.go:403	httpbeat start running.
2019-03-13T17:02:50.393+0800	INFO	beater/httpbeat.go:37	httpbeat is running! Hit CTRL-C to stop it.
2019-03-13T17:02:50.393+0800	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-03-13T17:02:51.395+0800	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-03-13T09:02:51.395Z",
  "@metadata": {
    "beat": "httpbeat",
    "type": "doc",
    "version": "6.6.3"
  },
  "type": "ubuntu-1604",
  "beat": {
    "name": "ubuntu-1604",
    "hostname": "ubuntu-1604",
    "version": "6.6.3"
  },
  "host": {
    "containerized": false,
    "architecture": "x86_64",
    "os": {
      "name": "Ubuntu",
      "codename": "xenial",
      "platform": "ubuntu",
      "version": "16.04.3 LTS (Xenial Xerus)",
      "family": "debian"
    },
    "id": "36048e2ecb0f4be19d57697a529c5d3e",
    "name": "ubuntu-1604"
  },
  "counter": 1
}
2019-03-13T17:02:51.395+0800	INFO	beater/httpbeat.go:62	Event sent
2019-03-13T17:02:52.395+0800	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-03-13T09:02:52.395Z",
  "@metadata": {
    "beat": "httpbeat",
    "type": "doc",
    "version": "6.6.3"
  },
  "type": "ubuntu-1604",
  "counter": 2,
  "beat": {
    "hostname": "ubuntu-1604",
    "version": "6.6.3",
    "name": "ubuntu-1604"
  },
  "host": {
    "containerized": false,
    "architecture": "x86_64",
    "name": "ubuntu-1604",
    "os": {
      "version": "16.04.3 LTS (Xenial Xerus)",
      "family": "debian",
      "name": "Ubuntu",
      "codename": "xenial",
      "platform": "ubuntu"
    },
    "id": "36048e2ecb0f4be19d57697a529c5d3e"
  }
}
2019-03-13T17:02:52.395+0800	INFO	beater/httpbeat.go:62	Event sent
2019-03-13T17:02:52.396+0800	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://localhost:9200))
2019-03-13T17:02:52.396+0800	DEBUG	[elasticsearch]	elasticsearch/client.go:697	ES Ping(url=http://localhost:9200)
2019-03-13T17:02:52.397+0800	DEBUG	[elasticsearch]	elasticsearch/client.go:701	Ping request failed with: Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:53.395+0800	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-03-13T09:02:53.395Z",
  "@metadata": {
    "beat": "httpbeat",
    "type": "doc",
    "version": "6.6.3"
  },
  "type": "ubuntu-1604",
  "counter": 3,
  "beat": {
    "name": "ubuntu-1604",
    "hostname": "ubuntu-1604",
    "version": "6.6.3"
  },
  "host": {
    "name": "ubuntu-1604",
    "id": "36048e2ecb0f4be19d57697a529c5d3e",
    "containerized": false,
    "architecture": "x86_64",
    "os": {
      "platform": "ubuntu",
      "version": "16.04.3 LTS (Xenial Xerus)",
      "family": "debian",
      "name": "Ubuntu",
      "codename": "xenial"
    }
  }
}
2019-03-13T17:02:53.396+0800	INFO	beater/httpbeat.go:62	Event sent
2019-03-13T17:02:54.235+0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:54.235+0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)
2019-03-13T17:02:54.235+0800	DEBUG	[elasticsearch]	elasticsearch/client.go:697	ES Ping(url=http://localhost:9200)
2019-03-13T17:02:54.235+0800	DEBUG	[elasticsearch]	elasticsearch/client.go:701	Ping request failed with: Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:54.395+0800	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-03-13T09:02:54.395Z",
  "@metadata": {
    "beat": "httpbeat",
    "type": "doc",
    "version": "6.6.3"
  },
  "host": {
    "containerized": false,
    "architecture": "x86_64",
    "os": {
      "codename": "xenial",
      "platform": "ubuntu",
      "version": "16.04.3 LTS (Xenial Xerus)",
      "family": "debian",
      "name": "Ubuntu"
    },
    "name": "ubuntu-1604",
    "id": "36048e2ecb0f4be19d57697a529c5d3e"
  },
  "type": "ubuntu-1604",
  "counter": 4,
  "beat": {
    "name": "ubuntu-1604",
    "hostname": "ubuntu-1604",
    "version": "6.6.3"
  }
}
2019-03-13T17:02:54.395+0800	INFO	beater/httpbeat.go:62	Event sent
2019-03-13T17:02:55.394+0800	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-03-13T09:02:55.394Z",
  "@metadata": {
    "beat": "httpbeat",
    "type": "doc",
    "version": "6.6.3"
  },
  "counter": 5,
  "beat": {
    "hostname": "ubuntu-1604",
    "version": "6.6.3",
    "name": "ubuntu-1604"
  },
  "host": {
    "name": "ubuntu-1604",
    "os": {
      "platform": "ubuntu",
      "version": "16.04.3 LTS (Xenial Xerus)",
      "family": "debian",
      "name": "Ubuntu",
      "codename": "xenial"
    },
    "id": "36048e2ecb0f4be19d57697a529c5d3e",
    "containerized": false,
    "architecture": "x86_64"
  },
  "type": "ubuntu-1604"
}
2019-03-13T17:02:55.395+0800	INFO	beater/httpbeat.go:62	Event sent
^C2019-03-13T17:02:55.849+0800	DEBUG	[service]	service/service.go:50	Received sigterm/sigint, stopping
2019-03-13T17:02:55.849+0800	DEBUG	[publish]	pipeline/client.go:148	client: closing acker
2019-03-13T17:02:55.849+0800	DEBUG	[publish]	pipeline/client.go:150	client: done closing acker
2019-03-13T17:02:55.849+0800	DEBUG	[publish]	pipeline/client.go:154	client: cancelled 0 events
2019-03-13T17:02:55.851+0800	INFO	[monitoring]	log/log.go:152	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":4}},"total":{"ticks":20,"time":{"ms":32},"value":0},"user":{"ticks":20,"time":{"ms":28}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":5},"info":{"ephemeral_id":"35fbc8ce-486d-4a6b-98c3-14e0161d6d95","uptime":{"ms":8468}},"memstats":{"gc_next":4194304,"memory_alloc":2101912,"memory_total":4043280,"rss":25067520}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":5,"published":5,"retry":2,"total":5}}},"system":{"cpu":{"cores":2},"load":{"1":0.06,"15":0.04,"5":0.06,"norm":{"1":0.03,"15":0.02,"5":0.03}}}}}}
2019-03-13T17:02:55.851+0800	INFO	[monitoring]	log/log.go:153	Uptime: 8.469085575s
2019-03-13T17:02:55.851+0800	INFO	[monitoring]	log/log.go:130	Stopping metrics logging.
2019-03-13T17:02:55.851+0800	INFO	instance/beat.go:413	httpbeat stopped.
[#566#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$