Filebeat

Filebeat is probably the most popular beat of all the beats provided by Elastic. The repo is very large containing over 1M+ lines of code. I will gradually add learning notes about this repo.

Metrics

Metrics are useful to debug filebeat related issues. Filebeat periodically logs metrics. See one example

2023-02-17T16:05:04.736Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31680,"time":{"ms":12}},"total":{"ticks":98500,"time":{"ms":66},"value":98500},"user":{"ticks":66820,"time":{"ms":54}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":15},"info":{"ephemeral_id":"7f8db87b-b600-4077-a904-316c6724484b","uptime":{"ms":43920282}},"memstats":{"gc_next":27294768,"memory_alloc":15822752,"memory_total":11222196024},"runtime":{"goroutines":64}},"filebeat":{"events":{"added":65,"done":65},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":65,"batches":27,"total":65}},"outputs":{"kafka":{"bytes_read":2756,"bytes_write":51664}},"pipeline":{"clients":1,"events":{"active":2,"published":65,"total":65},"queue":{"acked":65}}},"registrar":{"states":{"current":2,"update":65},"writes":{"success":27,"total":27}},"system":{"load":{"1":0.33,"15":0.7,"5":0.37,"norm":{"1":0.0825,"15":0.175,"5":0.0925}}}}}}

Let’s focus on the libbeat.output.events part:

"output": {
  "events": {
    "acked": 65,
    "batches": 27,
    "total": 65
  }
}

The corresponding code definition is here. If something goes wrong, then we will see a key failed inside events. Specifically, for kafka output, you will see that for each deliver error, it adds the error count to the stats.