kafka-consuemr-group.sh is a script released together with Kafka that helps users to inspect or change consumers. The operations it supports are ListGroups, DescribeGroups, DeleteGroups, ResetOffs...

There are tons of parameters to configure Kafka. The configuration can be classified into 3 buckets: broker configurations, producer configurations and consumer configurations. Broker configuratio...

Deep diving into TLS is an item in my TODO list for a long time. But as you know, procrastination is not easy to overcome. The opportunity finally comes: I need to set up mTLS for a webhook endpoin...

Why linkers? I became interested in linker by reading Ian Lance Taylor’s blog series. He was writing a new linker gold for binutils-gdb. There are lots of things I do not understand from his blogs...

We know that Kafka metadata such as locations of partitions is stored in Zookeeper. Out of curiosity, I decide to take a look inside Zookeeper. However, this seems not as simple as I thought. Bef...

Authentication and authorization are two different concepts. Authentication answers the question “who can log in to the system?” Authorization answers what you can do after login. Authentication is...

Debezium is a Kafka SourceConnector. It is able to capture the database changes and stream them to Kafka. There are quite a few pitfalls when using debezium. Run unit test cd debezium-core/ mvn t...

Recently I have been writing the base library code to allow engineers to use Kafka in my company easily. I followed some examples online to write a consumer boilerplate code but quickly realized th...

Worker vs Connector vs Task When I read the Kafka Connect source code, the relationship between worker, connector and task confused me. Confluent has a nice illustration about their relationship. ...

list topics ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ./bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic filebeat ./bin/kafka-run-class.sh kafka.tools....