Source code Kafka provides a script kafka-topics.sh to manage topics. It simple class TopicCommand.scala. As you can see it supports create, alert, list, describe and delete commands. Change numb...

Kafka python client There are two popular python client library: kafka-python and confluent-python-kafka. The former is a python translation of the official Java client. dpkp/kafka-python # How ...

kafka-consuemr-group.sh is a script released together with Kafka that helps users to inspect or change consumers. The operations it supports are ListGroups, DescribeGroups, DeleteGroups, ResetOffs...

There are tons of parameters to configure Kafka. The configuration can be classified into 3 buckets: broker configurations, producer configurations and consumer configurations. Broker configuratio...

Deep diving into TLS is an item in my TODO list for a long time. But as you know, procrastination is not easy to overcome. The opportunity finally comes: I need to set up mTLS for a webhook endpoin...

Why linkers? I became interested in linker by reading Ian Lance Taylor’s blog series. He was writing a new linker gold for binutils-gdb. There are lots of things I do not understand from his blogs...

We know that Kafka metadata such as locations of partitions is stored in Zookeeper. Out of curiosity, I decide to take a look inside Zookeeper. However, this seems not as simple as I thought. Bef...

Authentication and authorization are two different concepts. Authentication answers the question “who can log in to the system?” Authorization answers what you can do after login. Authentication is...

Debezium is a Kafka SourceConnector. It is able to capture the database changes and stream them to Kafka. There are quite a few pitfalls when using debezium. Run unit test cd debezium-core/ mvn t...

Recently I have been writing the base library code to allow engineers to use Kafka in my company easily. I followed some examples online to write a consumer boilerplate code but quickly realized th...