We know that Kafka metadata such as locations of partitions is stored in Zookeeper. Out of curiosity, I decide to take a look inside Zookeeper. However, this seems not as simple as I thought. Bef...

Authentication and authorization are two different concepts. Authentication answers the question “who can log in to the system?” Authorization answers what you can do after login. Authentication is...

Debezium is a Kafka SourceConnector. It is able to capture the database changes and stream them to Kafka. There are quite a few pitfalls when using debezium. Run unit test cd debezium-core/ mvn t...

Recently I have been writing the base library code to allow engineers to use Kafka in my company easily. I followed some examples online to write a consumer boilerplate code but quickly realized th...

Worker vs Connector vs Task When I read the Kafka Connect source code, the relationship between worker, connector and task confused me. Confluent has a nice illustration about their relationship. ...

list topics ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ./bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic filebeat ./bin/kafka-run-class.sh kafka.tools....

First question: what is the unit of timestamp inside Kafka repo. The Long timestamp variable shows everywhere inside Kafka repo. What is its unit? Take a step back. The timestamp must come from the...

Recently, I was revisiting the C++ std::time function and suddenly get triggered by the std::time_t type. Quoted from cppreference, Although not defined, this is almost always an integral value...

Configurations tidb_mem_quota_query: each query has a memory limit, default is 1GB. Lighting Mydumps file pattern code.

Kafka protocol Kafka has its own binary protocol built on top of TCP to allow clients to talk to it. See the official documentation. There are less than twenty request types (ApiKey) in total, so ...