Most important Apache Kafka Commands

In this article, I am going to sharing the most important Apache Kafka commands to perform activities in Apache Kafka. In this explaining start the zookeeper server, start the Apache Kafka server, create a topic, create the partition, delete the topic, topics list, Kafka Console Consumer, Kafka Console Producer, Kafka Consumer Groups, Kafka Re-Assign Partitions, etc..

Zookeeper Installation home Directory: D:\ApacheKafka-ZooKeeper\zookeeper-3.4.14

Kafka Installation home Directory: D:\ApacheKafka-ZooKeeper\kafka_2.11-2.3.0

Windows Apache Kafka Commands Directory: D:\ApacheKafka-ZooKeeper\zookeeper-3.4.14\bin\windows

Linux/Unix Apache Kafka Commands Directory: /ApacheKafka-ZooKeeper/zookeeper-3.4.14/bin

The following commands executed under the above path mentioned.

How to start Zookeeper Server If Installed?

If you have installed Zookeeper separately then you have to go to the Zookeeper bin folder and there you can execute the following command.

Zookeeper default port: 2181

Windows:

zkServer.cmd

Linux/Unix:

kServer.sh

How to start Zookeeper Server from Apache Kafka?

If you have not installed Zookeeper separately then you have to go to the Kafka bin folder and there you can execute the following command.

Go to <Kafka-Home>/bin/windows/

Zookeeper default port: 2181

Windows:

zookeeper-server-start.bat ..\..\config\zookeeper.properties

Linux/Unix:

zookeeper-server-start.sh ../../config\zookeeper.properties

How to start Apache Kafka Server?

Apache Kafka default port:9092

Before going to start the Apache Kafka server, we should have to start the Zookeeper server first.

Windows:

kafka-server-start.bat ..\..\config\server.properties

Linux/Unix:

kafka-server-start.sh  ../config/server.properties

 

How to create the Topic with partitions and replication factor?

–partitions and –replication-factor parameters are mandatory to create the topic.

Windows:

Syntax:

kafka-topics.bat --zookeeper <zookeeper-ip>:<port> --create --topic <Topic Name> --partitions <No.Of Partitions> --replication-factor <No.Of Factors>

Example:

kafka-topics.bat --zookeeper localhost:2181 --create --topic MyFirstTopic1 --partitions 1 --replication-factor 1

Linux/Unix:

Syntax:

kafka-topics.sh--zookeeper <zookeeper-ip>:<port> --create --topic <Topic Name> --partitions <No.Of Partitions> --replication-factor <No.Of Factors>    

Example:

kafka-topics.sh --zookeeper localhost:2181 --create --topic MyFirstTopic1 --partitions 1 --replication-factor 1

How to create the Topic If Not Exist?

If topic not exist, then the topic will be created else the topic will not be created

Windows:

Syntax:

kafka-topics.bat --zookeeper <zookeeper-ip>:<port> --create --topic <Topic Name>  --partitions <No.Of Partitions> --replication-factor <No.Of Factors> --if-not-exists

Example:

kafka-topics.bat --zookeeper localhost:2181 --create --topic MyFirstTopic1 --partitions 1 --replication-factor 1 --if-not-exists

Linux/Unix:

Syntax:

kafka-topics.sh--zookeeper <zookeeper-ip>:<port> --create --topic <Topic Name>  --partitions <No.Of Partitions> --replication-factor <No.Of Factors>  --if-not-exists   

Example:

kafka-topics.sh --zookeeper localhost:2181 --create --topic MyFirstTopic1 --partitions 1 --replication-factor 1 --if-not-exists

How to Alter Topic to Increase Partitions?

Only topic partitions will be altered due to this, partition logic or ordering of message will be affected

Windows:

Syntax:

kafka-topics.bat --zookeeper <ZooKeeper-IP>:<Port> --alter --topic <Topic-Name> --partitions 5

Example:

kafka-topics.bat --zookeeper localhost:2181 --alter --topic MyFirstTopic1 --partitions 5

Linux/Unix:

Syntax:

kafka-topics.sh --zookeeper <ZooKeeper-IP>:<Port> --alter --topic <Topic-Name> --partitions 5   

Example:

kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyFirstTopic1 --partitions 5

 

How to Delete the Topic?

You can delete the topic if no longer needed.

Windows:

Syntax:

kafka-topics.bat — zookeeper <ZooKeeper-IP>:<Port>--delete --topic <Topic-Name>

Example:

kafka-topics.bat — zookeeper localhost:2181 --delete --topic MyFirstTopic1

Linux/Unix:

Syntax:

kafka-topics.sh — zookeeper <ZooKeeper-IP>:<Port>--delete --topic <Topic-Name>

Example:

kafka-topics.sh — zookeeper localhost:2181 --delete --topic MyFirstTopic1

 

How to Purge the Topic?

There is no command to purge the topic. There is a way to purge i.e change the data retention time to a very short period like 1 min. Before changing the retention time, you need to know the existing configured retention time because we have to reset the retention time.

Wait 1 min, older than1 minute topic data will be purged from the topic. And then you can reset the retention time as per the previous.

Windows:

Syntax:

kafka-configs.bat --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --add-config retention.ms=1000

Example:

kafka-configs.bat --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --add-config retention.ms=1000

Linux/Unix:

Syntax:

kafka-configs.sh --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --add-config retention.ms=1000     

Example:

kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --add-config retention.ms=1000

How to Change the Retention Period of Topic?

Here retention time in MilliSeconds.

2 days = 172800000

Windows:

Syntax:

kafka-configs.bat --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --add-config retention.ms=172800000

Example:

kafka-configs.bat --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --delete-config retention.ms

Linux/Unix:

Syntax:

kafka-configs.sh --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --add-config retention.ms=172800000

Example:

kafka-configs.sh  --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --delete-config retention.ms                

How to Delete Retention Time of Topic?

Once we delete the retention period, then the retention period will be back to default.

Default retention period of the topic is 7 days

Windows:

Syntax:

kafka-configs.bat --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --delete-config retention.ms

Example:

kafka-configs.bat --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --add-config retention.ms=172800000

Linux/Unix:

Syntax:

kkafka-configs.sh --zookeeper <ZooKeeper-IP>:<Port> --alter --entity-type topics --entity-name <Topic-Name> --add-config retention.ms=172800000

Example:

kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name MyFirstTopic1 --add-config retention.ms=172800000          

How to get List of Topics Whose Configuration Override?

You can get the list of topics whose configuration has been overridden.

Windows:

Syntax:

kafka-topics.bat --zookeeper <ZooKeeper-IP>:<Port> --describe --topics-with-overrides

Example:

kafka-topics.bat --zookeeper localhost:2181 --describe --topics-with-overrides

Linux/Unix:

Syntax:

kafka-topics.sh --zookeeper <ZooKeeper-IP>:<Port> --describe --topics-with-overrides  

Example:

kafka-topics.sh --zookeeper localhost:2181 --describe --topics-with-overrides

 

Apache Kafka Console Producer

Produce will send the message to the topic. By using the following command line, we can send the publish the message to the topic.

Windows:

Syntax:

kafka-console-producer.bat --broker-list <Kafka-IP>:<Port> --topic <Topic Name>

Example:

kafka-console-producer.bat --broker-list localhost:9092 --topic MyFirstTopic1

Linux/Unix:

Syntax:

kafka-console-producer.sh --broker-list <Kafka-IP>:<Port> --topic <Topic Name>

Example:

kafka-console-producer.sh  --broker-list localhost:9092 --topic MyFirstTopic1

Apache Kafka Console Producer Send Message From File

If we have a large message, then create a text file and add the message into a file, and then we can publish the text file.

Suppose we have a text file with the name of Example.txt which contains the following data

Hello World
NarayanaTutorial
Online Java Tutorial

Now send the message from the Example.txt.

Windows:

Syntax:

kafka-console-producer.bat --broker-list <Kafka-IP>:<Port> --topic <Topic Name> <Input-File-Path>

Example:

kafka-console-producer.bat --broker-list localhost:9092 --topic MyFirstTopic1  <  D:/Example.txt

Linux/Unix:

Syntax:

kafka-console-producer.sh --broker-list <Kafka-IP>:<Port> --topic <Topic Name>  <Input-File-Path>

Example:

kafka-console-producer.sh  --broker-list localhost:9092 --topic MyFirstTopic1 < /home/Example.txt
 

Apache Kafka Console Consumer

A consumer consumes the message from the topic. By using the following Kafka Console Consumer, we can consume the message on the console.

Windows:

Syntax:

kafka-console-consumer.bat --bootstrap-server <Kafka-IP>:<Port> --topic <Topic Name>

Example:

kafka-console-producer.bat --broker-list localhost:9092 --topic MyFirstTopic1

Linux/Unix:

Syntax:

kafka-console-consumer.sh  --bootstrap-server <Kafka-IP>:<Port> --topic <Topic Name>

Example:

kafka-console-producer.sh  --broker-list localhost:9092 --topic MyFirstTopic1

Apache Kafka Console Consumer From Beginning

On the consumer console, we can consume the messages from the beginning for the topic.

Windows:

Syntax:

kafka-console-consumer.bat --bootstrap-server <Kafka-IP>:<Port> --topic <Topic Name> --from-beginning

Example:

kafka-console-producer.bat --broker-list localhost:9092 --topic MyFirstTopic1 --from-beginning

Linux/Unix:

Syntax:

kafka-console-consumer.sh --bootstrap-server <Kafka-IP>:<Port> --topic <Topic Name> --from-beginning    

Example:

kafka-console-producer.sh  --broker-list localhost:9092 --topic MyFirstTopic1 --from-beginning

 

Apache Kafka Topics List

You can see the list of topics by executing the following command.

Windows:

Syntax:

kafka-topics.bat --zookeeper <Zookeeper-IP>:<Port> --list

OR

kafka-topics.bat  --bootstrap-server <Kafka-IP>:<Kafka-Port> --list

Example:

kafka-topics.bat --zookeeper localhost:2181 --list

OR

kafka-topics.bat   --bootstrap-server localhost:9092 --list

Linux/Unix:

Syntax:

kafka-topics.sh  --zookeeper <Zookeeper-IP>:<Port> --list

OR

kafka-topics.sh  --bootstrap-server <Kafka-IP>:<Kafka-Port> --list

Example:

kafka-topics.sh--zookeeper localhost:2181 --list

OR

kafka-topics.sh  --bootstrap-server localhost:9092  --list

Apache Kafka Topics List and Exclude Internal Topics

Exclude the listing of external topics like “__consumer_offsets”

Windows:

Syntax:

kafka-topics.bat --zookeeper <Zookeeper-IP>:<Port> --list --exclude-internal

OR

kafka-topics.bat  --bootstrap-server <Kafka-IP>:<Kafka-Port> --list --exclude-internal

Example:

kafka-topics.bat --zookeeper localhost:2181 --list  --exclude-internal

OR

kafka-topics.bat  --bootstrap-server localhost:9092  --list   --exclude-internal

Linux/Unix:

Syntax:

kafka-topics.sh--zookeeper <Zookeeper-IP>:<Port> --list --exclude-internal

OR

kafka-topics.sh  --list --bootstrap-server <Kafka-IP>:<Kafka-Port>  --list --exclude-internal    

Example:

kafka-topics.sh  --zookeeper localhost:2181  --list  --exclude-internal

OR

kafka-topics.sh  --bootstrap-server localhost:9092  --list --exclude-internal

Apache Kafka all Topics Describe/Details

We can get all topics configuration details

Windows:

Syntax:

kafka-topics.bat --zookeeper <Zookeeper-IP>:<Port>  --describe

OR

kafka-topics.bat  --bootstrap-server <Kafka-IP>:<Kafka-Port>  --describe

Example:

kafka-topics.bat --zookeeper localhost:2181  --describe

OR

kafka-topics.bat  --bootstrap-server localhost:9092  --describe

Linux/Unix

Syntax:

kafka-topics.sh --zookeeper <Zookeeper-IP>:<Port>  --describe

OR

kafka-topics.sh  --bootstrap-server <Kafka-IP>:<Kafka-Port>    --describe

Example:

kafka-topics.sh --zookeeper localhost:2181   --describe

OR

kafka-topics.sh  --bootstrap-server localhost:9092   --describe

Apache Kafka Topic Describe/Details

We can get individual topic configuration details

Windows:

Syntax:

kafka-topics.bat --zookeeper <Zookeeper-IP>:<Port> --topic <Topic-Name> --describe

OR

kafka-topics.bat  --bootstrap-server <Kafka-IP>:<Kafka-Port> --topic <Topic-Name> --describe   --describe

Example:

kafka-topics.bat --zookeeper localhost:2181  --topic MyFirstTopic1 --describe

OR

kafka-topics.bat  --bootstrap-server localhost:9092  --topic MyFirstTopic1 --describe

Linux/Unix:

Syntax:

kafka-topics.sh --zookeeper <Zookeeper-IP>:<Port> --topic <Topic-Name> --describe

OR

kafka-topics.sh  --bootstrap-server <Kafka-IP>:<Kafka-Port> --topic <Topic-Name>  --describe    

Example:

kafka-topics.sh --zookeeper localhost:2181  --topic MyFirstTopic1 --describe

OR

kafka-topics.sh  --bootstrap-server localhost:9092  --topic MyFirstTopic1 --describe

Apache Kafka Send Message With Key From Console Producer

Produce can publish the message to a topic by specifying the Key. The Key assigned to one partition. so the published message will be assigned to the specific partition which is assigned to the Key.

Windows:

Syntax

kafka-console-producer.bat --broker-list <Kafka-Ip>:<Kafka-Port>--topic <TopicName> --property "parse.key=true" --property "key.separator=:"

Example:

kafka-console-producer.bat --broker-list localhost:9092 --topic MyFirstTopic1 --property "parse.key=true" --property "key.separator=:"

Sample Data:

Key1:Value1

Key2:Value2

Linux/Unix:

Syntax:

kafka-console-producer.sh--broker-list <Kafka-Ip>:<Kafka-Port>--topic <TopicName> --property "parse.key=true" --property "key.separator=:"     

Example:

kafka-console-producer.sh--broker-list localhost:9092 --topic MyFirstTopic1 --property "parse.key=true" --property "key.separator=:"

Sample Data:

Key1:Value1

Key2:Value2

Apache Kafka Broker Version

You can see the Kafka broker version by using the following command.

Windows:

Syntax:

kafka-broker-api-versions.bat --bootstrap-server <Kafka-IP>:9092 --version

Example:

kafka-broker-api-versions.bat --bootstrap-server localhost:9092 --version

Linux/Unix:

Syntax:

kafka-broker-api-versions.sh--bootstrap-server <Kafka-IP>:9092 --version

Example:

kafka-broker-api-versions.sh--bootstrap-server localhost:9092 --version

 

References

Apache Kafka® is a distributed streaming platform

Apache Kafka CLI commands cheat sheet