Most important Apache Kafka Commands

Most important Apache Kafka Commands

Table of Contents

In this article, I am going to sharing the most important Apache Kafka commands to perform activities in Apache Kafka. In this explaining start the zookeeper server, start the Apache Kafka server, create a topic, create the partition, delete the topic, topics list, Kafka Console Consumer, Kafka Console Producer, Kafka Consumer Groups, Kafka Re-Assign Partitions, etc..

Zookeeper Installation home Directory: D:\ApacheKafka-ZooKeeper\zookeeper-3.4.14

Kafka Installation home Directory: D:\ApacheKafka-ZooKeeper\kafka_2.11-2.3.0

Windows Apache Kafka Commands Directory: D:\ApacheKafka-ZooKeeper\zookeeper-3.4.14\bin\windows

Linux/Unix Apache Kafka Commands Directory: /ApacheKafka-ZooKeeper/zookeeper-3.4.14/bin

The following commands executed under the above path mentioned.

How to start Zookeeper Server If Installed?

If you have installed Zookeeper separately then you have to go to the Zookeeper bin folder and there you can execute the following command.

Zookeeper default port: 2181

Windows

zkServer.cmd

Linux/Unix

kServer.sh

How to start Zookeeper Server from Apache Kafka?

If you have not installed Zookeeper separately then you have to go to the Kafka bin folder and there you can execute the following command.

Go to <Kafka-Home>/bin/windows/

Zookeeper default port: 2181

Windows

zookeeper-server-start.bat ..\..\config\zookeeper.properties

Linux/Unix

zookeeper-server-start.sh ../../config\zookeeper.properties

 

How to start Apache Kafka Server?

Apache Kafka default port:9092

Before going to start the Apache Kafka server, we should have to start the Zookeeper server first.

Windows

kafka-server-start.bat ..\..\config\server.properties

Linux/Unix

kafka-server-start.sh  ../config/server.properties

 

How to create the Topic with partitions and replication factor?

–partitions and –replication-factor parameters are mandotary to create the topic.

Windows

Syntax

kafka-topics.bat –zookeeper <zookeeper-ip>:<port> –create –topic <Topic Name> –partitions <No.Of Partitions> –replication-factor <No.Of Factors>

Example

kafka-topics.bat –zookeeper localhost:2181 –create –topic MyFirstTopic1 –partitions 1 –replication-factor 1

Linux/Unix

Syntax

kafka-topics.sh–zookeeper <zookeeper-ip>:<port> –create –topic <Topic Name> –partitions <No.Of Partitions> –replication-factor <No.Of Factors>

Example

kafka-topics.sh –zookeeper localhost:2181 –create –topic MyFirstTopic1 –partitions 1 –replication-factor 1

How to create the Topic If Not Exist?

If topic not exist, then the topic will be created else the topic will not be created

Windows

Syntax

kafka-topics.bat –zookeeper <zookeeper-ip>:<port> –create –topic <Topic Name>  –partitions <No.Of Partitions> –replication-factor <No.Of Factors> –if-not-exists

Example

kafka-topics.bat –zookeeper localhost:2181 –create –topic MyFirstTopic1 –partitions 1 –replication-factor 1 –if-not-exists

Linux/Unix

Syntax

kafka-topics.sh–zookeeper <zookeeper-ip>:<port> –create –topic <Topic Name>  –partitions <No.Of Partitions> –replication-factor <No.Of Factors>  –if-not-exists

Example

kafka-topics.sh –zookeeper localhost:2181 –create –topic MyFirstTopic1 –partitions 1 –replication-factor 1 –if-not-exists

How to Alter Topic to Increase Partitions?

Only topic partitions will be altered due to this, partition logic or ordering of message will be affected

Windows

Syntax

kafka-topics.bat –zookeeper <ZooKeeper-IP>:<Port> –alter –topic <Topic-Name> –partitions 5

Example

kafka-topics.bat –zookeeper localhost:2181 –alter –topic MyFirstTopic1 –partitions 5

Linux/Unix

Syntax

kafka-topics.sh –zookeeper <ZooKeeper-IP>:<Port> –alter –topic <Topic-Name> –partitions 5

Example

kafka-topics.sh –zookeeper localhost:2181 –alter –topic MyFirstTopic1 –partitions 5

 

How to Delete the Topic?

You can delete the topic if no longer needed.

Windows

Syntax

kafka-topics.bat — zookeeper <ZooKeeper-IP>:<Port>–delete –topic <Topic-Name>

Example

kafka-topics.bat — zookeeper localhost:2181 –delete –topic MyFirstTopic1

Linux/Unix

Syntax

kafka-topics.sh — zookeeper <ZooKeeper-IP>:<Port>–delete –topic <Topic-Name>

Example

kafka-topics.sh — zookeeper localhost:2181 –delete –topic MyFirstTopic1

 

How to Purge the Topic?

There is no command to purge the topic. There is a way to purge i.e change the data retention time to a very short period like 1 min. Before changing the retention time, you need to know the existing configured retention time because we have to reset the retention time.

Wait 1 min, older than1 minute topic data will be purged from the topic. And then you can reset the retention time as per the previous.

Windows

Syntax

kafka-configs.bat –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –add-config retention.ms=1000

Example

kafka-configs.bat –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –add-config retention.ms=1000

Linux/Unix

Syntax

kafka-configs.sh –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –add-config retention.ms=1000

Example

kafka-configs.sh –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –add-config retention.ms=1000

How to Change Retention Period of Topic?

Here retention time in MilliSeconds.

2 days = 172800000

Windows

Syntax

kafka-configs.bat –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –add-config retention.ms=172800000

Example

kafka-configs.bat –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –delete-config retention.ms

Linux/Unix

Syntax

kafka-configs.sh –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –add-config retention.ms=172800000

Example

kafka-configs.sh  –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –delete-config retention.ms

How to Delete Retention Time of Topic?

Once we delete the retention period, then the retention period will be back to default.

Default retention period of the topic is 7 days

Windows

Syntax

kafka-configs.bat –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –delete-config retention.ms

Example

kafka-configs.bat –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –add-config retention.ms=172800000

Linux/Unix

Syntax

kkafka-configs.sh –zookeeper <ZooKeeper-IP>:<Port> –alter –entity-type topics –entity-name <Topic-Name> –add-config retention.ms=172800000

Example

kafka-configs.sh –zookeeper localhost:2181 –alter –entity-type topics –entity-name MyFirstTopic1 –add-config retention.ms=172800000

How to get List of Topics Whose Configuration Override?

You can get the list of topics whose configuration has been overridden.

Windows

Syntax

kafka-topics.bat –zookeeper <ZooKeeper-IP>:<Port> –describe –topics-with-overrides

Example

kafka-topics.bat –zookeeper localhost:2181 –describe –topics-with-overrides

Linux/Unix

Syntax

kafka-topics.sh –zookeeper <ZooKeeper-IP>:<Port> –describe –topics-with-overrides

Example

kafka-topics.sh –zookeeper localhost:2181 –describe –topics-with-overrides

 

Apache Kafka Console Producer

Produce will send the message to the topic. By using the following command line, we can send the publish the message to the topic.

Windows

Syntax

kafka-console-producer.bat –broker-list <Kafka-IP>:<Port> –topic <Topic Name>

Example

kafka-console-producer.bat –broker-list localhost:9092 –topic MyFirstTopic1

Linux/Unix

Syntax

kafka-console-producer.sh –broker-list <Kafka-IP>:<Port> –topic <Topic Name>

Example

kafka-console-producer.sh  –broker-list localhost:9092 –topic MyFirstTopic1

Apache Kafka Console Producer Send Message From File

If we have a large message, then create a text file and add the message into a file, and then we can publish the text file.

Suppose we have a text file with the name of Example.txt which contains the following data

Hello World
NarayanaTutorial
Online Java Tutorial

Now send the message from the Example.txt.

Windows

Syntax

kafka-console-producer.bat –broker-list <Kafka-IP>:<Port> –topic <Topic Name> < <Input-File-Path>

Example

kafka-console-producer.bat –broker-list localhost:9092 –topic MyFirstTopic1  <  D:/Example.txt

Linux/Unix

Syntax

kafka-console-producer.sh –broker-list <Kafka-IP>:<Port> –topic <Topic Name>  < <Input-File-Path>

Example

kafka-console-producer.sh  –broker-list localhost:9092 –topic MyFirstTopic1 < /home/Example.txt

 

Apache Kafka Console Consumer

A consumer consumes the message from the topic. By using the following Kafka Console Consumer, we can consume the message on the console.

Windows

Syntax

kafka-console-consumer.bat –bootstrap-server <Kafka-IP>:<Port> –topic <Topic Name>

Example

kafka-console-producer.bat –broker-list localhost:9092 –topic MyFirstTopic1

Linux/Unix

Syntax

kafka-console-consumer.sh  –bootstrap-server <Kafka-IP>:<Port> –topic <Topic Name>

Example

kafka-console-producer.sh  –broker-list localhost:9092 –topic MyFirstTopic1

Apache Kafka Console Consumer From Beginning

On the consumer console, we can consume the messages from beginning for the topic.

Windows

Syntax

kafka-console-consumer.bat –bootstrap-server <Kafka-IP>:<Port> –topic <Topic Name> –from-beginning

Example

kafka-console-producer.bat –broker-list localhost:9092 –topic MyFirstTopic1 –from-beginning

Linux/Unix

Syntax

kafka-console-consumer.sh –bootstrap-server <Kafka-IP>:<Port> –topic <Topic Name> –from-beginning

Example

kafka-console-producer.sh  –broker-list localhost:9092 –topic MyFirstTopic1 –from-beginning

 

Apache Kafka Topics List

You can see the list of topics by executing the following command.

Windows

Syntax

kafka-topics.bat –zookeeper <Zookeeper-IP>:<Port> –list

OR

kafka-topics.bat  –bootstrap-server <Kafka-IP>:<Kafka-Port> –list

Example

kafka-topics.bat –zookeeper localhost:2181 –list

OR

kafka-topics.bat   –bootstrap-server localhost:9092 –list

Linux/Unix

Syntax

kafka-topics.sh  –zookeeper <Zookeeper-IP>:<Port> –list

OR

kafka-topics.sh  –bootstrap-server <Kafka-IP>:<Kafka-Port> –list

Example

kafka-topics.sh–zookeeper localhost:2181 –list

OR

kafka-topics.sh  –bootstrap-server localhost:9092  –list

Apache Kafka Topics List and Exclude Internal Topics

Exclude the listing of external topics like “__consumer_offsets

Windows

Syntax

kafka-topics.bat –zookeeper <Zookeeper-IP>:<Port> –list –exclude-internal

OR

kafka-topics.bat  –bootstrap-server <Kafka-IP>:<Kafka-Port> –list  –exclude-internal

Example

kafka-topics.bat –zookeeper localhost:2181 –list  –exclude-internal

OR

kafka-topics.bat  –bootstrap-server localhost:9092  –list   –exclude-internal

Linux/Unix

Syntax

kafka-topics.sh–zookeeper <Zookeeper-IP>:<Port> –list –exclude-internal

OR

kafka-topics.sh  –list –bootstrap-server <Kafka-IP>:<Kafka-Port>  –list   –exclude-internal

Example

kafka-topics.sh  –zookeeper localhost:2181  –list  –exclude-internal

OR

kafka-topics.sh  –bootstrap-server localhost:9092  –list   –exclude-internal

 

Apache Kafka all Topics Describe/Details

We can get all topics configuration details

Windows

Syntax

kafka-topics.bat –zookeeper <Zookeeper-IP>:<Port>  –describe

OR

kafka-topics.bat  –bootstrap-server <Kafka-IP>:<Kafka-Port>  –describe

Example

kafka-topics.bat –zookeeper localhost:2181  –describe

OR

kafka-topics.bat  –bootstrap-server localhost:9092  –describe

Linux/Unix

Syntax

kafka-topics.sh –zookeeper <Zookeeper-IP>:<Port>  –describe

OR

kafka-topics.sh  –bootstrap-server <Kafka-IP>:<Kafka-Port>    –describe

Example

kafka-topics.sh –zookeeper localhost:2181   –describe

OR

kafka-topics.sh  –bootstrap-server localhost:9092   –describe

 

Apache Kafka Topic Describe/Details

We can get individual topic configuration details

Windows

Syntax

kafka-topics.bat –zookeeper <Zookeeper-IP>:<Port> –topic <Topic-Name> –describe

OR

kafka-topics.bat  –bootstrap-server <Kafka-IP>:<Kafka-Port> –topic <Topic-Name>   –describe

Example

kafka-topics.bat –zookeeper localhost:2181  –topic MyFirstTopic1 –describe

OR

kafka-topics.bat  –bootstrap-server localhost:9092  –topic MyFirstTopic1 –describe

Linux/Unix

Syntax

kafka-topics.sh –zookeeper <Zookeeper-IP>:<Port> –topic <Topic-Name> –describe

OR

kafka-topics.sh  –bootstrap-server <Kafka-IP>:<Kafka-Port> –topic <Topic-Name>   –describe

Example

kafka-topics.sh –zookeeper localhost:2181  –topic MyFirstTopic1 –describe

OR

kafka-topics.sh  –bootstrap-server localhost:9092  –topic MyFirstTopic1 –describe

Apache Kafka Send Message With Key From Console Producer

Produce can publish the message to a topic by specifying the Key. The Key assigned to one partition. so the published message will be assigned to the specific partition which is assigned to the Key.

Windows

Syntax

kafka-console-producer.bat –broker-list <Kafka-Ip>:<Kafka-Port>–topic <TopicName> –property “parse.key=true” –property “key.separator=:”

Example

kafka-console-producer.bat –broker-list localhost:9092 –topic MyFirstTopic1 –property “parse.key=true” –property “key.separator=:”

Sample Data:

Key1:Value1

Key2:Value2

Linux/Unix

Syntax

kafka-console-producer.sh–broker-list <Kafka-Ip>:<Kafka-Port>–topic <TopicName> –property “parse.key=true” –property “key.separator=:”

Example

kafka-console-producer.sh–broker-list localhost:9092 –topic MyFirstTopic1 –property “parse.key=true” –property “key.separator=:”

Sample Data:

Key1:Value1

Key2:Value2

Apache Kafka Broker Version

You can see Kafka broker version by using the following command.

Windows

Syntax

kafka-broker-api-versions.bat –bootstrap-server <Kafka-IP>:9092 –version

Example

kafka-broker-api-versions.bat –bootstrap-server localhost:9092 –version

Linux/Unix

Syntax

kafka-broker-api-versions.sh–bootstrap-server <Kafka-IP>:9092 –version

Example

kafka-broker-api-versions.sh–bootstrap-server localhost:9092 –version

 

References

Apache Kafka® is a distributed streaming platform

Apache Kafka CLI commands cheat sheet

 

 

Leave a Reply