Open-IM-Server/docs/contrib/kafka.md

OpenIM Kafka Guide

This document aims to provide a set of concise guidelines to help you quickly install and use Kafka through Docker Compose.

Installing Kafka

With the Docker Compose script provided by OpenIM, you can easily install Kafka. Use the following command to start Kafka:

docker compose up -d

After executing this command, Kafka will be installed and started. You can confirm the Kafka container is running with the following command:

docker ps | grep kafka

The output of this command, as shown below, displays the status information of the Kafka container:

be416b5a0851   bitnami/kafka:3.5.1 "/opt/bitnami/script…"   3 days ago   Up 2 days   9092/tcp, 0.0.0.0:19094->9094/tcp, :::19094->9094/tcp   kafka

References

• Official Docker installation documentation: Click here
• Detailed installation guide: Tutorial on Towards Data Science

Using Kafka

Entering the Kafka Container

To execute Kafka commands, you first need to enter the Kafka container. Use the following command:

docker exec -it kafka bash

Kafka Command Tools

Inside the Kafka container, you can use various command-line tools to manage Kafka. These tools include but are not limited to:

• kafka-topics.sh: For creating, deleting, listing, or altering topics.
• kafka-console-producer.sh: Allows sending messages to a specified topic from the command line.
• kafka-console-consumer.sh: Allows reading messages from the command line, with the ability to specify topics.
• kafka-consumer-groups.sh: For managing consumer group information.

Kafka Client Tool Installation

For easier Kafka management, you can install Kafka client tools. If you installed Kafka through OpenIM's Docker Compose, you can install the Kafka client tools with the following command:

make install.kafkactl

Automatic Topic Creation

When installing Kafka through OpenIM's Docker Compose method, OpenIM automatically creates the following topics:

• latestMsgToRedis
• msgToPush
• offlineMsgToMongoMysql

These topics are created using the scripts/create-topic.sh script. The script waits for Kafka to be ready before executing the commands to create topics:

# Wait for Kafka to be ready
until /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092; do
  echo "Waiting for Kafka to be ready..."
  sleep 2
done

# Create topics
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic latestMsgToRedis
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic msgToPush
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic offlineMsgToMongoMysql

echo "Topics created."

The optimized and expanded documentation further details some basic commands for operations inside the Kafka container, as well as basic commands for managing Kafka using kafkactl. Here is a more detailed guide.

Basic Commands in the Kafka Container

Listing Topics

To list all existing topics, you can use the following command:

kafka-topics.sh --list --bootstrap-server localhost:9092

Creating a New Topic

When creating a new topic, you can specify the number of partitions and the replication factor. Here is the command to create a new topic:

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic your_topic_name

Producing Messages

To send messages to a specific topic, you can use the producer command. The following command prompts you to enter messages, which are sent to the specified topic with each press of the Enter key:

kafka-console-producer.sh --broker-list localhost:9092 --topic your_topic_name

Consuming Messages

To read messages from a specific topic, you can use the consumer command. The following command reads new messages from the specified topic and outputs them on the console:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning

The `

--from-beginning` parameter reads messages from the beginning of the topic. If this parameter is omitted, only new messages will be read.

Basic Commands Using kafkactl

kafkactl is a command-line tool for managing and operating Kafka clusters. It offers a more modern way to interact with Kafka.

Listing Topics

To list all topics, you can use:

kafkactl get topics

Creating a New Topic

To create a new topic with kafkactl, use:

kafkactl create topic your_topic_name --partitions 1 --replication-factor 1

Producing Messages

To send messages to a topic, you can use:

kafkactl produce your_topic_name --value "your message"

Here, "your message" is the content of the message you want to send.

Consuming Messages

To consume messages from a topic, use:

kafkactl consume your_topic_name --from-beginning

Again, the --from-beginning parameter will start consuming messages from the beginning of the topic. If you do not wish to start from the beginning, you can omit this parameter.