diff --git a/docs/contrib/kafka.md b/docs/contrib/kafka.md new file mode 100644 index 000000000..4547c9480 --- /dev/null +++ b/docs/contrib/kafka.md @@ -0,0 +1,162 @@ +# OpenIM Kafka Guide + +This document aims to provide a set of concise guidelines to help you quickly install and use Kafka through Docker Compose. + +## Installing Kafka + +With the Docker Compose script provided by OpenIM, you can easily install Kafka. Use the following command to start Kafka: + +```bash +docker compose up -d +``` + +After executing this command, Kafka will be installed and started. You can confirm the Kafka container is running with the following command: + +```bash +docker ps | grep kafka +``` + +The output of this command, as shown below, displays the status information of the Kafka container: + +``` +be416b5a0851 bitnami/kafka:3.5.1 "/opt/bitnami/script…" 3 days ago Up 2 days 9092/tcp, 0.0.0.0:19094->9094/tcp, :::19094->9094/tcp kafka +``` + +### References + +- Official Docker installation documentation: [Click here](http://events.jianshu.io/p/b60afa35303a) +- Detailed installation guide: [Tutorial on Towards Data Science](https://towardsdatascience.com/how-to-install-apache-kafka-using-docker-the-easy-way-4ceb00817d8b) + +## Using Kafka + +### Entering the Kafka Container + +To execute Kafka commands, you first need to enter the Kafka container. Use the following command: + +```bash +docker exec -it kafka bash +``` + +### Kafka Command Tools + +Inside the Kafka container, you can use various command-line tools to manage Kafka. These tools include but are not limited to: + +- `kafka-topics.sh`: For creating, deleting, listing, or altering topics. +- `kafka-console-producer.sh`: Allows sending messages to a specified topic from the command line. +- `kafka-console-consumer.sh`: Allows reading messages from the command line, with the ability to specify topics. +- `kafka-consumer-groups.sh`: For managing consumer group information. + +### Kafka Client Tool Installation + +For easier Kafka management, you can install Kafka client tools. If you installed Kafka through OpenIM's Docker Compose, you can install the Kafka client tools with the following command: + +```bash +make install.kafkactl +``` + +### Automatic Topic Creation + +When installing Kafka through OpenIM's Docker Compose method, OpenIM automatically creates the following topics: + +- `latestMsgToRedis` +- `msgToPush` +- `offlineMsgToMongoMysql` + +These topics are created using the `scripts/create-topic.sh` script. The script waits for Kafka to be ready before executing the commands to create topics: + +```bash +# Wait for Kafka to be ready +until /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092; do + echo "Waiting for Kafka to be ready..." + sleep 2 +done + +# Create topics +/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic latestMsgToRedis +/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic msgToPush +/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic offlineMsgToMongoMysql + +echo "Topics created." +``` + +The optimized and expanded documentation further details some basic commands for operations inside the Kafka container, as well as basic commands for managing Kafka using `kafkactl`. Here is a more detailed guide. + + +## Basic Commands in the Kafka Container + +### Listing Topics + +To list all existing topics, you can use the following command: + +```bash +kafka-topics.sh --list --bootstrap-server localhost:9092 +``` + +### Creating a New Topic + +When creating a new topic, you can specify the number of partitions and the replication factor. Here is the command to create a new topic: + +```bash +kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic your_topic_name +``` + +### Producing Messages + +To send messages to a specific topic, you can use the producer command. The following command prompts you to enter messages, which are sent to the specified topic with each press of the Enter key: + +```bash +kafka-console-producer.sh --broker-list localhost:9092 --topic your_topic_name +``` + +### Consuming Messages + +To read messages from a specific topic, you can use the consumer command. The following command reads new messages from the specified topic and outputs them on the console: + +```bash +kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning +``` + +The ` + +--from-beginning` parameter reads messages from the beginning of the topic. If this parameter is omitted, only new messages will be read. + + +## Basic Commands Using `kafkactl` + +`kafkactl` is a command-line tool for managing and operating Kafka clusters. It offers a more modern way to interact with Kafka. + +### Listing Topics + +To list all topics, you can use: + +```bash +kafkactl get topics +``` + +### Creating a New Topic + +To create a new topic with `kafkactl`, use: + +```bash +kafkactl create topic your_topic_name --partitions 1 --replication-factor 1 +``` + +### Producing Messages + +To send messages to a topic, you can use: + +```bash +kafkactl produce your_topic_name --value "your message" +``` + +Here, `"your message"` is the content of the message you want to send. + +### Consuming Messages + +To consume messages from a topic, use: + +```bash +kafkactl consume your_topic_name --from-beginning +``` + +Again, the `--from-beginning` parameter will start consuming messages from the beginning of the topic. If you do not wish to start from the beginning, you can omit this parameter. \ No newline at end of file diff --git a/scripts/make-rules/common-versions.mk b/scripts/make-rules/common-versions.mk index 04d062ce8..572585fce 100644 --- a/scripts/make-rules/common-versions.mk +++ b/scripts/make-rules/common-versions.mk @@ -50,6 +50,7 @@ STANDARDIZER_VERSION ?= latest GO_TESTS_VERSION ?= v1.6.0 GO_APIDIFF_VERSION ?= v0.8.2 KAFKACTL_VERSION ?= latest +GOTESTSUM_VERSION ?= latest WIRE_VERSION ?= latest # WIRE_VERSION ?= $(call get_go_version,github.com/google/wire) diff --git a/scripts/make-rules/tools.mk b/scripts/make-rules/tools.mk index a72649885..5335d094d 100644 --- a/scripts/make-rules/tools.mk +++ b/scripts/make-rules/tools.mk @@ -125,6 +125,11 @@ install.swagger: # Tools that might be used include go gvm # +## install.gotestsum: Install gotestsum, used to run go tests +.PHONY: install.gotestsum +install.gotestsum: + @$(GO) install gotest.tools/gotestsum@$(GOTESTSUM_VERSION) + ## install.kube-score: Install kube-score, used to check kubernetes yaml files .PHONY: install.kube-score install.kube-score: