From 607d97fb9d78d83aa82233f595a2aeaf4cc75de9 Mon Sep 17 00:00:00 2001 From: "Xinwei Xiong(cubxxw-openim)" <3293172751nss@gmail.com> Date: Wed, 23 Aug 2023 20:52:33 +0800 Subject: [PATCH] feat: add more test code Signed-off-by: Xinwei Xiong(cubxxw-openim) <3293172751nss@gmail.com> --- .github/workflows/greetings.yml | 15 ++-- .github/workflows/project-progress.yml | 2 +- docs/contrib/offline-deployment.md | 116 +++++++++++++++++++++++++ scripts/install/environment.sh | 40 ++++++--- scripts/install/openim-msggateway.sh | 30 ++++--- scripts/install/openim-rpc.sh | 1 - scripts/lib/util.sh | 4 +- tools/imctl/README.md | 90 ++++++++++++++----- 8 files changed, 236 insertions(+), 62 deletions(-) create mode 100644 docs/contrib/offline-deployment.md diff --git a/.github/workflows/greetings.yml b/.github/workflows/greetings.yml index 667b6af08..2803f0d0e 100644 --- a/.github/workflows/greetings.yml +++ b/.github/workflows/greetings.yml @@ -28,13 +28,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/first-interaction@v1.1.1 with: - repo-token: ${{ secrets.REDBOT_GITHUB_TOKEN }} - issue-message: | - Hello! Thank you for filing an issue. - - If this is a bug report, please include relevant logs to help us debug the problem. - - [Join slack 🤖](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg) to connect and communicate with our developers. + repo-token: ${{ secrets.BOT_GITHUB_TOKEN }} pr-message: | Hello! Thank you for your contribution. @@ -42,4 +36,11 @@ jobs: If you are implementing a feature request, please check with the maintainers that the feature will be accepted first. + [Join slack 🤖](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg) to connect and communicate with our developers. + + issue-message: | + Hello! Thank you for filing an issue. + + If this is a bug report, please include relevant logs to help us debug the problem. + [Join slack 🤖](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg) to connect and communicate with our developers. \ No newline at end of file diff --git a/.github/workflows/project-progress.yml b/.github/workflows/project-progress.yml index d0cc10604..3ec8b7b1f 100644 --- a/.github/workflows/project-progress.yml +++ b/.github/workflows/project-progress.yml @@ -31,6 +31,6 @@ jobs: steps: - uses: alex-page/github-project-automation-plus@v0.8.3 with: - project: OpenIM-V3.1 + project: openim-powerful column: In Progress repo-token: ${{ secrets.BOT_GITHUB_TOKEN }} \ No newline at end of file diff --git a/docs/contrib/offline-deployment.md b/docs/contrib/offline-deployment.md new file mode 100644 index 000000000..6272c7499 --- /dev/null +++ b/docs/contrib/offline-deployment.md @@ -0,0 +1,116 @@ + + +# OpenIM Offline Deployment Design + +## 1. Base Images + +Below are the base images and their versions you'll need: + +- wurstmeister/kafka +- redis:7.0.0 +- mongo:6.0.2 +- mysql:5.7 +- wurstmeister/zookeeper +- minio/minio + +Use the following commands to pull these base images: + +``` +docker pull wurstmeister/kafka +docker pull redis:7.0.0 +docker pull mongo:6.0.2 +docker pull mysql:5.7 +docker pull wurstmeister/zookeeper +docker pull minio/minio +``` + +## 2. OpenIM & Chat Images + +**For detailed understanding of version management and storage of OpenIM and Chat**: [version.md](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md) + +### OpenIM Image + +- Get image version info: [images.md](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/images.md) +- Depending on the required version, execute the following command: + +```bash +docker pull ghcr.io/openimsdk/openim-server: +``` + +### Chat Image + +- Execute the following command to pull the image: + +```bash +docker pull ghcr.io/openimsdk/openim-server: +``` + +## 3. Image Storage Selection + +**Repositories**: + +- Alibaba Cloud: `registry.cn-hangzhou.aliyuncs.com/openimsdk/openim-server` +- Docker Hub: `openim/openim-server` + +**Version Selection**: + +- Stable: e.g. release-v3.2 (or 3.1, 3.3) +- Latest: latest +- Latest of main: main + +## 4. Version Selection + +You can select from the following versions: + +- Stable: e.g. release-v3.2 +- Latest: latest +- Latest from main branch: main + +## 5. Offline Deployment Steps + +1. **Pull images**: Execute the above `docker pull` commands to pull all required images locally. +2. **Save images**: + +``` +docker save -o .tar +``` + +1. **Fetch code**: Clone the repository: + +``` +git clone https://github.com/OpenIMSDK/openim-docker.git +``` + +Or download the code from [Releases](https://github.com/OpenIMSDK/openim-docker/releases/). + +1. **Transfer files**: Use `scp` to transfer all images and code to the intranet server. + +``` +scp .tar user@remote-ip:/path/on/remote/server +``` + +Or choose other transfer methods such as a hard drive. + +1. **Import images**: On the intranet server: + +``` +docker load -i .tar +``` + +1. **Deploy**: Navigate to the `openim-docker` repository directory and follow the README guide for deployment. +2. **Deploy using Docker-compose**: + +``` +docker-compose up -d + +# Verify +docker-compose ps +``` + +> **Note**: If you're using a version of Docker prior to 20, make sure you've installed `docker-compose`. + +## 6. Reference Links + +- [OpenIMSDK Issue #432](https://github.com/OpenIMSDK/Open-IM-Server/issues/432) +- [Notion Link](https://nsddd.notion.site/435ee747c0bc44048da9300a2d745ad3?pvs=25) +- [OpenIMSDK Issue #474](https://github.com/OpenIMSDK/Open-IM-Server/issues/474) \ No newline at end of file diff --git a/scripts/install/environment.sh b/scripts/install/environment.sh index 1e09b7e72..b553a00b0 100755 --- a/scripts/install/environment.sh +++ b/scripts/install/environment.sh @@ -102,7 +102,7 @@ def "MONGO_MAX_POOL_SIZE" "100" # 最 ###################### Object 配置信息 ###################### def "OBJECT_ENABLE" "minio" # 对象是否启用 -def "OBJECT_APIURL" "http://${IP}:10002" # 对象的API地址 +def "OBJECT_APIURL" "http://${IP}:10002" # 对象的API地址 def "MINIO_BUCKET" "openim" # MinIO的存储桶名称 def "MINIO_PORT" "10005" # MinIO的端口 # MinIO的端点URL @@ -147,7 +147,7 @@ def "RPC_LISTEN_IP" "0.0.0.0" # RPC的监听IP ###################### API 配置信息 ###################### # API的开放端口, 只能设置一个端口 -def "API_OPENIM_PORT" "10002" +readonly API_OPENIM_PORT=${API_OPENIM_PORT:-'10002'} def "API_LISTEN_IP" "0.0.0.0" # API的监听IP ###################### RPC Port Configuration Variables ###################### @@ -162,7 +162,7 @@ readonly OPENIM_FRIEND_PORT=${OPENIM_FRIEND_PORT:-'10120'} # OpenIM消息服务端口 readonly OPENIM_MESSAGE_PORT=${OPENIM_MESSAGE_PORT:-'10130'} # OpenIM消息网关服务端口 -readonly OPENIM_MESSAGE_GATEWAY_PORT=${OPENIM_MESSAGE_GATEWAY_PORT:-'10140'} +readonly OPENIM_MESSAGE_GATEWAY_PORT=${OPENIM_MESSAGE_GATEWAY_PORT:-'10140,11140'} # OpenIM组服务端口 readonly OPENIM_GROUP_PORT=${OPENIM_GROUP_PORT:-'10150'} # OpenIM授权服务端口 @@ -195,7 +195,8 @@ def "LOG_IS_JSON" "false" # 日志是否为JSON格式 def "LOG_WITH_STACK" "false" # 日志是否带有堆栈信息 ###################### Variables definition ###################### -def "OPENIM_WS_PORT" "10001" # OpenIM WS端口 +# OpenIM WS端口 +readonly OPENIM_WS_PORT=${OPENIM_WS_PORT:-'10001,11001'} def "WEBSOCKET_MAX_CONN_NUM" "100000" # Websocket最大连接数 def "WEBSOCKET_MAX_MSG_LEN" "4096" # Websocket最大消息长度 def "WEBSOCKET_TIMEOUT" "10" # Websocket超时 @@ -232,16 +233,27 @@ def "IOS_PRODUCTION" "false" # IOS生产 ###################### Prometheus 配置信息 ###################### def "PROMETHEUS_ENABLE" "false" # 是否启用 Prometheus -def "USER_PROM_PORT" "20110" # User 服务的 Prometheus 端口 -def "FRIEND_PROM_PORT" "20120" # Friend 服务的 Prometheus 端口 -def "MESSAGE_PROM_PORT" "20130" # Message 服务的 Prometheus 端口 -def "MSG_GATEWAY_PROM_PORT" "20140" # Message Gateway 服务的 Prometheus 端口 -def "GROUP_PROM_PORT" "20150" # Group 服务的 Prometheus 端口 -def "AUTH_PROM_PORT" "20160" # Auth 服务的 Prometheus 端口 -def "PUSH_PROM_PORT" "20170" # Push 服务的 Prometheus 端口 -def "CONVERSATION_PROM_PORT" "20230" # Conversation 服务的 Prometheus 端口 -def "RTC_PROM_PORT" "21300" # RTC 服务的 Prometheus 端口 -def "THIRD_PROM_PORT" "21301" # Third 服务的 Prometheus 端口 +# User 服务的 Prometheus 端口 +readonly USER_PROM_PORT=${USER_PROM_PORT:-'20110'} +# Friend 服务的 Prometheus 端口 +readonly FRIEND_PROM_PORT=${FRIEND_PROM_PORT:-'20120'} +# Message 服务的 Prometheus 端口 +readonly MESSAGE_PROM_PORT=${MESSAGE_PROM_PORT:-'20130'} +# Message Gateway 服务的 Prometheus 端口 +readonly MSG_GATEWAY_PROM_PORT=${MSG_GATEWAY_PROM_PORT:-'20140,21140'} +# Group 服务的 Prometheus 端口 +readonly GROUP_PROM_PORT=${GROUP_PROM_PORT:-'20150'} +# Auth 服务的 Prometheus 端口 +readonly AUTH_PROM_PORT=${AUTH_PROM_PORT:-'20160'} +# Push 服务的 Prometheus 端口 +readonly PUSH_PROM_PORT=${PUSH_PROM_PORT:-'20170'} +# Conversation 服务的 Prometheus 端口 +readonly CONVERSATION_PROM_PORT=${CONVERSATION_PROM_PORT:-'20230'} +# RTC 服务的 Prometheus 端口 +readonly RTC_PROM_PORT=${RTC_PROM_PORT:-'21300'} +# Third 服务的 Prometheus 端口 +readonly THIRD_PROM_PORT=${THIRD_PROM_PORT:-'21301'} + # Message Transfer 服务的 Prometheus 端口列表 readonly MSG_TRANSFER_PROM_PORT=${MSG_TRANSFER_PROM_PORT:-'21400, 21401, 21402, 21403'} diff --git a/scripts/install/openim-msggateway.sh b/scripts/install/openim-msggateway.sh index de5656b0c..d31dc9612 100755 --- a/scripts/install/openim-msggateway.sh +++ b/scripts/install/openim-msggateway.sh @@ -33,32 +33,36 @@ function openim::msggateway::start() openim::util::stop_services_with_name ${SERVER_NAME} # OpenIM message gateway service port - OPENIM_RPC_PORTS=$(openim::util::list-to-string ${OPENIM_MESSAGE_GATEWAY_PORT} ) + OPENIM_MESSAGE_GATEWAY_PORTS=$(openim::util::list-to-string ${OPENIM_MESSAGE_GATEWAY_PORT} ) + read -a OPENIM_MSGGATEWAY_PORTS_ARRAY <<< $(openim::util::list-to-string ${OPENIM_MESSAGE_GATEWAY_PORT} ) + # OpenIM WS port OPENIM_WS_PORTS=$(openim::util::list-to-string ${OPENIM_WS_PORT} ) + read -a OPENIM_WS_PORTS_ARRAY <<< $(openim::util::list-to-string ${OPENIM_WS_PORT} ) + # Message Gateway Prometheus port of the service MSG_GATEWAY_PROM_PORTS=$(openim::util::list-to-string ${MSG_GATEWAY_PROM_PORT} ) + read -a MSG_GATEWAY_PROM_PORTS_ARRAY <<< $(openim::util::list-to-string ${MSG_GATEWAY_PROM_PORT} ) - openim::log::status "OpenIM RPC ports: ${OPENIM_RPC_PORTS[*]}" + openim::log::status "OpenIM Mssage Getway ports: ${OPENIM_MESSAGE_GATEWAY_PORTS[*]}" openim::log::status "OpenIM WS ports: ${OPENIM_WS_PORTS[*]}" openim::log::status "OpenIM Prometheus ports: ${MSG_GATEWAY_PROM_PORTS[*]}" openim::log::status "OpenIM Msggateway config path: ${OPENIM_MSGGATEWAY_CONFIG}" - if [ ${#OPENIM_RPC_PORTS[@]} -ne ${#OPENIM_WS_PORTS[@]} ]; then - openim::log::error "ws_ports does not match push_rpc_ports or prome_ports in quantity!!!" - exit 1 + if [ ${#OPENIM_MSGGATEWAY_PORTS_ARRAY[@]} -ne ${#OPENIM_WS_PORTS_ARRAY[@]} ]; then + openim::log::error_exit "ws_ports does not match push_rpc_ports or prome_ports in quantity!!!" fi - for ((i = 0; i < ${#OPENIM_WS_PORTS[@]}; i++)); do - openim::log::info "start push process, port: ${OPENIM_MSGGATEWAY_PORTS_ARRAY[$i]}, prometheus port: ${PUSH_MSGGATEWAY_PORTS_ARRAY[$i]}" - - PROMETHEUS_PORT_OPTION="" - if [[ -n "${MSG_GATEWAY_PROM_PORTS[$i]}" ]]; then - PROMETHEUS_PORT_OPTION="--prometheus_port ${MSG_GATEWAY_PROM_PORTS[$i]}" - fi + for ((i = 0; i < ${#OPENIM_WS_PORTS_ARRAY[@]}; i++)); do + openim::log::info "start push process, port: ${OPENIM_MSGGATEWAY_PORTS_ARRAY[$i]}, prometheus port: ${MSG_GATEWAY_PROM_PORTS_ARRAY[$i]}" + + PROMETHEUS_PORT_OPTION="" + if [[ -n "${MSG_GATEWAY_PROM_PORTS_ARRAY[$i]}" ]]; then + PROMETHEUS_PORT_OPTION="--prometheus_port ${MSG_GATEWAY_PROM_PORTS_ARRAY[$i]}" + fi - nohup ${OPENIM_MSGGATEWAY_BINARY} --port ${OPENIM_RPC_PORTS[$i]} --ws_port ${OPENIM_WS_PORTS[$i]} $PROMETHEUS_PORT_OPTION -c ${OPENIM_MSGGATEWAY_CONFIG} >> ${LOG_FILE} 2>&1 & + nohup ${OPENIM_MSGGATEWAY_BINARY} --port ${OPENIM_MSGGATEWAY_PORTS_ARRAY[$i]} --ws_port ${OPENIM_WS_PORTS_ARRAY[$i]} $PROMETHEUS_PORT_OPTION -c ${OPENIM_MSGGATEWAY_CONFIG} >> ${LOG_FILE} 2>&1 & done openim::util::check_process_names ${SERVER_NAME} diff --git a/scripts/install/openim-rpc.sh b/scripts/install/openim-rpc.sh index 35745bff3..5e77e149e 100755 --- a/scripts/install/openim-rpc.sh +++ b/scripts/install/openim-rpc.sh @@ -128,7 +128,6 @@ function openim::rpc::start() { # Get the service and Prometheus ports. OPENIM_RPC_SERVICE_PORTS=( $(openim::util::list-to-string ${OPENIM_RPC_PORT_LISTARIES[$i]}) ) OPENIM_RPC_PROM_PORTS=( $(openim::util::list-to-string ${OPENIM_RPC_PROM_PORT_LISTARIES[$i]}) ) - for ((j = 0; j < ${#OPENIM_RPC_SERVICE_PORTS[@]}; j++)); do openim::log::info "Starting ${OPENIM_RPC_SERVICE_LISTARIES[$i]} service, port: ${OPENIM_RPC_SERVICE_PORTS[j]}, prometheus port: ${OPENIM_RPC_PROM_PORTS[j]}, binary root: ${OPENIM_OUTPUT_HOSTBIN}/${OPENIM_RPC_SERVICE_LISTARIES[$i]}" openim::rpc::start_service "${OPENIM_RPC_SERVICE_LISTARIES[$i]}" "${OPENIM_RPC_SERVICE_PORTS[j]}" "${OPENIM_RPC_PROM_PORTS[j]}" diff --git a/scripts/lib/util.sh b/scripts/lib/util.sh index 56f2a16c6..4359f6066 100755 --- a/scripts/lib/util.sh +++ b/scripts/lib/util.sh @@ -895,13 +895,13 @@ function openim::util::list-to-string() { # 2. Replace commas with spaces # 3. Remove opening and closing brackets ports_array=$(echo "$ports_list" | sed 's/ //g; s/,/ /g; s/^\[\(.*\)\]$/\1/') - # For external use, we might want to echo the result so that it can be captured by callers echo "$ports_array" } # MSG_GATEWAY_PROM_PORTS=$(openim::util::list-to-string "10023, 2323, 34 34") +# read -a MSG_GATEWAY_PROM_PORTS <<< $(openim::util::list-to-string "10023, 2323, 34 34") # echo ${MSG_GATEWAY_PROM_PORTS} - +# echo "${#MSG_GATEWAY_PROM_PORTS[@]}" # Downloads cfssl/cfssljson/cfssl-certinfo into $1 directory if they do not already exist in PATH # # Assumed vars: diff --git a/tools/imctl/README.md b/tools/imctl/README.md index d5f674b55..520df6421 100644 --- a/tools/imctl/README.md +++ b/tools/imctl/README.md @@ -1,47 +1,89 @@ -# OpenIM CTL 模块 +# [RFC #0005] OpenIM CTL Module Proposal +## Meta -## 为什么设计这个模块 +- Name: OpenIM CTL Module Enhancement +- Start Date: 2023-08-23 +- Author(s): @cubxxw +- Status: Draft +- RFC Pull Request: (leave blank) +- OpenIMSDK Pull Request: (leave blank) +- OpenIMSDK Issue: https://github.com/OpenIMSDK/Open-IM-Server/issues/924 +- Supersedes: N/A -OpenIM 后期功能扩展，不能总依赖一些单独的模块，而是整合到 Imctl 中。 +## 📇Topics -测试同学做自动化测试或者是 e2e 测试，接口测试等等，每一次调用 API 很麻烦，用 imctl 的方式为 api 的调用提供了方便 +- RFC #0000 OpenIMSDK CTL Module Proposal + - [Meta](#meta) + - [Summary](#summary) + - [Definitions](#definitions) + - [Motivation](#motivation) + - [What it is](#what-it-is) + - [How it Works](#how-it-works) + - [Migration](#migration) + - [Drawbacks](#drawbacks) + - [Alternatives](#alternatives) + - [Prior Art](#prior-art) + - [Unresolved Questions](#unresolved-questions) + - [Spec. Changes (OPTIONAL)](#spec-changes-optional) + - [History](#history) -和 scripts 深度交互，同样减少了 IM 本身的耦合度，提高了 IM 的可扩展性。 +## Summary +The OpenIM CTL module proposal aims to provide an integrated tool for the OpenIM system, offering utilities for user management, system monitoring, debugging, configuration, and more. This tool will enhance the extensibility of the OpenIM system and reduce dependencies on individual modules. -## 功能设计 +## Definitions -+ 用户管理：例如，添加、删除或禁用用户账户。 +- **OpenIM**: An Instant Messaging system. +- **`imctl`**: The control command-line tool for OpenIM. +- **E2E Testing**: End-to-End Testing. +- **API**: Application Programming Interface. -+ 系统监控：查看在线用户数量、消息传送速率等关键性能指标。 +## Motivation -+ 调试：如查看日志、调整日志级别、查看系统状态等。 +- Improve the OpenIM system's extensibility and reduce dependencies on individual modules. +- Simplify the process for testers to perform automated tests. +- Enhance interaction with scripts and reduce the system's coupling. +- Implement a consistent tool similar to kubectl for a streamlined user experience. -+ 配置管理：更新系统设置、管理插件或模块等。 +## What it is -+ 数据管理：备份和恢复数据、导入和导出数据等。 +`imctl` is a command-line utility designed for OpenIM to provide functionalities including: -+ 系统维护：例如，执行更新、重启服务、进行维护模式等。 +- User Management: Add, delete, or disable user accounts. +- System Monitoring: View metrics like online users, message transfer rate. +- Debugging: View logs, adjust log levels, check system states. +- Configuration Management: Update system settings, manage plugins/modules. +- Data Management: Backup, restore, import, or export data. +- System Maintenance: Update, restart services, or maintenance mode. +## How it Works -## 设计思路 +`imctl`, inspired by kubectl, will have sub-commands and options for the functionalities mentioned. Developers, operations, and testers can invoke these commands to manage and monitor the OpenIM system. -参考 kubectl, 方便 开发、运维、测试同学使用系统功能，并且实现自动化功能。 +## Migration -**自动化设计思路：** -1. 为后面的扩展子模块或者子命令提供自动化的功能，提供子命令 `imctl new` 自动创建新的子命令 -2. 以通过 imctl 对用户、密钥和策略进行CURD操作 -3. 设置 imctl 自动补全脚本 -4. 版本管理，问题：https://github.com/OpenIMSDK/Open-IM-Server/issues/574，做 IM 自动化的版本管理，查看 IM 系统版本 +Currently, the `imctl` will be housed in `tools/imctl`, and later on, the plan is to move it to `cmd/imctl`. Migration guidelines will be provided to ensure smooth transitions. -一些简单的 IM 解决方案可能不需要这样的工具，而复杂、高度定制的系统可能会从中受益。 +## Drawbacks -所以暂时将这个模块放在 tools/imctl 中，后面迁移到 cmd/imctl 中 +- Overhead in learning and adapting to a new tool for existing users. +- Potential complexities in implementing some of the advanced functionalities. +## Alternatives -## 目录结构设计 +- Continue using individual modules for OpenIM management. +- Utilize third-party tools or platforms with similar functionalities, customizing them for OpenIM. -为了方便迁移，将 imctl 工程化设计，命令工具放入到 tools/imctl/cmd 中，其他的模块放入到 tools/imctl/pkg 中 +## Prior Art -``` \ No newline at end of file +Kubectl from Kubernetes is a significant inspiration for `imctl`, offering a comprehensive command-line tool for managing clusters. + +## Unresolved Questions + +- What other functionalities might be required in future versions of `imctl`? +- What's the expected timeline for transitioning from `tools/imctl` to `cmd/imctl`? + +## Spec. Changes (OPTIONAL) + +As of now, there are no proposed changes to the core specifications or extensions. Future changes based on community feedback might necessitate spec changes, which will be documented accordingly. \ No newline at end of file