diff --git a/README.md b/README.md index f3391c786..38285ef9b 100644 --- a/README.md +++ b/README.md @@ -29,3 +29,336 @@

+## ✨ About OpenIM + +Open-IM-Server, meticulously developed in pure Golang, is a powerful instant messaging server. Its distinct communication method, employing pb+websocket, views every interaction as a message, streamlining customization without altering server code. Built on a microservice architecture, it offers deployment through clusters for outstanding performance and scalability. + +Open-IM-Server is more than an instant messaging server; it's a powerful tool for incorporating real-time networking into your applications, positioning itself as your premier choice for integration! 🚀 + +Please be aware that Open-IM-Server does not function as a standalone product and does not offer built-in account registration or login services. To ease your implementation process, we've open-sourced the [chat repository](https://github.com/OpenIMSDK/chat), which comprises these features. Deploying this chat business server in conjunction with Open-IM-Server expedites the setup of a comprehensive chat product. 👥 + +Further enhancing your experience, we also provide an SDK client, wherein most complex logics are implemented. The [SDK repository](https://github.com/OpenIMSDK/openim-sdk-core) can be found at [this link](https://github.com/OpenIMSDK/openim-sdk-core). The [chat repository](https://github.com/OpenIMSDK/chat) is our business server while the 'core' represents the high-level encapsulation of the SDK, synergistically working together to deliver superior results. ✨ + +## :star2: Why OpenIM + +**🔍 Function screenshot display** + +
+ +| 💻🔄📱 Multi Terminal Synchronization 🔄🖥️ | 📅⚡ Efficient Meetings 🚀💼 | +| :----------------------------------------------------------: | :---------------------------------------------------------: | +| ![multiple-message](./assets/demo/multi-terminal-synchronization.png) | ![efficient-meetings](./assets/demo/efficient-meetings.png) | +| 📲🔄 **One-to-one and Group Chats** 👥🗣️ | 🎁💻 **Special Features - Custom Messages** ✉️🎨| +| ![group-chat](./assets/demo/group-chat.png) | ![special-function](./assets/demo/special-function.png) | + +
+ +1. **Comprehensive Message Type Support :speech_balloon:** + + ✅ Supports almost all types of messages, including text, images, emojis, voice, video, geographical location, files, quotes, business cards, system notifications, custom messages and more + + ✅ Supports one-on-one and multi-person audio and video calls + + ✅ Provides terminal support for multiple platforms such as iOS, Android, Flutter, uni-app, ReactNative, Electron, Web, H5 + +2. **Efficient Meetings Anytime, Anywhere :earth_americas:** + + ✅ Based on IM (Instant Messaging) with 100% reliable forced signaling capabilities, it paves the way for IM systems, deeply integrated with chat applications + + ✅ Supports hundreds of people in a single meeting, with subscription numbers reaching thousands, and server-side audio and video recording + +3. **One-on-one and Group Chats for Various Social Scenarios :busts_in_silhouette:** + + ✅ OpenIM has four roles: application administrator, group owner, group administrator, and regular member + + ✅ Powerful group features such as muting, group announcements, group validation, unlimited group members, and loading group messages as needed + +4. **Unique Features :star2:** + + ✅ Supports read-and-burn private chats, customizable duration + + ✅ Message editing function broadens social scenarios, making instant communication more diverse and interesting + +5. **Open Source :open_hands:** + + ✅ The code of OpenIM is open source, self-controlled data, aimed at building a globally leading IM open source community, including client SDK and server + + ✅ Based on open source Server, many excellent open source projects have been developed, such as [OpenKF](https://github.com/OpenIMSDK/OpenKF) (Open source AI customer service system) + +6. **Easy to Expand :wrench:** + + ✅ The OpenIM server is implemented in Golang, introducing an innovative "everything is a message" communication model, simplifying the implementation of custom messages and extended features + +7. **High Performance :racing_car:** + + ✅ OpenIM supports a hierarchical governance architecture in the cluster, tested by a large number of users, and abstracts the storage model of online messages, offline messages, and historical messages + +8. **Full Platform Support :tv:** + + ✅ Supports native iOS, Android; cross-platform Flutter, uni-app, ReactNative; major web front-end frameworks such as React, Vue; applets; and PC platforms supported by Electron + +9. **The ultimate deployment experience 🤖** + + ✅ Supports [cluster deployment](https://github.com/OpenIMSDK/Open-IM-Server/edit/main/deployments/README.md) + + ✅ Supports multi-architecture mirroring, our Docker images are hosted not only on GitHub but also on Alibaba Cloud and Docker Hub supporting multiple architectures. Visit [our GitHub packages](https://github.com/orgs/OpenIMSDK/packages?repo_name=Open-IM-Server) and read our [version management document](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md) for more information. + +11. **A large ecosystem of open source communities 🤲** + + ✅ We have tens of thousands of users and many solutions to problems. + + ✅ We have a large open source community called [OpenIMSDK](https://github.com/OpenIMSDK) that runs the core modules, we have an open source community called [openim-sigs](https://github.com/openim-sigs) to explore more IM-based infrastructure products. + +## :rocket: Quick Start + +
Deploying with Docker Compose + +1. Clone the project + +```bash +# choose what you need +BRANCH=release-v3.1 +git clone -b $BRANCH https://github.com/OpenIMSDK/Open-IM-Server openim && export openim=$(pwd)/openim && cd $openim && make build +``` + +> **Note** +> Read our release policy: https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md + +2. Modify `.env` + +```bash +USER=root #no need to modify +PASSWORD=openIM123 #A combination of 8 or more numbers and letters, this password applies to redis, mysql, mongo, as well as accessSecret in config/config.yaml +ENDPOINT=http://127.0.0.1:10005 #minio's external service IP and port, or use the domain name storage.xx.xx, the app must be able to access this IP and port or domain, +API_URL=http://127.0.0.1:10002/object/ #the app must be able to access this IP and port or domain, +DATA_DIR=./ #designate large disk directory +``` + +3. Deploy and start + +> **Note** +> This command can only be executed once. It will modify the component passwords in docker-compose based on the `PASSWORD` variable in `.env`, and modify the component passwords in `config/config.yaml`. If the password in `.env` changes, you need to first execute `docker-compose down`; `rm components -rf` and then execute this command. + +```bash +make install +``` + +4. Check the service + +```bash +make check +``` + +![https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/images/docker_build.png](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/images/docker_build.png) + +
+ +
Compile from Source + +Ur need `Go 1.18` or higher version, and `make`. + +Version Details: https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md + +```bash +# choose what you need +BRANCH=release-v3.1 +git clone -b $BRANCH https://github.com/OpenIMSDK/Open-IM-Server openim && export openim=$(pwd)/openim && cd $openim && make build +``` + +Read about the [OpenIM Version Policy](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md) + +`make help` to help you see the instructions supported by OpenIM. + +All services have been successfully built as shown in the figure + +![Successful Compilation](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/images/build.png) + +
+ +
Component Configuration Instructions + +The config/config.yaml file has detailed configuration instructions for the storage components. + +- Zookeeper + + - Used for RPC service discovery and registration, cluster support. + + ```bash + zookeeper: + schema: openim #Not recommended to modify + address: [ 127.0.0.1:2181 ] #address + username: #username + password: #password + ``` + +- MySQL + + - Used for storing users, relationships, and groups, supports master-slave database. + + ```bash + mysql: + address: [ 127.0.0.1:13306 ] #address + username: root #username + password: openIM123 #password + database: openIM_v2 #Not recommended to modify + maxOpenConn: 1000 #maximum connection + maxIdleConn: 100 #maximum idle connection + maxLifeTime: 60 #maximum time a connection can be reused (seconds) + logLevel: 4 #log level 1=slient 2=error 3=warn 4=info + slowThreshold: 500 #slow statement threshold (milliseconds) + ``` + +- Mongo + + - Used for storing offline messages, supports mongo sharded clusters. + + ```bash + mongo: + uri: #Use this value directly if not empty + address: [ 127.0.0.1:37017 ] #address + database: openIM #default mongo db + username: root #username + password: openIM123 #password + maxPoolSize: 100 #maximum connections + ``` + +- Redis + + - Used for storing message sequence numbers, latest messages, user tokens, and mysql cache, supports cluster deployment. + + ```bash + redis: + address: [ 127.0.0.1:16379 ] #address + username: #username + password: openIM123 #password + ``` + +- Kafka + + - Used for message queues, for message decoupling, supports cluster deployment. + + ```bash + kafka: + username: #username + password: #password + addr: [ 127.0.0.1:9092 ] #address + latestMsgToRedis: + topic: "latestMsgToRedis" + offlineMsgToMongo: + topic: "offlineMsgToMongoMysql" + msgToPush: + topic: "msqToPush" + msgToModify: + topic: "msgToModify" + consumerGroupID: + msgToRedis: redis + msgToMongo: mongo + msgToMySql: mysql + msgToPush: push + msgToModify: modify + ``` + +
+ +
Start and Stop Services + +Start services + +``` +./scripts/start_all.sh; +``` + +Check services + +``` +./scripts/check_all.sh +``` + +Stop services + +``` +./scripts/stop_all.sh +``` + +
+ +
Open IM Ports + +| TCP Port | Description | Operation | +| --------- | ------------------------------------------------------------ | ----------------------------------------------------- | +| TCP:10001 | ws protocol, message port such as message sending, pushing etc, used for client SDK | Port release or nginx reverse proxy, and firewall off | +| TCP:10002 | api port, such as user, friend, group, message interfaces. | Port release or nginx reverse proxy, and firewall off | +| TCP:10005 | Required when choosing minio storage (openIM uses minio storage by default) | Port release or nginx reverse proxy, and firewall off | + +
+ +
Open Chat Ports + ++ chat warehouse: https://github.com/OpenIMSDK/chat + +| TCP Port | Description | Operation | +| --------- | --------------------------------------------------- | ----------------------------------------------------- | +| TCP:10008 | Business system, such as registration, login etc | Port release or nginx reverse proxy, and firewall off | +| TCP:10009 | Management backend, such as statistics, banning etc | Port release or nginx reverse proxy, and firewall off | + +
+ +## :link: Relationship Between APP and OpenIM + +OpenIM isn't just an open-source instant messaging component, it's an integral part of your application ecosystem. Check out this diagram to understand how AppServer, AppClient, Open-IM-Server, and Open-IM-SDK interact. + +![App-OpenIM Relationship](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/images/open-im-server.png) + +## :building_construction: Overall Architecture + +Delve into the heart of Open-IM-Server's functionality with our architecture diagram. + +![Overall Architecture](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/images/Architecture.jpg) + +## :hammer_and_wrench: To start developing OpenIM + +OpenIM Our goal is to build a top-level open source community. We have a set of standards, in the [Community repository](https://github.com/OpenIMSDK/community). + +If you'd like to contribute to this Open-IM-Server repository, please read our [contributor documentation](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/CONTRIBUTING.md). + +Before you start, please make sure your changes are in demand. The best for that is to create a [new discussion](https://github.com/OpenIMSDK/Open-IM-Server/discussions/new/choose) OR [Slack Communication](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg), or if you find an issue, [report it](https://github.com/OpenIMSDK/Open-IM-Server/issues/new/choose) first. + +- [Code Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/go_code.md) +- [Docker Images Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/images.md) +- [Directory Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/directory.md) +- [Commit Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/commit.md) +- [Versioning Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/version.md) +- [Interface Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/api.md) +- [Log Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/log.md) +- [Error Code Standards](https://github.com/OpenIMSDK/Open-IM-Server/blob/main/docs/conversions/error_code.md) + +## :busts_in_silhouette: Community + ++ 📚 [OpenIM Community](https://github.com/OpenIMSDK/community) ++ 💕 [OpenIM Interest Group](https://github.com/Openim-sigs) ++ 🚀 [Join our Slack community](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg) ++ :eyes: [Join our wechat (微信群)](https://openim-1253691595.cos.ap-nanjing.myqcloud.com/WechatIMG20.jpeg) + +## :calendar: Community Meetings + +We want anyone to get involved in our community and contributing code, we offer gifts and rewards, and we welcome you to join us every Thursday night. + +Our conference is in the [OpenIM Slack](https://join.slack.com/t/openimsdk/shared_invite/zt-1tmoj26uf-_FDy3dowVHBiGvLk9e5Xkg) 🎯, then you can search the Open-IM-Server pipeline to join + +We take notes of each [biweekly meeting](https://github.com/orgs/OpenIMSDK/discussions/categories/meeting) in [GitHub discussions](https://github.com/OpenIMSDK/Open-IM-Server/discussions/categories/meeting), Our historical meeting notes, as well as replays of the meetings are available at [Google Docs :bookmark_tabs:](https://docs.google.com/document/d/1nx8MDpuG74NASx081JcCpxPgDITNTpIIos0DS6Vr9GU/edit?usp=sharing). + +## :eyes: Who are using OpenIM + +Check out our [user case studies](https://github.com/OpenIMSDK/community/blob/main/ADOPTERS.md) page for a list of the project users. Don't hesitate to leave a [📝comment](https://github.com/OpenIMSDK/Open-IM-Server/issues/379) and share your use case. + +## :page_facing_up: License + +OpenIM is licensed under the Apache 2.0 license. See [LICENSE](https://github.com/OpenIMSDK/Open-IM-Server/tree/main/LICENSE) for the full license text. + +The OpenIM logo, including its variations and animated versions, displayed in this repository [OpenIM](https://github.com/OpenIMSDK/Open-IM-Server) under the [assets/logo](./assets/logo) and [assets/logo-gif](assets/logo-gif) directories, are protected by copyright laws. + +## 🔮 Thanks to our contributors! + + + + diff --git a/deployments/templates/init/openim-crontask.service b/deployments/templates/init/openim-crontask.service index 861ba1d20..50241b3e4 100644 --- a/deployments/templates/init/openim-crontask.service +++ b/deployments/templates/init/openim-crontask.service @@ -6,7 +6,7 @@ Documentation=https://github.com/OpenIMSDK/Open-IM-Server/blob/main/deployment/i WorkingDirectory=${OPENIM_DATA_DIR}/openim-crontask ExecStartPre=/usr/bin/mkdir -p ${OPENIM_DATA_DIR}/openim-crontask ExecStartPre=/usr/bin/mkdir -p ${OPENIM_LOG_DIR} -ExecStart=${OPENIM_INSTALL_DIR}/bin/openim-crontask --config=${OPENIM_CONFIG_DIR}/openim-crontask.yaml +ExecStart=${OPENIM_INSTALL_DIR}/bin/openim-crontask -c=${OPENIM_CONFIG_DIR} Restart=always RestartSec=5 StartLimitInterval=0 diff --git a/deployments/templates/openim-crontask.yaml b/deployments/templates/openim-crontask.yaml index 04f745e92..98fd1dd2b 100644 --- a/deployments/templates/openim-crontask.yaml +++ b/deployments/templates/openim-crontask.yaml @@ -1,3 +1,384 @@ -name: openim-crontask -description: "This is a description for openim-crontask" -type: "service" +# Copyright © 2023 OpenIM. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ----------------------------------------------------------------- +# This config file is the template file +# --| source: scripts/template/config-tmpl/openim_config.yaml +# --| env: scripts/install/environment.sh +# --| target: config/config.yaml +# ----------------------------------------------------------------- + +###################### Zookeeper ###################### +# Zookeeper configuration +# It's not recommended to modify the schema +# +# Zookeeper address +# Zookeeper username +# Zookeeper password +zookeeper: + schema: "openim" + address: [ "127.0.0.1:2181" ] + username: "" + password: "" + +###################### Mysql ###################### +# MySQL configuration +# Currently, only single machine setup is supported +# +# Maximum number of open connections +# Maximum number of idle connections +# Maximum lifetime in seconds a connection can be reused +# Log level: 1=slient, 2=error, 3=warn, 4=info +# Slow query threshold in milliseconds +mysql: + address: [ "127.0.0.1:13306" ] + username: "root" + password: "openIM123" + database: "openIM_v3" + maxOpenConn: "1000" + maxIdleConn: "100" + maxLifeTime: "60" + logLevel: "4" + slowThreshold: "500" + +###################### Mongo ###################### +# MongoDB configuration +# If uri is not empty, it will be used directly +# +# MongoDB address for standalone setup, Mongos address for sharded cluster setup +# Default MongoDB database name +# Maximum connection pool size +mongo: + uri: "" + address: [ "127.0.0.1:37017" ] + database: "openIM_v3" + username: "root" + password: "openIM123" + maxPoolSize: "100" + +###################### Redis configuration information ###################### +# Redis configuration +# +# Username is required only for Redis version 6.0+ +redis: + address: [ "127.0.0.1:16379" ] + username: "" + password: "openIM123" + +###################### Kafka configuration information ###################### +# Kafka configuration +# +# Kafka username +# Kafka password +# It's not recommended to modify this topic name +# Consumer group ID, it's not recommended to modify +kafka: + username: "" + password: "" + addr: [ "127.0.0.1:9092" ] + latestMsgToRedis: + topic: "latestMsgToRedis" + offlineMsgToMongo: + topic: "offlineMsgToMongoMysql" + msgToPush: + topic: "msgToPush" + consumerGroupID: + msgToRedis: "redis" + msgToMongo: "mongo" + msgToMySql: "mysql" + msgToPush: "push" + +###################### RPC configuration information ###################### +# RPC configuration +# +# IP address to register with zookeeper when starting RPC, the IP and corresponding rpcPort should be accessible by api/gateway +# Default listen IP is 0.0.0.0 +rpc: + registerIP: "" + listenIP: "0.0.0.0" + +###################### API configuration information ###################### +# API configuration +# +# API service port +# Default listen IP is 0.0.0.0 +api: + openImApiPort: [ "10002" ] + listenIP: "0.0.0.0" + +###################### Object configuration information ###################### +# Object storage configuration +# +# Use minio for object storage +# API URL should be accessible by the app +# It's not recommended to modify the bucket name +# Endpoint should be accessible by the app +# Session token +# Configuration for Tencent COS +# Configuration for Aliyun OSS +# It can be set by an environment variable or by a script +object: + enable: "minio" + apiURL: "http://127.0.0.1:10002/object/" + minio: + bucket: "openim" + endpoint: "http://113.90.83.134:10005" + accessKeyID: "root" + secretAccessKey: "openIM123" + sessionToken: "" + cos: + bucketURL: "https://temp-1252357374.cos.ap-chengdu.myqcloud.com" + secretID: "" + secretKey: "" + sessionToken: "" + oss: + endpoint: "https://oss-cn-chengdu.aliyuncs.com" + bucket: "demo-9999999" + bucketURL: "https://demo-9999999.oss-cn-chengdu.aliyuncs.com" + accessKeyID: "root" + accessKeySecret: "" + sessionToken: "" + + +###################### RPC Port Configuration ###################### +# RPC service ports +# These ports are passed into the program by the script and are not recommended to modify +# For launching multiple programs, just fill in multiple ports separated by commas +# For example, [10110, 10111] +rpcPort: + openImUserPort: [ "10110" ] + openImFriendPort: [ "10120" ] + openImMessagePort: [ "10130" ] + openImMessageGatewayPort: [ "10140" ] + openImGroupPort: [ "10150" ] + openImAuthPort: [ "10160" ] + openImPushPort: [ "10170" ] + openImConversationPort: [ "10180" ] + openImThirdPort: [ "10190" ] + +###################### RPC Register Name Configuration ###################### +# RPC service names for registration, it's not recommended to modify these +rpcRegisterName: + openImUserName: "User" + openImFriendName: "Friend" + openImMsgName: "Msg" + openImPushName: "Push" + openImMessageGatewayName: "MessageGateway" + openImGroupName: "Group" + openImAuthName: "Auth" + openImConversationName: "Conversation" + openImThirdName: "Third" + +###################### Log Configuration ###################### +# Log configuration +# +# Storage directory +# Log rotation time +# Maximum number of logs to retain +# Log level, 6 means all levels +# Whether to output to stdout +# Whether to output in json format +# Whether to include stack trace in logs +log: + storageLocation: "/root/workspaces/openim/openim-server/log/" + rotationTime: 24 + remainRotationCount: 2 + remainLogLevel: 6 + isStdout: false + isJson: false + withStack: false + +###################### Variables definition ###################### +# Long connection server configuration +# +# Websocket port for msg_gateway +# Maximum number of websocket connections +# Maximum length of websocket request package +# Websocket connection handshake timeout +longConnSvr: + openImWsPort: [ "10001" ] + websocketMaxConnNum: 100000 + websocketMaxMsgLen: 4096 + websocketTimeout: 10 + +# Push notification service configuration +# +# Use GeTui for push notifications +# GeTui offline push configuration +# FCM offline push configuration +# Account file, place it in the config directory +# JPush configuration, modify these after applying in JPush backend +push: + enable: "getui" + geTui: + pushUrl: "https://restapi.getui.com/v2/$appId" + masterSecret: "" + appKey: "" + intent: "" + channelID: "" + channelName: "" + fcm: + serviceAccount: "x.json" + jpns: + appKey: "" + masterSecret: "" + pushUrl: "" + pushIntent: "" + +# App manager configuration +# +# Built-in app manager user IDs +# Built-in app manager nicknames +manager: + userID: [ "openIM123456", "openIM654321", "openIMAdmin" ] + nickname: [ "system1", "system2", "system3" ] + +# Multi-platform login policy +# For each platform(Android, iOS, Windows, Mac, web), only one can be online at a time +multiLoginPolicy: 1 + +# Whether to store messages in MySQL, messages in MySQL are only used for management background +chatPersistenceMysql: true + +# Message cache timeout in seconds, it's not recommended to modify +msgCacheTimeout: 86400 + +# Whether to enable read receipts for group chat +groupMessageHasReadReceiptEnable: true + +# Whether to enable read receipts for single chat +singleMessageHasReadReceiptEnable: true + +# MongoDB offline message retention period in days +retainChatRecords: 365 + +# Schedule to clear expired messages(older than retainChatRecords days) in MongoDB every Wednesday at 2am +# This deletion is just for cleaning up disk usage according to previous configuration retainChatRecords, no notification will be sent +chatRecordsClearTime: "0 2 * * *" + +# Schedule to auto delete messages every day at 2am +# This deletion is for messages that have been retained for more than msg_destruct_time (seconds) in the conversation field +msgDestructTime: "0 2 * * *" + +# Secret key +secret: "openIM123" + +# Token policy +# +# Token expiration period in days +tokenPolicy: + expire: 90 + +# Message verification policy +# +# Whether to verify friendship when sending messages +messageVerify: + friendVerify: false + +# iOS push notification configuration +# +# iOS push notification sound +# Whether to count badge +# Whether it's production environment +iosPush: + pushSound: "xxx" + badgeCount: true + production: false + +###################### Third-party service configuration ###################### +# Callback configuration +# +# Callback URL +# Whether to enable this callback event +# Timeout in seconds +# Whether to continue execution if callback fails +callback: + url: + beforeSendSingleMsg: + enable: false + timeout: 5 + failedContinue: true + afterSendSingleMsg: + enable: false + timeout: 5 + beforeSendGroupMsg: + enable: false + timeout: 5 + failedContinue: true + afterSendGroupMsg: + enable: false + timeout: 5 + msgModify: + enable: false + timeout: 5 + failedContinue: true + userOnline: + enable: false + timeout: 5 + userOffline: + enable: false + timeout: 5 + userKickOff: + enable: false + timeout: 5 + offlinePush: + enable: false + timeout: 5 + failedContinue: true + onlinePush: + enable: false + timeout: 5 + failedContinue: true + superGroupOnlinePush: + enable: false + timeout: 5 + failedContinue: true + beforeAddFriend: + enable: false + timeout: 5 + failedContinue: true + beforeCreateGroup: + enable: false + timeout: 5 + failedContinue: true + beforeMemberJoinGroup: + enable: false + timeout: 5 + failedContinue: true + beforeSetGroupMemberInfo: + enable: false + timeout: 5 + failedContinue: true + setMessageReactionExtensions: + enable: false + timeout: 5 + failedContinue: true + +###################### Prometheus ###################### +# Prometheus configuration for various services +# The number of Prometheus ports per service needs to correspond to rpcPort +# The number of ports needs to be consistent with msg_transfer_service_num in script/path_info.sh +prometheus: + enable: "false" + userPrometheusPort: [ "20110" ] + friendPrometheusPort: [ "20120" ] + messagePrometheusPort: [ "20130" ] + messageGatewayPrometheusPort: [ "20140" ] + groupPrometheusPort: [ "20150" ] + authPrometheusPort: [ "20160" ] + pushPrometheusPort: [ "20170" ] + conversationPrometheusPort: [ "20230" ] + rtcPrometheusPort: [ "21300" ] + thirdPrometheusPort: [ "21301" ] + messageTransferPrometheusPort: [ "21400, 21401, 21402, 21403" ] # List of ports diff --git a/scripts/README.md b/scripts/README.md index d0ce8eeb5..e0cd5bd56 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,5 +1,26 @@ # OpenIM Scripts Directory Structure +- [OpenIM Scripts Directory Structure](#openim-scripts-directory-structure) + - [log directory](#log-directory) + - [Supported platforms](#supported-platforms) + - [Get started quickly - demo.sh](#get-started-quickly---demosh) + - [Guide: Using and Understanding OpenIM Utility Functions](#guide-using-and-understanding-openim-utility-functions) + - [Table of Contents](#table-of-contents) + - [1. Checking the Status of Services by Ports](#1-checking-the-status-of-services-by-ports) + - [Function: `openim::util::check_ports`](#function-openimutilcheck_ports) + - [Example:](#example) + - [2. Checking the Status of Services by Process Names](#2-checking-the-status-of-services-by-process-names) + - [Function: `openim::util::check_process_names`](#function-openimutilcheck_process_names) + - [Example:](#example-1) + - [3. Stopping Services by Ports](#3-stopping-services-by-ports) + - [Function: `openim::util::stop_services_on_ports`](#function-openimutilstop_services_on_ports) + - [Example:](#example-2) + - [4. Stopping Services by Process Names](#4-stopping-services-by-process-names) + - [Function: `openim::util::stop_services_with_name`](#function-openimutilstop_services_with_name) + - [Example:](#example-3) + - [examples](#examples) + + This document outlines the directory structure for scripts in the OpenIM Server project. These scripts play a critical role in various areas like building, deploying, running and managing the services of OpenIM. ```bash @@ -126,6 +147,119 @@ asciinema rec svg-term --cast= --out _output/demo.svg --window ``` +Here you will learn how to test a script, We take the four functions for starting and checking a service as an example. + +## Guide: Using and Understanding OpenIM Utility Functions + +This document provides an overview of the four utility functions designed for managing processes and services. These functions can check the status of services based on ports and process names, as well as stop services based on the same criteria. + +### Table of Contents +- [1. Checking the Status of Services by Ports](#checking-the-status-of-services-by-ports) +- [2. Checking the Status of Services by Process Names](#checking-the-status-of-services-by-process-names) +- [3. Stopping Services by Ports](#stopping-services-by-ports) +- [4. Stopping Services by Process Names](#stopping-services-by-process-names) + +### 1. Checking the Status of Services by Ports + +#### Function: `openim::util::check_ports` + +This function checks the status of services running on specified ports. + +**Usage**: + +```bash +openim::util::check_ports ... +``` + +**Design**: + +- The function iterates through each provided port. +- It uses the `lsof` command to identify if there is a service running on the specified port. +- If a service is running, it logs the command, PID, and start time of the service. +- If a service is not running, it logs that the port is not started. +- If any service is not running, the function returns a status of 1. + +#### Example: + +```bash +openim::util::check_ports 8080 8081 8082 +``` + +### 2. Checking the Status of Services by Process Names + +#### Function: `openim::util::check_process_names` + +This function checks the status of services based on their process names. + +**Usage**: + +```bash +openim::util::check_process_names ... +``` + +**Design**: + +- The function uses `pgrep` to find process IDs associated with the given process names. +- If processes are found, it logs the command, PID, associated port, and start time. +- If no processes are found for a name, it logs that the process is not started. +- If any process is not running, the function returns a status of 1. + +#### Example: + +```bash +openim::util::check_process_names nginx mysql redis +``` + +### 3. Stopping Services by Ports + +#### Function: `openim::util::stop_services_on_ports` + +This function attempts to stop services running on the specified ports. + +**Usage**: + +```bash +openim::util::stop_services_on_ports ... +``` + +**Design**: + +- The function uses the `lsof` command to identify services running on the specified ports. +- If a service is running on a port, it tries to terminate the associated process using the `kill` command. +- It logs successful terminations and any failures. +- If any service couldn't be stopped, the function returns a status of 1. + +#### Example: + +```bash +openim::util::stop_services_on_ports 8080 8081 8082 +``` + +### 4. Stopping Services by Process Names + +#### Function: `openim::util::stop_services_with_name` + +This function attempts to stop services based on their process names. + +**Usage**: + +```bash +openim::util::stop_services_with_name ... +``` + +**Design**: + +- The function uses `pgrep` to identify processes associated with the specified names. +- If processes are found, it tries to terminate them using the `kill` command. +- It logs successful terminations and any failures. +- If any service couldn't be stopped, the function returns a status of 1. + +#### Example: + +```bash +openim::util::stop_services_with_name nginx apache +``` + ## examples Scripts to perform various build, install, analysis, etc operations. diff --git a/scripts/install/environment.sh b/scripts/install/environment.sh index 00ddb9879..41fb1160e 100755 --- a/scripts/install/environment.sh +++ b/scripts/install/environment.sh @@ -137,6 +137,8 @@ def "API_OPENIM_PORT" "10002" # API的开放端口 def "API_LISTEN_IP" "0.0.0.0" # API的监听IP ###################### RPC Port Configuration Variables ###################### +# For launching multiple programs, just fill in multiple ports separated by commas +# For example, [10110, 10111] def "OPENIM_USER_PORT" "10110" # OpenIM用户服务端口 def "OPENIM_FRIEND_PORT" "10120" # OpenIM朋友服务端口 def "OPENIM_MESSAGE_PORT" "10130" # OpenIM消息服务端口 diff --git a/scripts/install/openim-crontask b/scripts/install/openim-crontask deleted file mode 100755 index eee30295c..000000000 --- a/scripts/install/openim-crontask +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env bash - -# Copyright © 2023 OpenIM. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -o errexit -set +o nounset -set -o pipefail - -OPENIM_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd -P) -[[ -z ${COMMON_SOURCED} ]] && source ${OPENIM_ROOT}/scripts/install/common.sh - -SERVER_NAME="openim-crontask" - -openim::log::status "Start OpenIM Cron, binary root: ${SERVER_NAME}" -openim::log::info "Start OpenIM Cron, path: ${OPENIM_CRONTASK_BINARY}" - -# openim::util::stop_services_with_name ${SERVER_NAME} - -# sleep 1 - -# openim::log::status "start cron_task process, path: ${OPENIM_CRONTASK_BINARY}" -# nohup ${OPENIM_CRONTASK_BINARY} >>${LOG_FILE} 2>&1 & -# openim::util::check_process_names ${SERVER_NAME} - -# # Print the necessary information after installation -# function openim::crontask::info() { -# cat << EOF -# openim-crontask listen on: ${OPENIM_CRONTASK_HOST} -# EOF -# } - -# install openim-crontask -function openim::crontask::install() -{ - pushd ${OPENIM_ROOT} - - # 1. Build openim-crontask - make build BINS=${SERVER_NAME} - openim::common::sudo "cp ${OPENIM_OUTPUT_HOSTBIN}/${SERVER_NAME} ${OPENIM_INSTALL_DIR}/bin" - - # 2. Generate and install the openim-crontask configuration file (openim-crontask.yaml) - echo ${LINUX_PASSWORD} | sudo -S bash -c \ - "./scripts/genconfig.sh ${ENV_FILE} deployments/templates/openim-crontask.yaml > ${OPENIM_CONFIG_DIR}/openim-crontask.yaml" - - # 3. Create and install the openim-crontask systemd unit file - echo ${LINUX_PASSWORD} | sudo -S bash -c \ - "./scripts/genconfig.sh ${ENV_FILE} deployments/templates/init/openim-crontask.service > /etc/systemd/system/openim-crontask.service" - - # 4. Start the openim-crontask service - openim::common::sudo "systemctl daemon-reload" - openim::common::sudo "systemctl restart openim-crontask" - openim::common::sudo "systemctl enable openim-crontask" - openim::crontask::status || return 1 - openim::crontask::info - - openim::log::info "install openim-crontask successfully" - popd -} - -# Unload -function openim::crontask::uninstall() -{ - set +o errexit - openim::common::sudo "systemctl stop openim-crontask" - openim::common::sudo "systemctl disable openim-crontask" - openim::common::sudo "rm -f ${OPENIM_INSTALL_DIR}/bin/openim-crontask" - openim::common::sudo "rm -f ${OPENIM_CONFIG_DIR}/openim-crontask.yaml" - openim::common::sudo "rm -f /etc/systemd/system/openim-crontask.service" - set -o errexit - openim::log::info "uninstall openim-crontask successfully" -} - -# Status Check -function openim::crontask::status() -{ - # 查看 openim-crontask 运行状态,如果输出中包含 active (running) 字样说明 openim-crontask 成功启动。 - systemctl status openim-crontask|grep -q 'active' || { - openim::log::error "openim-crontask failed to start, maybe not installed properly" - return 1 - } - - # 监听端口在配置文件中是 hardcode - if echo | telnet 127.0.0.1 7070 2>&1|grep refused &>/dev/null;then - openim::log::error "cannot access health check port, openim-crontask maybe not startup" - return 1 - fi -} - -if [[ "$*" =~ openim::crontask:: ]];then - eval $* -fi diff --git a/scripts/install/push_start.sh b/scripts/install/push_start.sh index 36840945f..a4beb4145 100755 --- a/scripts/install/push_start.sh +++ b/scripts/install/push_start.sh @@ -13,8 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Common utilities, variables and checks for all build scripts. - set -o errexit set +o nounset set -o pipefail @@ -22,9 +20,94 @@ set -o pipefail OPENIM_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd -P) [[ -z ${COMMON_SOURCED} ]] && source ${OPENIM_ROOT}/scripts/install/common.sh -#Include shell font styles and some basic information -source $SCRIPTS_ROOT/lib/init.sh -source $SCRIPTS_ROOT/path_info.sh +SERVER_NAME="openim-push" + +# openim::log::status "Start OpenIM Push, binary root: ${SERVER_NAME}" +# openim::log::info "Start OpenIM Push, path: ${OPENIM_PUSH_BINARY}" + +# openim::util::stop_services_with_name ${SERVER_NAME} + +# openim::log::status "start push process, path: ${OPENIM_PUSH_BINARY}" +# nohup ${OPENIM_PUSH_BINARY} >>${LOG_FILE} 2>&1 & +# openim::util::check_process_names ${SERVER_NAME} + +###################################### Linux Systemd ###################################### +SYSTEM_FILE_PATH="/etc/systemd/system/${SERVER_NAME}.service" + +# Print the necessary information after installation +function openim::push::info() { +cat << EOF +openim-push listen on: ${OPENIM_PUSH_HOST} +EOF +} + +# install openim-push +function openim::push::install() +{ + pushd ${OPENIM_ROOT} + + # 1. Build openim-push + make build BINS=${SERVER_NAME} + openim::common::sudo "cp ${OPENIM_OUTPUT_HOSTBIN}/${SERVER_NAME} ${OPENIM_INSTALL_DIR}/bin" + + openim::log::status "${SERVER_NAME} binary: ${OPENIM_INSTALL_DIR}/bin/${SERVER_NAME}" + + # 2. Generate and install the openim-push configuration file (openim-push.yaml) + echo ${LINUX_PASSWORD} | sudo -S bash -c \ + "./scripts/genconfig.sh ${ENV_FILE} deployments/templates/${SERVER_NAME}.yaml > ${OPENIM_CONFIG_DIR}/${SERVER_NAME}.yaml" + openim::log::status "${SERVER_NAME} config file: ${OPENIM_CONFIG_DIR}/${SERVER_NAME}.yaml" + + # 3. Create and install the ${SERVER_NAME} systemd unit file + echo ${LINUX_PASSWORD} | sudo -S bash -c \ + "./scripts/genconfig.sh ${ENV_FILE} deployments/templates/init/${SERVER_NAME}.service > ${SYSTEM_FILE_PATH}" + openim::log::status "${SERVER_NAME} systemd file: ${SYSTEM_FILE_PATH}" + + # 4. Start the openim-push service + openim::common::sudo "systemctl daemon-reload" + openim::common::sudo "systemctl restart ${SERVER_NAME}" + openim::common::sudo "systemctl enable ${SERVER_NAME}" + openim::push::status || return 1 + openim::push::info + + openim::log::info "install ${SERVER_NAME} successfully" + popd +} + + +# Unload +function openim::push::uninstall() +{ + set +o errexit + openim::common::sudo "systemctl stop ${SERVER_NAME}" + openim::common::sudo "systemctl disable ${SERVER_NAME}" + openim::common::sudo "rm -f ${OPENIM_INSTALL_DIR}/bin/${SERVER_NAME}" + openim::common::sudo "rm -f ${OPENIM_CONFIG_DIR}/${SERVER_NAME}.yaml" + openim::common::sudo "rm -f /etc/systemd/system/${SERVER_NAME}.service" + set -o errexit + openim::log::info "uninstall ${SERVER_NAME} successfully" +} + +# Status Check +function openim::push::status() +{ + # Check the running status of the ${SERVER_NAME}. If active (running) is displayed, the ${SERVER_NAME} is started successfully. + systemctl status ${SERVER_NAME}|grep -q 'active' || { + openim::log::error "${SERVER_NAME} failed to start, maybe not installed properly" + return 1 + } + + # The listening port is hardcode in the configuration file + if echo | telnet 127.0.0.1 7071 2>&1|grep refused &>/dev/null;then # Assuming a different port for push + openim::log::error "cannot access health check port, ${SERVER_NAME} maybe not startup" + return 1 + fi +} + +if [[ "$*" =~ ${SERVER_NAME}:: ]];then + eval $* +fi + + cd $SCRIPTS_ROOT diff --git a/scripts/lib/util.sh b/scripts/lib/util.sh index 6a6fb34dc..83da3b6e4 100755 --- a/scripts/lib/util.sh +++ b/scripts/lib/util.sh @@ -18,6 +18,10 @@ # Usage: source scripts/lib/util.sh ################################################################################ +# TODO Debug: Just for testing, please comment out +# OPENIM_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd -P) +# source "${OPENIM_ROOT}/scripts/lib/logging.sh" + #1、将IP写在一个文件里,比如文件名为hosts_file,一行一个IP地址。 #2、修改ssh-mutual-trust.sh里面的用户名及密码,默认为root用户及密码123。 # hosts_file_path="path/to/your/hosts/file" @@ -260,6 +264,7 @@ openim::util::check_ports() { # An array to collect information about processes that are running. local started=() + openim::log::info "Checking ports: $*" # Iterate over each given port. for port in "$@"; do # Use the `lsof` command to find process information related to the given port. @@ -303,6 +308,7 @@ openim::util::check_ports() { return 0 fi } +# openim::util::check_ports 9090 9092 # The `openim::util::check_process_names` function analyzes the state of processes based on given names. # It accepts multiple process names as arguments and prints: @@ -312,33 +318,37 @@ openim::util::check_ports() { # openim::util::check_process_names nginx mysql redis # The function returns a status of 1 if any of the processes is not running. openim::util::check_process_names() { - # An array to collect names of processes that are not running. + # Arrays to collect details of processes local not_started=() - - # An array to collect information about processes that are running. local started=() - # Iterate over each given process name. + openim::log::info "Checking processes: $*" + # Iterate over each given process name for process_name in "$@"; do - # Use the `pgrep` command to find process information related to the given process name. - local pid=$(pgrep -f $process_name) - # If there's no process information, it means the process with the given name is not running. - if [[ -z $pid ]]; then + # Use `pgrep` to find process IDs related to the given process name + local pids=($(pgrep -f $process_name)) + + # Check if any process IDs were found + if [[ ${#pids[@]} -eq 0 ]]; then not_started+=($process_name) else - # If there's process information, extract relevant details: - # Command Name, and Start Time. - # local pid=$(echo $info | awk '{print $2}') - local command=$(ps -p $pid -o cmd=) - local start_time=$(ps -p $pid -o lstart=) - started+=("Process $process_name - Command: $command, PID: $pid, Start time: $start_time") - echo "---------------command=$command" - echo "---------------pid=$pid" - echo "---------------start_time=$start_time" + # If there are PIDs, loop through each one + for pid in "${pids[@]}"; do + local command=$(ps -p $pid -o cmd=) + local start_time=$(ps -p $pid -o lstart=) + local port=$(ss -ltnp 2>/dev/null | grep $pid | awk '{print $4}' | cut -d ':' -f2) + + # Check if port information was found for the PID + if [[ -z $port ]]; then + port="N/A" + fi + + started+=("Process $process_name - Command: $command, PID: $pid, Port: $port, Start time: $start_time") + done fi done - # Print information about processes which are not running. + # Print information if [[ ${#not_started[@]} -ne 0 ]]; then openim::log::info "Not started processes:" for process_name in "${not_started[@]}"; do @@ -346,7 +356,6 @@ openim::util::check_process_names() { done fi - # Print information about processes which are running. if [[ ${#started[@]} -ne 0 ]]; then echo openim::log::info "Started processes:" @@ -355,7 +364,7 @@ openim::util::check_process_names() { done fi - # If any of the processes is not running, return a status of 1. + # Return status if [[ ${#not_started[@]} -ne 0 ]]; then return 1 else @@ -363,6 +372,7 @@ openim::util::check_process_names() { return 0 fi } +# openim::util::check_process_names docker-pr # The `openim::util::stop_services_on_ports` function stops services running on specified ports. # It accepts multiple ports as arguments and performs the following: @@ -378,22 +388,25 @@ openim::util::stop_services_on_ports() { # An array to collect information about processes that were stopped. local stopped=() + openim::log::info "Stopping services on ports: $*" # Iterate over each given port. for port in "$@"; do # Use the `lsof` command to find process information related to the given port. info=$(lsof -i :$port -n -P | grep LISTEN || true) - + # If there's process information, it means the process associated with the port is running. if [[ -n $info ]]; then # Extract the Process ID. - local pid=$(echo $info | awk '{print $2}') - - # Try to stop the service by killing its process. - if kill -TERM $pid; then - stopped+=($port) - else - not_stopped+=($port) - fi + while read -r line; do + local pid=$(echo $line | awk '{print $2}') + + # Try to stop the service by killing its process. + if kill -TERM $pid; then + stopped+=($port) + else + not_stopped+=($port) + fi + done <<< "$info" fi done @@ -422,6 +435,10 @@ openim::util::stop_services_on_ports() { return 0 fi } +# nc -l -p 12345 +# nc -l -p 123456 +# ps -ef | grep "nc -l" +# openim::util::stop_services_on_ports 1234 12345 # The `openim::util::stop_services_with_name` function stops services with specified names. @@ -438,22 +455,39 @@ openim::util::stop_services_with_name() { # An array to collect information about processes that were stopped. local stopped=() + openim::log::info "Stopping services with names: $*" # Iterate over each given service name. for server_name in "$@"; do # Use the `pgrep` command to find process IDs related to the given service name. - local pids=$(ps aux | awk -v pattern="$server_name" '$0 ~ pattern {print $2}') + local pids=$(pgrep -f "$server_name") + # If no process was found with the name, add it to the not_stopped list + if [[ -z $pids ]]; then + not_stopped+=("$server_name") + continue + fi + local stopped_this_time=false for pid in $pids; do + + # Exclude the PID of the current script + if [[ "$pid" == "$$" ]]; then + continue + fi + # If there's a Process ID, it means the service with the name is running. if [[ -n $pid ]]; then # Try to stop the service by killing its process. - if kill -TERM $pid; then - stopped+=($server_name) - else - not_stopped+=($server_name) + if kill -TERM $pid 2>/dev/null; then + stopped_this_time=true fi fi done + + if $stopped_this_time; then + stopped+=("$server_name") + else + not_stopped+=("$server_name") + fi done # Print information about services whose processes couldn't be stopped. @@ -472,9 +506,13 @@ openim::util::stop_services_with_name() { openim::log::info "Successfully stopped the $name service." done fi + openim::log::success "All specified services were stopped." } - +# sleep 333333& +# sleep 444444& +# ps -ef | grep "sleep" +# openim::util::stop_services_with_name "sleep 333333" "sleep 444444" # This figures out the host platform without relying on golang. We need this as # we don't want a golang install to be a prerequisite to building yet we need @@ -1150,7 +1188,6 @@ function openim::util::get_server_ip() { echo "$IP" } - function openim::util::onCtrlC () { #Capture CTRL+C, terminate the background process of the program when the script is terminated in the form of ctrl+c kill -9 ${do_sth_pid} ${progress_pid}