feat(main): 🚀 Database Name Correction and S3 Module Int32 Overflow Fix with Go Routine Integration for Automated Checks and Script Optimization (#1799)
In the source code deployment method, the configuration generation process involves executing `make init`, which fundamentally runs the script `./scripts/init-config.sh`. This script utilizes variables defined in the [`environment.sh`](https://github.com/openimsdk/open-im-server/blob/main/scripts/install/environment.sh) script to render the [`openim.yaml`](https://github.com/openimsdk/open-im-server/blob/main/deployments/templates/openim.yaml) template file, subsequently generating the [`config.yaml`](https://github.com/openimsdk/open-im-server/blob/main/config/config.yaml) configuration file.
In the source code deployment method, the configuration generation process involves executing `make init`, which fundamentally runs the script `./scripts/init-config.sh`. This script utilizes variables defined in the [`environment.sh`](https://github.com/openimsdk/open-im-server/blob/main/scripts/install/environment.sh) script to render the [`config.yaml`](https://github.com/openimsdk/open-im-server/blob/main/deployments/templates/config.yaml) template file, subsequently generating the [`config.yaml`](https://github.com/openimsdk/open-im-server/blob/main/config/config.yaml) configuration file.
To optimize and add logic to your instructions, particularly regarding deploying on a Mac, you can modify them as follows:
`Devcontainers` provide a Docker-based isolated development environment.
1. **Fork the OpenIM Repository**: Fork the OpenIM repository on GitHub to your account.
Read [README.md](https://github.com/openimsdk/open-im-server/tree/main/.devcontainer) in the `.devcontainer` directory of the project to learn more about the devcontainer.
2. **Clone Your Fork to Your Local Machine**:
Open your terminal and execute the following commands:
1. Install Docker Desktop for Mac from [Docker Hub](https://docs.docker.com/desktop/install/mac-install/).
2. Install Visual Studio Code and the Remote - Containers extension.
3. Open the cloned OpenIM repository in VS Code.
4. VS Code will prompt to reopen the project in a container. Accept this to set up the environment automatically.
# Set Docker bridging network mode for Mac
export DOCKER_BRIDGE_SUBNET=127.0.0.0/16
### Installing Go and Dependencies
# Set OpenIM IP
export OPENIM_IP=<your-ip>
Use Homebrew to install Go:
# Initialize configuration
make init
```sh
brew install go
```
# Start components using Docker
docker compose up -d
Ensure the version of Go is compatible with the version required by OpenIM (refer to the main documentation for version requirements).
# Start OpenIM Server
make start
```
### Additional Tools
3. **Additional Steps for Mac Deployment**:
If you are deploying on a Mac and wish to use the chat feature, you need to modify the `docker-compose` file. Specifically, you'll need to uncomment the sections related to `openim-chat` and `mysql`.
Install other required tools like Docker, Vagrant, and necessary GNU utils as described in the main documentation.
Here's how to do it:
## Mac Deployment openim-chat and openim-server
- Open the `docker-compose.yml` file in a text editor.
- Find the sections for `openim-chat` and `mysql`.
- Remove the comment marks (`#`) at the beginning of the lines in these sections to enable them.
- Save the file after making these changes.
To integrate the Chinese document into an English document for Linux deployment, we will first translate the content and then adapt it to suit the Linux environment. Here's how the translated and adapted content might look:
4. **Update and Restart Services**:
After modifying the `docker-compose` file, you need to update and restart the services to apply these changes. Run the following command in your terminal:
### Ensure a Clean Environment
```sh
# Update and restart services
docker compose up -d
- It's recommended to execute in a new directory.
- Run `ps -ef | grep openim` to ensure no OpenIM processes are running.
- Run `ps -ef | grep chat` to check for absence of chat-related processes.
- Execute `docker ps` to verify there are no related containers running.
# Check openim-chat start
docker compose logs openim-chat
```
### Source Code Deployment
This command will re-read the `docker-compose.yml` file, apply the new configuration, and restart the necessary containers.
#### Deploying openim-server
Remember, replacing `<your-username>` and `<your-ip>` with your actual GitHub username and desired IP address for OpenIM is crucial. These steps should streamline the setup process, particularly for Mac users wishing to use the chat feature.
Source code deployment is slightly more complex because Docker's networking on Linux differs from Mac.
`Devcontainers` provide a Docker-based isolated development environment.
export OPENIM_IP="Your IP" # If it's a cloud server, setting might not be needed
make init # Generates configuration files
```
Read [README.md](https://github.com/openimsdk/open-im-server/tree/main/.devcontainer) in the `.devcontainer` directory of the project to learn more about the devcontainer.
Before deploying openim-server, modify the Kafka logic in the docker-compose.yml file. Replace:
Ensure the version of Go is compatible with the version required by OpenIM (refer to the main documentation for version requirements).
Before starting the openim-server source, set `config/config.yaml` by replacing all instances of `172.28.0.1` with `127.0.0.1`:
### Additional Tools
```bash
vim config/config.yaml -c "%s/172\.28\.0\.1/127.0.0.1/g" -c "wq"
```
Install other required tools like Docker, Vagrant, and necessary GNU utils as described in the main documentation.
Then start openim-server:
```bash
make start
```
To check the startup:
```bash
make check
```
<aside>
🚧 To avoid mishaps, it's best to wait five minutes before running `make check` again.
</aside>
#### Deploying openim-chat
There are several ways to deploy openim-chat, either by source code or using Docker.
Navigate back to the parent directory:
```bash
cd ..
```
First, let's look at deploying chat from source:
```bash
git clone https://github.com/openimsdk/chat
cd chat
make init # Generates configuration files
```
If openim-chat has not deployed MySQL, you will need to deploy it. Note that the official Docker Hub for MySQL does not support architectures like ARM, so you can use the newer version of the open-source edition:
print_with_delay "Open-IM-Server is not just a product; it's a revolution. It's about bringing the power of seamless," 0.01
print_with_delay "real-time messaging to your fingertips. And it's about joining a global community of developers, dedicated to pushing the boundaries of what's possible." 0.01
print_progress 50 0.02
# Reset text color back to normal
echo -e "\033[0m"
# Set text color to yellow for the Slack link
echo -e "\033[1;33m"
print_with_delay "Join our developer community on Slack: https://join.slack.com/t/openimsdk/shared_invite/zt-22720d66b-o_FvKxMTGXtcnnnHiMqe9Q" 0.01
# Reset text color back to normal
echo -e "\033[0m"
p:::::p
p:::::p
p:::::::p
p:::::::p
p:::::::p
ppppppppp
' 0.0001
# Display product URL
print_with_delay "Discover more and contribute at: https://github.com/openimsdk/open-im-server" 0.01
print_with_delay "Open-IM-Server is not just a product; it's a revolution. It's about bringing the power of seamless," 0.01
print_with_delay "real-time messaging to your fingertips. And it's about joining a global community of developers, dedicated to pushing the boundaries of what's possible." 0.01
print_progress 50 0.02
# Reset text color back to normal
echo -e "\033[0m"
# Set text color to yellow for the Slack link
echo -e "\033[1;33m"
print_with_delay "Join our developer community on Slack: https://join.slack.com/t/openimsdk/shared_invite/zt-22720d66b-o_FvKxMTGXtcnnnHiMqe9Q" 0.01
# Download link (Windows): https://github.com/OpenIMSDK/Open-IM-Protoc/releases/download/v1.0.0/windows.zip
# Download link (Linux): https://github.com/OpenIMSDK/Open-IM-Protoc/releases/download/v1.0.0/linux.zip
#
#
# Installation steps (taking Windows as an example):
# 1. Visit the above link and download the version suitable for Windows.
# 2. Extract the downloaded file.
# 3. Add the extracted tool to your PATH environment variable so that it can be run directly from the command line.
#
#
# Note: The specific installation and usage instructions may vary based on the tool's actual implementation. It's advised to refer to official documentation.
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
# OpenIM CronTask Control Script
#
#
# Description:
# This script provides a control interface for the OpenIM CronTask service within a Linux environment. It supports two installation methods: installation via function calls to systemctl, and direct installation through background processes.
#
#
# Features:
# 1. Robust error handling leveraging Bash built-ins such as 'errexit', 'nounset', and 'pipefail'.
# 2. Capability to source common utility functions and configurations, ensuring environmental consistency.
@ -30,13 +30,13 @@
# 1. Direct Script Execution:
# This will start the OpenIM CronTask directly through a background process.
# 2. Controlling through Functions for systemctl operations:
# Specific operations like installation, uninstallation, and status check can be executed by passing the respective function name as an argument to the script.
# This script provides a control interface for the OpenIM Push service within a Linux environment. It supports two installation methods: installation via function calls to systemctl, and direct installation through background processes.
#
#
# Features:
# 1. Robust error handling leveraging Bash built-ins such as 'errexit', 'nounset', and 'pipefail'.
# 2. Capability to source common utility functions and configurations, ensuring environmental consistency.
@ -29,7 +29,7 @@
# 1. Direct Script Execution:
# This will start the OpenIM push directly through a background process.
# Example: ./openim-push.sh
#
#
# 2. Controlling through Functions for systemctl operations:
# Specific operations like installation, uninstallation, and status check can be executed by passing the respective function name as an argument to the script.
# Example: ./openim-push.sh openim::push::install
@ -39,7 +39,7 @@
# export OPENIM_PUSH_PORT="9090 9091 9092"
#
# Note: Ensure that the appropriate permissions and environmental variables are set prior to script execution.
# This script provides a control interface for the OpenIM RPC service within a Linux environment. It offers functionalities to start multiple RPC services, each denoted by their respective names under openim::rpc::service_name.
#
#
# Features:
# 1. Robust error handling using Bash built-ins like 'errexit', 'nounset', and 'pipefail'.
# 2. The capability to source common utility functions and configurations to ensure uniform environmental settings.
@ -125,7 +125,7 @@ function openim::rpc::start() {
# this lint prefers command -v to which, they are not the same
2230
# Error SC2155 indicates that you should separate variable declaration and assignment to avoid masking the return value of the command.
# In Bash scripts, when you declare and assign a local variable at the same time a command is executed, you only get the output of the command, but not the exit status (return value) of the command. #
2155
# ShellCheck issues SC2086 warnings when you refer to a variable in a script but don't put it in double quotes.This can lead to unexpected behavior when scripts encounter Spaces,
# newlines, and wildcards in file names or other data.
2086
2206
# TODO: 需要修复，然后开启
2034
2048
2148
2059
2214
2145
2128
2550
2046
2181
1102
2045
2068
2145
2207
2231
2013
2154
2120
1083
2001
2012
2016
2164
2223
2166
2119
2162
2295
2002
2004
2202
2178
2064
2260
2043
2178
2044
2153
)
# comma separate for passing to shellcheck
join_by(){
@ -65,14 +113,18 @@ cd "${OPENIM_ROOT}"
# forked should be linted and fixed.
all_shell_scripts=()
whileIFS=$'\n'read -r script;
do git check-ignore -q "$script"||all_shell_scripts+=("$script");
done < <(find . -name "*.sh"\
do git check-ignore -q "$script"||all_shell_scripts+=("$script");
done < <(find . -name "*.sh"\
-not \(\
-path ./_\* -o \
-path ./.git\* -o \
-path ./vendor\* -o \
\( -path ./third_party\* -a -not -path ./third_party/forked\*\)\
\))
-path ./_\* -o \
-path ./.git\* -o \
-path ./Godeps\* -o \
-path ./_output\* -o \
-path ./components\* -o \
-path ./logs\* -o \
-path ./vendor\* -o \
\( -path ./third_party\* -a -not -path ./third_party/forked\*\)\
\) -print 2>/dev/null)
# detect if the host machine has the required shellcheck version installed
# OpenIM Typecheck: Cross-Platform Source Code Type Checking for Go
OpenIM Typecheck 为所有 Go 构建平台进行跨平台源代码类型检查。
## Introduction
## 优点
OpenIM Typecheck is a robust tool designed for cross-platform source code type checking across all Go build platforms. This utility leverages Go’s built-in parsing and type-check libraries (`go/parser` and `go/types`) to deliver efficient and reliable code analysis.
- **Speed**: A complete compilation with OpenIM can take approximately 3 minutes. In contrast, OpenIM Typecheck achieves this in mere seconds, significantly enhancing productivity.
- **Resource Efficiency**: Unlike the typical requirement of over 40GB of RAM for standard processes, Typecheck operates effectively with less than 8GB of RAM. This reduction in resource consumption makes it highly suitable for a variety of systems, reducing overheads and facilitating smoother operations.
OpenIM Typecheck 使用 Go 内置的解析和类型检查库 (`go/parser` 和 `go/types`)。然而，这些库并不是 go 编译器所使用的。偶尔会出现不匹配的情况，但总的来说，它们是相当接近的。
## Implementation
## 错误处理
OpenIM Typecheck employs Go's native parsing and type-checking libraries (`go/parser` and `go/types`). However, it's important to note that these libraries aren't identical to those used by the Go compiler. While occasional mismatches may occur, these libraries generally provide close approximations to the compiler's functionality, offering a reliable basis for type checking.
如果错误不会阻止构建，可以忽略。
## Error Handling
**`go/types` 报告的错误，但 `go build` 不会**：
- **真正的错误**（根据规范）：
- 应尽量修复。如果无法修复或正在进行中（例如，已被外部引用的代码），则可以忽略。
- 例如：闭包中的未使用变量
- **不真实的错误**：
- 应忽略并在适当的情况下向上游报告。
- 例如：staging 和 generated 类型之间的类型检查不匹配
Typecheck's approach to error handling is pragmatic, focusing on practicality and build continuity.
**`go build` 报告的错误，但我们不会**：
- CGo 错误，包括语法和链接器错误。
**Errors reported by `go/types` but not by `go build`**:
- **Actual Errors** (as per the specification):
- These should ideally be rectified. If rectification is not feasible, such as in cases of ongoing work or external dependencies in the code, these errors can be overlooked.
- Example: Unused variables within a closure.
- **False Positives**:
- These errors should be ignored and, where appropriate, reported upstream for resolution.
- Example: Type mismatches between staging and generated types.
**Errors reported by `go build` but not by us**:
- CGo-related errors, including both syntax and linker issues, are outside our scope.
## Usage
### Locally
To run Typecheck locally, simply use the following command:
```bash
make verify
```
### Continuous Integration (CI)
In CI environments, Typecheck can be integrated into the workflow as follows:
```yaml
- name: Typecheck
run: make verify
```
This streamlined process facilitates efficient error detection and resolution, ensuring a robust and reliable build pipeline.
More to learn about typecheck [share blog](https://nsddd.top/posts/concurrent-type-checking-and-cross-platform-development-in-go/)
Xinwei Xiong
10 months ago
committed by
GitHub