Open-IM-Server/docs/contrib/prometheus-grafana.md

Deployment and Design of OpenIM's Management Backend and Monitoring

• 1. Source Code & Docker
  • 1.1. Deployment
  • 1.2. Configuration
  • 1.3. Monitoring Running in Docker Guide
    • 1.3.1. Introduction
    • 1.3.2. Prerequisites
    • 1.3.3. Step 1: Clone the Repository
    • 1.3.4. Step 2: Start Docker Compose
    • 1.3.5. Step 3: Use the OpenIM Web Interface
    • 1.3.6. Running Effect
    • 1.3.7. Step 4: Access the Admin Panel
    • 1.3.8. Step 5: Access the Monitoring Interface
    • 1.3.9. Next Steps
    • 1.3.10. Troubleshooting
• 2. Kubernetes
  • 2.1. Middleware Monitoring
  • 2.2. Custom OpenIM Metrics
  • 2.3. Node Exporter
• 3. Setting Up and Configuring AlertManager Using Environment Variables and make init
  • 3.1. Introduction
  • 3.2. Prerequisites
  • 3.3. Configuration Steps
    • 3.3.1. Exporting Environment Variables
    • 3.3.2. Initializing AlertManager
    • 3.3.3. Key Configuration Fields
    • 3.3.4. Configuring SMTP Authentication Password
    • 3.3.5. Useful Links for Common Email Servers
  • 3.4. Conclusion

OpenIM offers various flexible deployment options to suit different environments and requirements. Here is a simplified and optimized description of these deployment options:

1. Source Code Deployment:
   • Regular Source Code Deployment: Deployment using the nohup method. This is a basic deployment method suitable for development and testing environments. For details, refer to the Regular Source Code Deployment Guide.
   • Production-Level Deployment: Deployment using the system method, more suitable for production environments. This method provides higher stability and reliability. For details, refer to the Production-Level Deployment Guide.
2. Cluster Deployment:
   • Kubernetes Deployment: Provides two deployment methods, including deployment through Helm and sealos. This is suitable for environments that require high availability and scalability. Specific methods can be found in the Kubernetes Deployment Guide.
3. Docker Deployment:
   • Regular Docker Deployment: Suitable for quick deployments and small projects. For detailed information, refer to the Docker Deployment Guide.
   • Docker Compose Deployment: Provides more convenient service management and configuration, suitable for complex multi-container applications.

Next, we will introduce the specific steps, monitoring, and management backend configuration for each of these deployment methods, as well as usage tips to help you choose the most suitable deployment option according to your needs.

1. Source Code & Docker

1.1. Deployment

OpenIM deploys openim-server and openim-chat from source code, while other components are deployed via Docker.

For Docker deployment, you can deploy all components with a single command using the openimsdk/openim-docker repository. The deployment configuration can be found in the environment.sh document, which provides information on how to learn and familiarize yourself with various environment variables.

For Prometheus, it is not enabled by default. To enable it, set the environment variable before executing make init:

export PROMETHEUS_ENABLE=true   # Default is false

Then, execute:

make init
docker compose up -d

1.2. Configuration

To configure Prometheus data sources in Grafana, follow these steps:

1. Log in to Grafana: First, open your web browser and access the Grafana URL. If you haven't changed the port, the address is typically http://localhost:13000.

2. Log in with default credentials: Grafana's default username and password are both admin. You will be prompted to change the password on your first login.

3. Access Data Sources Settings:

   • In the left menu of Grafana, look for and click the "gear" icon representing "Configuration."
   • In the configuration menu, select "Data Sources."

4. Add a New Data Source:

   • On the Data Sources page, click the "Add data source" button.
   • In the list, find and select "Prometheus."

   

   Click Add New connection to add more data sources, such as Loki (responsible for log storage and query processing).

5. Configure the Prometheus Data Source:

   • On the configuration page, fill in the details of the Prometheus server. This typically includes the URL of the Prometheus service (e.g., if Prometheus is running on the same machine as OpenIM, the URL might be http://172.28.0.1:19090, with the address matching the DOCKER_BRIDGE_GATEWAY variable address). OpenIM and the components are linked via a gateway. The default port used by OpenIM is 19090.
   • Adjust other settings as needed, such as authentication and TLS settings.

   

6. Save and Test:

   • After completing the configuration, click the "Save & Test" button to ensure that Grafana can successfully connect to Prometheus.

Importing Dashboards in Grafana

Importing Grafana Dashboards is a straightforward process and is applicable to OpenIM Server application services and Node Exporter. Here are detailed steps and necessary considerations:

Key Metrics Overview and Deployment Steps

To monitor OpenIM in Grafana, you need to focus on three categories of key metrics, each with its specific deployment and configuration steps:

1. OpenIM Metrics (prometheus-dashboard.yaml):
   • Configuration File Path: Located at config/prometheus-dashboard.yaml.
   • Enabling Monitoring: Set the environment variable export PROMETHEUS_ENABLE=true to enable Prometheus monitoring.
   • More Information: Refer to the OpenIM Configuration Guide.
2. Node Exporter:
   • Container Deployment: Deploy the quay.io/prometheus/node-exporter container for node monitoring.
   • Get Dashboard: Access the Node Exporter Full Feature Dashboard and import it using YAML file download or ID import.
   • Deployment Guide: Refer to the Node Exporter Deployment Documentation.
3. Middleware Metrics: Each middleware requires specific steps and configurations to enable monitoring. Here is a list of common middleware and links to their respective setup guides:
   • MySQL:
     • Configuration: Ensure MySQL has performance monitoring enabled.
     • Link: Refer to the MySQL Monitoring Configuration Guide.
   • Redis:
     • Configuration: Configure Redis to allow monitoring data export.
     • Link: Refer to the Redis Monitoring Guide.
   • MongoDB:
     • Configuration: Set up monitoring metrics for MongoDB.
     • Link: Refer to the MongoDB Monitoring Guide.
   • Kafka:
     • Configuration: Integrate Kafka with Prometheus monitoring.
     • Link: Refer to the Kafka Monitoring Guide.
   • Zookeeper:
     • Configuration: Ensure Zookeeper can be monitored by Prometheus.
     • Link: Refer to the Zookeeper Monitoring Configuration.

Importing Steps:

1. Access the Dashboard Import Interface:

   • Click the + icon on the left menu or in the top right corner of Grafana, then select "Create."
   • Choose "Import" to access the dashboard import interface.

2. Perform Dashboard Import:

   • Upload via File: Directly upload your YAML file.
   • Paste Content: Open the YAML file, copy its content, and paste it into the import interface.
   • Import via Grafana.com Dashboard: Visit Grafana Dashboards, search for the desired dashboard, and import it using its ID.

3. Configure the Dashboard:

   • Select the appropriate data source, such as the previously configured Prometheus.
   • Adjust other settings, such as the dashboard name or folder.

4. Save and View the Dashboard:

   • After configuring, click "Import" to complete the process.
   • Immediately view the new dashboard after successful import.

Graph Examples:

1.3. Monitoring Running in Docker Guide

1.3.1. Introduction

This guide provides the steps to run OpenIM using Docker. OpenIM is an open-source instant messaging solution that can be quickly deployed using Docker. For more information, please refer to the OpenIM Docker GitHub.

1.3.2. Prerequisites

• Ensure that Docker and Docker Compose are installed.
• Basic understanding of Docker and containerization technology.

1.3.3. Step 1: Clone the Repository

First, clone the OpenIM Docker repository:

git clone https://github.com/openimsdk/openim-docker.git

Navigate to the repository directory and check the README file for more information and configuration options.

1.3.4. Step 2: Start Docker Compose

In the repository directory, run the following command to start the service:

docker-compose up -d

This will download the required Docker images and start the OpenIM service.

1.3.5. Step 3: Use the OpenIM Web Interface

• Open a browser in private mode and access OpenIM Web.
• Register two users and try adding friends.
• Test sending messages and pictures.

1.3.6. Running Effect

1.3.7. Step 4: Access the Admin Panel

• Access the OpenIM Admin Panel.
• Log in using the default username and password (admin1:admin1).

Running Effect Image:

1.3.8. Step 5: Access the Monitoring Interface

• Log in to the Monitoring Interface using the credentials (admin:admin).

1.3.9. Next Steps

• Configure and manage the services following the steps provided in the OpenIM source code.
• Refer to the README file for advanced configuration and management.

1.3.10. Troubleshooting

• If you encounter any issues, please check the documentation on OpenIM Docker GitHub or search for related issues in the Issues section.
• If the problem persists, you can create an issue on the openim-docker repository or the openim-server repository.

2. Kubernetes

Refer to openimsdk/helm-charts.

When deploying and monitoring OpenIM in a Kubernetes environment, you will focus on three main metrics: middleware, custom OpenIM metrics, and Node Exporter. Here are detailed steps and guidelines:

2.1. Middleware Monitoring

Middleware monitoring is crucial to ensure the overall system's stability. Typically, this includes monitoring the following components:

• MySQL: Monitor database performance, query latency, and more.
• Redis: Track operation latency, memory usage, and more.
• MongoDB: Observe database operations, resource usage, and more.
• Kafka: Monitor message throughput, latency, and more.
• Zookeeper: Keep an eye on cluster status, performance metrics, and more.

For Kubernetes environments, you can use the corresponding Prometheus Exporters to collect monitoring data for these middleware components.

2.2. Custom OpenIM Metrics

Custom OpenIM metrics provide essential information about the OpenIM application itself, such as user activity, message traffic, system performance, and more. To monitor these metrics in Kubernetes:

• Ensure OpenIM application configurations expose Prometheus metrics.
• When deploying using Helm charts (refer to OpenIM Helm Charts), pay attention to configuring relevant monitoring settings.

2.3. Node Exporter

Node Exporter is used to collect hardware and operating system-level metrics for Kubernetes nodes, such as CPU, memory, disk usage, and more. To integrate Node Exporter in Kubernetes:

• Deploy Node Exporter using the appropriate Helm chart. You can find information and guides on Prometheus Community.
• Ensure Node Exporter's data is collected by Prometheus instances within your cluster.

3. Setting Up and Configuring AlertManager Using Environment Variables and make init

3.1. Introduction

AlertManager, a component of the Prometheus monitoring system, handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver. This document outlines how to set up and configure AlertManager using environment variables and the make init command. We will focus on configuring key fields like the sender's email, SMTP settings, and SMTP authentication password.

3.2. Prerequisites

• Basic knowledge of terminal and command-line operations.
• AlertManager installed on your system.
• Access to an SMTP server for sending emails.

3.3. Configuration Steps

3.3.1. Exporting Environment Variables

Before initializing AlertManager, you need to set environment variables. These variables are used to configure the AlertManager settings without altering the code. Use the export command in your terminal. Here are some key variables you might set:

• export ALERTMANAGER_RESOLVE_TIMEOUT='5m'
• export ALERTMANAGER_SMTP_FROM='alert@example.com'
• export ALERTMANAGER_SMTP_SMARTHOST='smtp.example.com:465'
• export ALERTMANAGER_SMTP_AUTH_USERNAME='alert@example.com'
• export ALERTMANAGER_SMTP_AUTH_PASSWORD='your_password'
• export ALERTMANAGER_SMTP_REQUIRE_TLS='false'

3.3.2. Initializing AlertManager

After setting the necessary environment variables, you can initialize AlertManager by running the make init command. This command typically runs a script that prepares AlertManager with the provided configuration.

3.3.3. Key Configuration Fields

a. Sender's Email (ALERTMANAGER_SMTP_FROM)

This variable sets the email address that will appear as the sender in the notifications sent by AlertManager.

b. SMTP Configuration

• SMTP Server (ALERTMANAGER_SMTP_SMARTHOST): Specifies the address and port of the SMTP server used for sending emails.
• SMTP Authentication Username (ALERTMANAGER_SMTP_AUTH_USERNAME): The username for authenticating with the SMTP server.
• SMTP Authentication Password (ALERTMANAGER_SMTP_AUTH_PASSWORD): The password for SMTP server authentication. It's crucial to keep this value secure.

3.3.4. Configuring SMTP Authentication Password

The SMTP authentication password can be set using the ALERTMANAGER_SMTP_AUTH_PASSWORD environment variable. It's recommended to use a secure method to set this variable to avoid exposing sensitive information. For instance, you might read the password from a secure file or a secret management tool.

3.3.5. Useful Links for Common Email Servers

For specific configurations related to common email servers, you may refer to their respective documentation:

• Gmail SMTP Settings:
  • Gmail SMTP Configuration
• Microsoft Outlook SMTP Settings:
  • Outlook Email Settings
• Yahoo Mail SMTP Settings:
  • Yahoo SMTP Configuration

3.4. Conclusion

Setting up and configuring AlertManager with environment variables provides a flexible and secure way to manage alert settings. By following the above steps, you can easily configure AlertManager for your monitoring needs. Always ensure to secure sensitive information, especially when dealing with SMTP authentication credentials.