Operations Manual
# Preface
This document is the operations manual for Kingdee Apusic Distributed Message Queue for RabbitMQ (Apusic Distributed Message Queue, abbreviated as: ADMQ for RabbitMQ). It aims to provide middleware operations personnel with standardized operation guidelines for ADMQ for RabbitMQ, covering core scenarios such as installation and deployment, configuration optimization, daily operations, troubleshooting, and security hardening, ensuring stable, efficient, and secure operation of ADMQ for RabbitMQ services.
# Target Audience
This document is applicable to IT information business leaders, R&D managers, software project managers, software architects, and operations engineers.
# Related Documents
For more information about ADMQ for RabbitMQ products, please refer to the following ADMQ for RabbitMQ product documentation set:
| No. | Document | Description |
|---|---|---|
| 1 | Kingdee Apusic Distributed Message Queue for RabbitMQ Quick Start Guide | Briefly introduces how to get started with ADMQ for RabbitMQ. |
| 2 | Kingdee Apusic Distributed Message Queue for RabbitMQ Installation Guide | Details how to install ADMQ for RabbitMQ on various operating systems, as well as service start/stop operations. |
| 3 | Kingdee Apusic Distributed Message Queue for RabbitMQ Message Engine User Guide | Details the usage, configuration, management, and supporting tools of the ADMQ for RabbitMQ message engine. |
| 4 | Kingdee Apusic Distributed Message Queue for RabbitMQ Management Console User Guide | Details the usage and operation instructions of the ADMQ for RabbitMQ management console. |
| 5 | Kingdee Apusic Distributed Message Queue for RabbitMQ Developer Guide | Details the development of ADMQ for RabbitMQ client applications using various programming languages. |
| 6 | Kingdee Apusic Distributed Message Queue for RabbitMQ Migration Guide | Details the migration from RabbitMQ to ADMQ for RabbitMQ. |
| 7 | Kingdee Apusic Distributed Message Queue for RabbitMQ Operations Manual | Details monitoring, operations, security hardening, and other operational aspects of ADMQ for RabbitMQ. |
| 8 | Kingdee Apusic Distributed Message Queue for RabbitMQ Performance Optimization Guide | Details performance tuning for ADMQ for RabbitMQ. |
# Technical Support
ADMQ for RabbitMQ provides comprehensive technical support services. You can obtain technical support through the following channels:
- Website: www.apusic.com
- Phone: 400-855-5800
- Email: support@apusic.com
- Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN
When contacting technical support, please provide the following information:
- Your name
- Company information and contact details
- Operating system and its version
- Product version number
- Detailed information such as error logs and screenshots
# Daily Operations
# Service Management
# Start Service
# Start RabbitMQ service
bin/admq-daemon start rabbitmq server
# Start application (bin/admq rabbitmq admin ctl)
bin/admq rabbitmq admin ctl start_app
1
2
3
4
5
2
3
4
5
# Stop Service
# Stop application
bin/admq rabbitmq admin ctl stop_app
# Stop RabbitMQ service
bin/admq-daemon stop rabbitmq server
1
2
3
4
5
2
3
4
5
# View Service Status
# View service status
bin/admq rabbitmq admin status
# View node status
bin/admq rabbitmq admin ctl status
# View cluster status
bin/admq rabbitmq admin ctl cluster_status
1
2
3
4
5
6
7
8
2
3
4
5
6
7
8
# Restart Service
# Restart service
bin/admq-daemon restart rabbitmq server
1
2
2
# Log Management
# Log Locations
| Log Type | Path | Description |
|---|---|---|
| Service Log | ./log/rabbitmq/rabbit@hostname.log | RabbitMQ main log |
| Startup Log | ./log/rabbitmq/rabbit@hostname-sasl.log | SASL log |
| Management Console Log | ./logs/ | Management console runtime log |
# Monitoring Checks
# Daily Check List
| Check Item | Check Command | Normal Standard |
|---|---|---|
| Node Status | bin/admq rabbitmq admin ctl status | Node running normally |
| Cluster Status | bin/admq rabbitmq admin ctl cluster_status | All nodes online |
| Memory Usage | bin/admq rabbitmq admin ctl status | Below vm_memory_high_watermark |
| Disk Space | df -h | Above disk_free_limit |
| Connection Count | bin/admq rabbitmq admin ctl list_connections | Within normal range |
| Queue Backlog | bin/admq rabbitmq admin ctl list_queues messages | No abnormal backlog |
| Consumer Count | bin/admq rabbitmq admin ctl list_queues consumers | Consumers online normally |
# Health Check Script
#!/bin/bash
# Node health check
bin/admq rabbitmq admin diagnostics health_check
# Memory check
bin/admq rabbitmq admin diagnostics memory_breakdown
# Alert queue check
bin/admq rabbitmq admin ctl list_queues name messages consumers | awk '$2 > 10000 {print "WARNING: Queue "$1" has "$2" messages"}'
# Connection count check
CONN_COUNT=$(bin/admq rabbitmq admin ctl list_connections | wc -l)
if [ $CONN_COUNT -gt 1000 ]; then
echo "WARNING: Connection count $CONN_COUNT exceeds threshold"
fi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Configuration Management
# Configuration Files
# rabbitmq.conf
Main configuration file, located at ./config/rabbitmq.conf.
# Network configuration
listeners.tcp.default = 5672
listeners.ssl.default = 5671
# Memory configuration
vm_memory_high_watermark.relative = 0.4
vm_memory_high_watermark_paging_ratio = 0.5
# Disk configuration
disk_free_limit.relative = 2.0
# Heartbeat configuration
heartbeat = 60
# Channel configuration
channel_max = 2047
# Queue configuration
queue_master_locator = min-masters
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Dynamic Configuration
# View runtime parameters
bin/admq rabbitmq admin ctl list_parameters
# Set runtime parameters
bin/admq rabbitmq admin ctl set_parameter federation-upstream my-upstream '{"uri":"amqp://remote-host"}'
# Clear runtime parameters
bin/admq rabbitmq admin ctl clear_parameter federation-upstream my-upstream
# View policies
bin/admq rabbitmq admin ctl list_policies
# Set policy
bin/admq rabbitmq admin ctl set_policy ha-all "^" '{"ha-mode":"all"}' --priority 0 --apply-to queues
# Clear policy
bin/admq rabbitmq admin ctl clear_policy ha-all
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Troubleshooting
# Common Issues and Solutions
# Service Failed to Start
| Symptom | Possible Cause | Solution |
|---|---|---|
| Port occupied | Port 5672/15672 occupied by another process | Find and stop the occupying process, or change listener port |
| Erlang Cookie mismatch | Cluster node Cookie mismatch | Synchronize .erlang.cookie file |
| Insufficient memory | System memory insufficient | Increase memory or lower vm_memory_high_watermark |
| Insufficient disk space | Disk space below disk_free_limit | Clean up disk space or adjust disk_free_limit |
| Configuration error | rabbitmq.conf format error | Check configuration file syntax |
# Message Consumption Anomalies
| Symptom | Possible Cause | Solution |
|---|---|---|
| Message backlog | Consumer processing too slow or consumer offline | Increase consumer count or optimize consumption logic |
| Message loss | Persistence not enabled or manual acknowledgment not used | Enable message persistence and manual acknowledgment |
| Duplicate consumption | Redelivery after consumer acknowledgment timeout | Optimize consumer processing logic to ensure timely acknowledgment |
| Consumption delay | Network latency or consumer performance issues | Check network conditions, optimize consumer code |
# Cluster Anomalies
| Symptom | Possible Cause | Solution |
|---|---|---|
| Node offline | Network failure or service anomaly | Check network connectivity, restart service |
| Split-brain | Network partition causing cluster split | Use bin/admq rabbitmq admin ctl forget_cluster_node to clean up |
| Mirrored queue out of sync | Network latency or node failure | Check network conditions, wait for automatic synchronization |
| Queue unavailable | Queue node offline | Use mirrored queues or Quorum Queue |
# Troubleshooting Commands
# View node details
bin/admq rabbitmq admin ctl status
# View cluster status
bin/admq rabbitmq admin ctl cluster_status
# View connection details
bin/admq rabbitmq admin ctl list_connections peer_host peer_port state channels
# View channel details
bin/admq rabbitmq admin ctl list_channels connection pid consumer_count messages_unacknowledged
# View queue details
bin/admq rabbitmq admin ctl list_queues name pid slave_pids synchronised_slave_pids messages consumers memory
# View consumer details
bin/admq rabbitmq admin ctl list_consumers queue_name channel_pid ack_required prefetch_count
# View memory usage details
bin/admq rabbitmq admin diagnostics memory_breakdown
# View process information
bin/admq rabbitmq admin diagnostics processes
# View network connections
bin/admq rabbitmq admin diagnostics connections
# Collect diagnostic information
bin/admq rabbitmq admin diagnostics gather_auth_context
bin/admq rabbitmq admin diagnostics quorum_status "queue_name"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Backup and Recovery
# Configuration Backup
# Export configuration definitions
bin/admq rabbitmq admin ctl export_definitions /backup/definitions.json
# Import configuration definitions
bin/admq rabbitmq admin ctl import_definitions /backup/definitions.json
1
2
3
4
5
2
3
4
5
# Data Backup
# Mnesia Database Backup
# Backup Mnesia database
bin/admq rabbitmq admin ctl eval 'rabbit_mnesia:backup("/backup/mnesia.backup").'
# Restore Mnesia database
bin/admq rabbitmq admin ctl eval 'rabbit_mnesia:restore("/backup/mnesia.backup").'
1
2
3
4
5
2
3
4
5
# Security Hardening
# Network Security
# Firewall Configuration
# Open necessary ports
firewall-cmd --permanent --add-port=5672/tcp # AMQP
firewall-cmd --permanent --add-port=5671/tcp # AMQP over SSL
firewall-cmd --permanent --add-port=15672/tcp # Management HTTP
firewall-cmd --permanent --add-port=15671/tcp # Management HTTPS
firewall-cmd --permanent --add-port=25672/tcp # Erlang Distribution
firewall-cmd --reload
1
2
3
4
5
6
7
2
3
4
5
6
7
# SSL/TLS Configuration
# rabbitmq.conf
listeners.ssl.default = 5671
ssl_options.cacertfile = /path/to/ca_certificate.pem
ssl_options.certfile = /path/to/server_certificate.pem
ssl_options.keyfile = /path/to/server_key.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.versions.1 = tlsv1.2
ssl_options.versions.2 = tlsv1.3
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# Access Control
# Modify Default User
# Change guest user password
bin/admq rabbitmq admin ctl change_password guest newpassword
# Delete default guest user (recommended)
bin/admq rabbitmq admin ctl delete_user guest
# Create new admin user
bin/admq rabbitmq admin ctl add_user admin strongpassword
bin/admq rabbitmq admin ctl set_user_tags admin administrator
bin/admq rabbitmq admin ctl set_permissions -p / admin ".*" ".*" ".*"
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# Principle of Least Privilege
# Assign minimum permissions to business users
bin/admq rabbitmq admin ctl add_user app_user apppassword
bin/admq rabbitmq admin ctl set_user_tags app_user none
bin/admq rabbitmq admin ctl set_permissions -p /app_vhost app_user "^app-.*" "^app-.*" "^app-.*"
1
2
3
4
2
3
4
# Performance Monitoring
# Monitoring Metrics
| Metric | Description |
|---|---|
| rabbitmq_connections | Number of connections |
| rabbitmq_channels | Number of channels |
| rabbitmq_queues | Number of queues |
| rabbitmq_queue_messages_ready | Number of ready messages |
| rabbitmq_queue_messages_unacked | Number of unacknowledged messages |
| rabbitmq_queue_messages | Total number of messages |
| rabbitmq_node_mem_used | Memory usage |
| rabbitmq_node_disk_free | Free disk space |
| rabbitmq_node_cpu_user | CPU user utilization |
# Operations Best Practices
- Regular Checks: Establish daily inspection mechanisms to identify potential issues early
- Monitoring & Alerting: Configure comprehensive monitoring and alerting systems
- Backup Strategy: Develop regular backup strategies to ensure data security
- Documentation: Record all operations and changes
- Capacity Planning: Regularly evaluate cluster capacity, scale up in advance
- Security Updates: Apply security patches promptly
- Disaster Recovery Drills: Conduct regular fault recovery drills
编辑页面 (opens new window)