Operations Guide

# Preface

This document is the operations guide for Apusic In-Memory Data Cache (AMDC) V2.0.6, aiming to provide middleware operations personnel with a standardized operations guide for AMDC V2.0.6, covering core scenarios such as installation and deployment, configuration optimization, daily operations, troubleshooting, and security hardening, to ensure stable, efficient, and secure operation of AMDC product services.

# Target Audience

This document is intended for AMDC product operations engineers, IT system operations engineers, development engineers, software architects, R&D managers, and other personnel.

For more information about AMDC V2.0.6, please refer to the following AMDC V2.0.6 product manual documentation set:

No.	Document	Description
1	Apusic In-Memory Data Cache V2.0.6 Quick Start Guide	A brief introduction on how to quickly get started with AMDC.
2	Apusic In-Memory Data Cache V2.0.6 Installation Guide	Detailed introduction on how to install AMDC on various operating systems, AMDC service start/stop operations, and the product registration process.
3	Apusic In-Memory Data Cache V2.0.6 Cache Core User Manual	Detailed introduction on how to use, configure, and manage AMDC features and supporting tools.
4	Apusic In-Memory Data Cache V2.0.6 Console User Manual	Detailed instructions on using and operating AMDC console features.
5	Apusic In-Memory Data Cache V2.0.6 Development Guide	Detailed instructions for developing AMDC client applications using various programming languages.
6	Apusic In-Memory Data Cache V2.0.6 Migration Guide	Detailed instructions for migrating AMDC from historical versions to V2.0.6, as well as migrating from Redis to AMDC.
7	Apusic In-Memory Data Cache V2.0.6 Operations Guide	Detailed instructions for AMDC monitoring, operations, security hardening, and other operational procedures.
8	Apusic In-Memory Data Cache V2.0.6 Performance Tuning Guide	Detailed instructions for AMDC performance tuning.

# Technical Support

AMDC products provide comprehensive technical support services. You can obtain technical support through the following channels:

Website: https://www.apusic.com
Phone: 400-855-5800
Email: support@apusic.com
Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN

When requesting technical support, please provide the following information:

Your name
Company information and contact details
Operating system and version
Product version number
Detailed information including logs, screenshots, etc. of any exceptions or errors

# Daily Operations

# Service Start/Stop Management

# Normal Shutdown

# Command line method
./amdc-cli -h [IP] -p 6359 -a [password] shutdown

# Interactive client
127.0.0.1:6359> shutdown

1
2
3
4
5

# Forced Shutdown

ps -ef | grep amdc        # Find the PID
kill -9 [PID]             # Force termination (may cause data loss)

1
2

# License Management

Hot update: Directly replace license.lic, license.xml, or acls.properties; the system loads them automatically.
Priority order:
1. acls.properties (unified authorization center)
2. license.lic (KBC authorization)
3. license.xml (local authorization)

# Health Inspection Checklist

Perform the following checks regularly; it is recommended to incorporate them into automation scripts.

# Basic Connectivity Check

Mode	Command	Expected Result
Standalone	`./amdc-cli -h [IP] -p 6359 -a [password] info`	Contains `amdc_version:2.0.6`
Sentinel	`./amdc-cli -h [sentinel_ip] -p 26359 -a [password] info sentinel`	`sentinel_masters:1`, master-slave status normal
Cluster	`./amdc-cli -h [IP] -p 6359 -a [password] cluster info`	`cluster_state:ok`, slots fully assigned

# Master-Slave Replication Deployment

./amdc-cli -h [master_ip] -p 6359 -a [password] info replication

Key metrics:
- role:master
- connected_slaves = expected number
- All slaveX.state=online
- lag ≤ 5s (alert triggered if >30s)

# Memory Usage

./amdc-cli -h [IP] -p 6359 -a [password] info memory

Watch:
- used_memory_human
- mem_fragmentation_ratio (ideal value: 1.0–1.5)

# Client Connections

./amdc-cli -h [IP] -p 6359 -a [password] info clients

Watch:
- connected_clients (alert if >80% of maxclients)
- blocked_clients ≈ 0

# Persistence Status

./amdc-cli -h [IP] -p 6359 -a [password] info persistence

RDB: rdb_last_bgsave_status=ok
AOF (if enabled): aof_enabled=1 and aof_last_bgrewrite_status=ok

# Performance Monitoring and Optimization

# Benchmark Testing

./amdc-benchmark -h [IP] -p 6359 -a [password] -c 50 -n 10000 --csv

# Slow Query Analysis

./amdc-cli -h [IP] -p 6359 -a [password] slowlog get 10

Recommended configuration:

slowlog-log-slower-than 10000  # 10ms
slowlog-max-len 128

1
2

# Big Key and Hot Key Detection

./amdc-cli -h [IP] -p 6359 -a [password] --bigkeys   # Big values
./amdc-cli -h [IP] -p 6359 -a [password] --memkeys   # Big-memory keys
./amdc-cli -h [IP] -p 6359 -a [password] --hotkeys   # Hot keys (requires LFU policy)

1
2
3

# Real-Time Monitoring

./amdc-cli --stat          # Dynamic metrics
./amdc-cli monitor > log   # Command auditing (use with caution; high performance overhead)

1
2

# Monitoring Platform Integration (Recommended)

Prometheus + Grafana: Collect metrics via redis_exporter
ELK: Elasticsearch, Logstash, Kibana (ELK), centralized log analysis

# prometheus.yml example
scrape_configs:
  - job_name: 'amdc'
    static_configs:
      - targets: ['redis-exporter:9121']
    metrics_path: /scrape
    params:
      target: ['redis://amdc-host:6359']

1
2
3
4
5
6
7
8

# Troubleshooting and Emergency Handling

# Common Problem Diagnosis

Problem	Symptom	Troubleshooting Steps
Master-slave synchronization delay	The `lag` value of the slave node keeps growing	1. Check network connectivity and bandwidth between master and slave nodes 2. Check whether `repl_backlog_size` on the master node is too small 3. Check whether disk I/O performance of the slave node is a bottleneck 4. Check the AMDC logs for replication-related errors (e.g., "MASTER timeout")
Abnormal cluster status	`cluster_state:fail`	1. Check network connectivity between all nodes 2. Confirm that all master nodes are online 3. Check whether slot assignment is complete (`cluster_slots_ok == 16384`) 4. Check `cluster info` and cluster message statistics in the logs
Insufficient memory	Slow response, large numbers of keys evicted	1. Check whether the `maxmemory` configuration is reasonable 2. Analyze memory usage patterns (`info memory`) 3. Confirm whether the eviction policy (e.g., `allkeys-lru`) is appropriate 4. Use `--bigkeys` or `--memkeys` to identify big keys

# Key Error Log Keywords

"No valid license file found" → License missing
"MASTER timeout" → Master-slave disconnection
"Node timeout" → Cluster node lost

# Emergency Recovery Procedures

# Service Unavailable

Confirm the scope of impact (single node or cluster)
Check the process: ps -ef | grep amdc
Check resources: CPU/memory/disk
Try a graceful restart → force restart if necessary

# Data File Corruption

RDB repair: ./amdc-check-rdb file.rdb
AOF repair: ./aof-check-aof --fix file.aof
Backup recovery:
1. Stop the service
2. Replace the files
3. Start and verify the data

# Automated Health Checks

# Health Check Script

You can write a shell script to check the health status of the AMDC service, as shown in the following sample: amdc_health_check.sh

#!/bin/bash
HOST="127.0.0.1"; PORT="6359"; PWD="your_password"  # Please replace with the actual password

./amdc-cli -h $HOST -p $PORT -a $PWD ping >/dev/null || { echo "ERROR"; exit 1; }

echo "Memory usage: $(./amdc-cli -h $HOST -p $PORT -a $PWD info memory | grep used_memory_human)"
echo "Client connections: $(./amdc-cli -h $HOST -p $PORT -a $PWD info clients | grep connected_clients)"
echo "Slow query count: $(./amdc-cli -h $HOST -p $PORT -a $PWD slowlog len)"

1
2
3
4
5
6
7
8

# Scheduled Tasks

You can use crontab to implement scheduled health checks, log rotation, big Key scans, and other tasks for the AMDC service.

# Hourly health check
0 * * * * /opt/amdc/scripts/amdc_health_check.sh

# Daily early-morning big Key scan
0 2 * * * /opt/amdc/bin/amdc-cli -h [IP] -p 6359 -a [password] --bigkeys > /logs/bigkeys_$(date +\%Y\%m\%d).log

1
2
3
4
5

# Recommended Monitoring Alert Thresholds

Metric	Warning Threshold	Critical Threshold
Memory usage	70%	85%
Client connections	80% of maxclients	90%
Replication lag	5 seconds	30 seconds
Slow query volume	10 entries/hour	50 entries/hour
CPU usage	70%	90%

# Regular Maintenance Plan

Daily: Health checks, log rotation
Weekly: Stress testing, big Key analysis
Monthly: Configuration review, security audit
Quarterly: Disaster recovery drills, capacity planning

# Software Management

# Configure a System Service for Auto-Start on Boot

Create the service file:

sudo vim /etc/systemd/system/amdc.service

Write the configuration (adjust according to actual paths, including client connection IP and password):

[Unit]
Description=AMDC Service
After=network.target

[Service]
Type=simple
User=amdc
Group=amdc
WorkingDirectory=/opt/amdc
ExecStart=/opt/amdc/bin/amdc-server /opt/amdc/amdc.yaml
ExecStop=/opt/amdc/bin/amdc-cli -h 127.0.0.1 -p 6359 -a your_password shutdown  # Please replace with the actual password
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Enable the service:

sudo systemctl daemon-reload
sudo systemctl enable --now amdc.service

1
2

# Build a Docker Image

Prepare the installation package: AMDC-V2.0.6-xxxx-amd64.tar.gz

Write the Dockerfile:

FROM ubuntu:20.04
ADD AMDC-V2.0.6-xxxx-amd64.tar.gz /opt/
WORKDIR /opt/amdc
EXPOSE 6359
CMD ["/opt/amdc/bin/amdc-server", "/opt/amdc/amdc.yaml"]

1
2
3
4
5

Build and run:

docker build -t apusic/amdc:v2.0.6 .
docker run -d \
  --name amdc \
  -p 6359:6359 \
  -v ./license.lic:/opt/amdc/license.lic \
  -v ./amdc.yaml:/opt/amdc/amdc.yaml \
  -v ./data:/opt/amdc/data \
  apusic/amdc:v2.0.6

1
2
3
4
5
6
7
8

Note: In production environments, be sure to mount the license, configuration, and persistence directories as volumes.

# Appendix: Key Configuration Optimization Recommendations

Configuration Item	Recommended Value	Description
`maxmemory`	70% of physical memory	Avoid OOM
`maxmemory-policy`	`allkeys-lru` or `allkeys-lfu`	Reasonable eviction
`timeout`	300	Idle connection timeout (seconds)
`tcp-keepalive`	60	TCP keepalive detection (seconds)
`slowlog-log-slower-than`	10000	Slow query threshold (microseconds)

编辑页面

#Operations Guide

← Migration Guide Performance Tuning Guide→

Operations Guide

# Preface

# Target Audience

# Related Documentation

# Technical Support

# Daily Operations

# Service Start/Stop Management

# Normal Shutdown

# Forced Shutdown

# License Management

# Health Inspection Checklist

# Basic Connectivity Check

# Master-Slave Replication Deployment

# Memory Usage

# Client Connections

# Persistence Status

# Performance Monitoring and Optimization

# Benchmark Testing

# Slow Query Analysis

# Big Key and Hot Key Detection

# Real-Time Monitoring

# Monitoring Platform Integration (Recommended)

# Troubleshooting and Emergency Handling

# Common Problem Diagnosis

# Key Error Log Keywords

# Emergency Recovery Procedures

# Service Unavailable

# Data File Corruption

# Automated Health Checks

# Health Check Script

# Scheduled Tasks

# Recommended Monitoring Alert Thresholds

# Regular Maintenance Plan

# Software Management

# Configure a System Service for Auto-Start on Boot

# Build a Docker Image

# Appendix: Key Configuration Optimization Recommendations