Operations Manual

# Preface

This document is the operations manual for Apusic In-Memory Data Cache (AMDC) V2.0.4, designed to provide middleware operations personnel with standardized operational guidance for AMDC v2.0.4, covering core scenarios such as installation and deployment, configuration optimization, daily operations, troubleshooting, and security hardening, ensuring stable, efficient, and secure operation of AMDC product services.

# Target Audience

This document is intended for AMDC product operations engineers, IT system operations engineers, development engineers, software architects, and R&D managers.

For more information about AMDC V2.0.4, please refer to the following AMDC V2.0.4 product manual documentation set:

No.	Document	Description
1	Apusic In-Memory Data Cache V2.0.4 Quick Start Guide	A brief introduction on how to quickly get started with AMDC.
2	Apusic In-Memory Data Cache V2.0.4 Installation Guide	Detailed introduction on how to install AMDC on various operating systems, AMDC service start/stop operations, and product registration process.
3	Apusic In-Memory Data Cache V2.0.4 Core User Manual	Detailed introduction on how to use, configure, and manage AMDC features and supporting tools.
4	Apusic In-Memory Data Cache V2.0.4 Console User Manual	Detailed instructions on using and operating AMDC console features.
5	Apusic In-Memory Data Cache V2.0.4 Development Guide	Detailed instructions for developing AMDC client applications using various programming languages.
6	Apusic In-Memory Data Cache V2.0.4 Migration Guide	Detailed instructions for migrating AMDC from historical versions to V2.0.4, as well as migrating from Redis to AMDC.
7	Apusic In-Memory Data Cache V2.0.4 Operations Guide	Detailed instructions for AMDC monitoring, operations, security hardening, and other operational procedures.
8	Apusic In-Memory Data Cache V2.0.4 Performance Tuning Guide	Detailed instructions for AMDC performance tuning.

# Technical Support

AMDC products provide comprehensive technical support services. You can obtain technical support through the following channels:

Website: www.apusic.com
Phone: 400-855-5800
Email: support@apusic.com
Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN

When requesting technical support, please provide the following information:

Your name
Company information and contact details
Operating system and version
Product version number
Detailed information including logs, screenshots, etc. of any exceptions or errors

# Daily Operations

# Service Start/Stop Management

# Normal Shutdown

# Command line method
./amdc-cli -h [IP] -p [port] -a [password] shutdown

# Interactive client
127.0.0.1:6359> shutdown

1
2
3
4
5

# Forced Shutdown

ps -ef | grep amdc        # Find PID
kill -9 [PID]             # Force terminate (may cause data loss)

1
2

# License Management

Hot Update: Directly replace license.lic, license.xml or acls.properties, system will automatically load.
Priority Order:
1. acls.properties (Unified Authorization Center)
2. license.lic (KBC Authorization)
3. license.xml (Local Authorization)

# Health Check List

Perform the following checks regularly, recommended to be included in automated scripts.

# Basic Connection Check

Mode	Command	Expected Result
Standalone	`./amdc-cli -h [ip] -p 6359 -a [pwd] info`	Contains `amdc_version:2.0.4`
Sentinel	`./amdc-cli -h [sentinel_ip] -p 26359 -a [pwd] info sentinel`	`sentinel_masters:1`, master-slave status normal
Cluster	`./amdc-cli -h [ip] -p 6359 -a [pwd] cluster info`	`cluster_state:ok`, slots fully allocated

# Master-Slave Replication Check

./amdc-cli -h [master_ip] -p 6359 -a [pwd] info replication

Key indicators:
- role:master
- connected_slaves = expected count
- All slaveX.state=online
- lag ≤ 5s (>30s triggers alert)

# Memory Usage

./amdc-cli  -h [IP] -p 6359 -a [password] info memory

Monitor:
- used_memory_human
- mem_fragmentation_ratio (ideal value: 1.0–1.5)

# Client Connections

./amdc-cli  -h [IP] -p 6359 -a [password] info clients

Monitor:
- connected_clients (>80% of maxclients triggers alert)
- blocked_clients ≈ 0

# Persistence Status

./amdc-cli  -h [IP] -p 6359 -a [password] info persistence

RDB: rdb_last_bgsave_status=ok
AOF (if enabled): aof_enabled=1 and aof_last_bgrewrite_status=ok

# Performance Monitoring and Optimization

# Benchmark Testing

./amdc-benchmark -h [IP] -p 6359 -a [pwd] -c 50 -n 10000 --csv

Output displayed as follows:

# Slow Query Analysis

./amdc-cli  -h [IP] -p 6359 -a [password] slowlog get 10

Recommended configuration:

slowlog-log-slower-than 10000  # 10ms
slowlog-max-len 128

1
2

# Big Key and Hot Key Detection

./amdc-cli  -h [IP] -p 6359 -a [password] --bigkeys   # Big Value
./amdc-cli  -h [IP] -p 6359 -a [password] --memkeys   # Large Memory Key
./amdc-cli  -h [IP] -p 6359 -a [password] --hotkeys   # Hot Key (requires LFU policy)

1
2
3

# Real-time Monitoring

./amdc-cli  --stat          # Dynamic metrics
./amdc-cli  monitor > log   # Command audit (use with caution, high performance overhead)

1
2

# Monitoring Platform Integration (Recommended)

Prometheus + Grafana: Collect metrics via redis_exporter

ELK: Centralized log analysis

# prometheus.yml example
scrape_configs:
  - job_name: 'amdc'
    static_configs:
      - targets: ['redis-exporter:9121']
    metrics_path: /scrape
    params:
      target: ['redis://amdc-host:6359']

1
2
3
4
5
6
7
8

# Troubleshooting and Emergency Response

# Common Problem Diagnosis

Problem	Symptom	Troubleshooting Steps
Master-slave sync delay	Slave node `lag` value keeps increasing	1. Check network connectivity and bandwidth between master and slave nodes 2. Check if `repl_backlog_size` on master node is too small 3. Check if slave node disk I/O performance is bottlenecked 4. Check AMDC logs for replication-related errors (e.g. "MASTER timeout")
Cluster status abnormal	`cluster_state:fail`	1. Check if all nodes can communicate with each other 2. Confirm all master nodes are online 3. Check if slot allocation is complete (`cluster_slots_ok == 16384`) 4. Check `cluster info` and cluster message statistics in logs
Insufficient memory	Slow response, large number of keys evicted	1. Check if `maxmemory` configuration is reasonable 2. Analyze memory usage pattern (`info memory`) 3. Confirm if eviction policy (e.g. `allkeys-lru`) is appropriate 4. Use `--bigkeys` or `--memkeys` to identify big keys

# Key Error Log Keywords

"No valid license file found" → Missing license
"MASTER timeout" → Master-slave disconnection
"Node timeout" → Cluster node lost contact

# Emergency Recovery Procedures

# Service Unavailable

Confirm scope of impact (single point or cluster)
Check process: ps -ef | grep amdc
Check resources: CPU/memory/disk
Attempt graceful restart → Force restart if necessary

# Data File Corruption

RDB Repair: ./amdc-check-rdb file.rdb
AOF Repair: ./aof-check-aof --fix file.aof
Backup Recovery:
1. Stop service
2. Replace files
3. Start and verify data

# Automated Health Check

# Health Check Script

You can write a shell script to check the health status of AMDC service, as shown in the following example: amdc_health_check.sh

#!/bin/bash
HOST="127.0.0.1"; PORT="6359"; PWD="your_password"

./amdc-cli -h $HOST -p $PORT -a $PWD ping >/dev/null || { echo "ERROR"; exit 1; }

echo "Memory Usage: $(./amdc-cli -h $HOST -p $PORT -a $PWD info memory | grep used_memory_human)"
echo "Client Connections: $(./amdc-cli -h $HOST -p $PORT -a $PWD info clients | grep connected_clients)"
echo "Slow Query Count: $(./amdc-cli -h $HOST -p $PORT -a $PWD slowlog len)"

1
2
3
4
5
6
7
8

# Scheduled Tasks

You can use crontab to implement scheduled tasks for AMDC service health checks, log rotation, big key scanning, etc.

# Hourly health check
0 * * * * /opt/amdc/scripts/amdc_health_check.sh

# Daily big key scan at 2 AM
0 2 * * * /opt/amdc/bin/amdc-cli  -h [IP] -p 6359 -a [password] --bigkeys > /logs/bigkeys_$(date +\%Y\%m\%d).log

1
2
3
4
5

# Monitoring Alert Threshold Recommendations

Indicator	Warning Threshold	Critical Threshold
Memory Usage	70%	85%
Client Connections	80% of maxclients	90%
Replication Lag	5 seconds	30 seconds
Slow Query Count	10 per hour	50 per hour
CPU Usage	70%	90%

# Regular Maintenance Plan

Daily: Health check, log rotation
Weekly: Stress test, big key analysis
Monthly: Configuration review, security audit
Quarterly: Disaster recovery drill, capacity planning

# Software Management

# Configure System Service for Auto-start on Boot

Create service file:

sudo vim /etc/systemd/system/amdc.service

Write configuration (adjust according to actual paths, including client connection IP, password):

[Unit]
Description=AMDC Service
After=network.target

[Service]
Type=simple
User=amdc
Group=amdc
WorkingDirectory=/opt/amdc
ExecStart=/opt/amdc/bin/amdc-server /opt/amdc/amdc.yaml
ExecStop=/opt/amdc/bin/amdc-cli -h 127.0.0.1 -p 6359 -a your_password shutdown
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Enable service:

sudo systemctl daemon-reload
sudo systemctl enable --now amdc.service

1
2

# Build Docker Image

Prepare installation package: AMDC-V2.0.4-xxxx-amd64.tar.gz

Write Dockerfile:

FROM ubuntu:20.04
ADD AMDC-V2.0.4-xxxx-amd64.tar.gz /opt/
WORKDIR /opt/amdc
EXPOSE 6359
CMD ["/opt/amdc/bin/amdc-server", "/opt/amdc/amdc.yaml"]

1
2
3
4
5

Build and run:

docker build -t apusic/amdc:v2.0.4 .
docker run -d \
  --name amdc \
  -p 6359:6359 \
  -v ./license.lic:/opt/amdc/license.lic \
  -v ./amdc.yaml:/opt/amdc/amdc.yaml \
  -v ./data:/opt/amdc/data \
  apusic/amdc:v2.0.4

1
2
3
4
5
6
7
8

Note: In production environments, always mount license, configuration, and persistence directories.

# Appendix: Key Configuration Optimization Recommendations

Configuration Item	Recommended Value	Description
`maxmemory`	70% of physical memory	Avoid OOM
`maxmemory-policy`	`allkeys-lru` or `allkeys-lfu`	Reasonable eviction
`timeout`	300	Idle connection timeout (seconds)
`tcp-keepalive`	60	TCP keepalive detection (seconds)
`slowlog-log-slower-than`	10000	Slow query threshold (microseconds)

编辑页面

#Operations Manual

← Migration Guide 发版说明→

Operations Manual

# Preface

# Target Audience

# Related Documentation

# Technical Support

# Daily Operations

# Service Start/Stop Management

# Normal Shutdown

# Forced Shutdown

# License Management

# Health Check List

# Basic Connection Check

# Master-Slave Replication Check

# Memory Usage

# Client Connections

# Persistence Status

# Performance Monitoring and Optimization

# Benchmark Testing

# Slow Query Analysis

# Big Key and Hot Key Detection

# Real-time Monitoring

# Monitoring Platform Integration (Recommended)

# Troubleshooting and Emergency Response

# Common Problem Diagnosis

# Key Error Log Keywords

# Emergency Recovery Procedures

# Service Unavailable

# Data File Corruption

# Automated Health Check

# Health Check Script

# Scheduled Tasks

# Monitoring Alert Threshold Recommendations

# Regular Maintenance Plan

# Software Management

# Configure System Service for Auto-start on Boot

# Build Docker Image

# Appendix: Key Configuration Optimization Recommendations