Troubleshooting Common Issues

Issue 1: Port Occupation and Resolution

Installing AMDC occupies certain ports, and the client connection relies on internal DNS services. If you cannot guarantee that the ports are unused, or if the installation environment lacks effective DNS, startup may fail. Default ports occupied by the product:

• AMDC Cache Service default port:6359
• AMDC Sentinel Service default port: 26359
• AMDC Console default port: 9001

If the aforementioned ports are occupied before startup, it will result in a failure. You can switch the ports or stop the programs currently using them.

Issue 2: Modifying the AMDC Cache Service Port

Edit the configuration file: vim conf.yaml，port field

Issue 3: Modifying the AMDC Console Port

Edit the configuration file: vim amdc-console/config.yaml, modify the HTTP_PORT field.

For more configuration files, refer to Chapter III of Section I.

Issue 4: Loading the ACL File

Place the ACL file in the directory specified by the AMDC cache configuration item "ACLFile", and execute ACL load in the AMDC cache command line to reload the user configurations in the ACL file.

Note:

1. Reloading the ACL file will replace existing ACL rules in the cache.
2. If there is any invalid data in one or more entries in the ACL file, the entire file will fail to load, and the existing rules in the cache system will continue to be used.

Issue 5: Resolving AMDC Master-Slave Failover

In the master-slave mode of AMDC, when a slave process is started, it will send a SYNC Command to the master to request a synchronous connection. Whether it is the first connection or a reconnection, the master will start a background process to save the data snapshot to the data file. At the same time, the master will record all commands that modify the data and cache them in the data file. After the background process completes the caching operation, the master will send the data file to the slave. The slave side saves the data file on the hard disk, then loads it into the memory, and then the master will send all the operations that modify the data to the slave side.

1. In the cache service cluster, if a slave fails, the system will reconnect automatically. Should a Slave fail and go offline, it will automatically reconnect upon recovery. Upon receiving the Slave's connection, the Master sends its complete data file to the Slave. If the Master receives synchronization requests from multiple Slaves simultaneously, it will only start one background process to save the data file, then send it to all Slaves to ensure the Slave's normal operation.
2. In the cache service cluster, if the master fails, manually restarting the main service enables read-write operations by the client. When the master cache service fails, it still supports read operations by the client on the slave service. You need to manually restart the main service. Once the main service is restarted and the system has recovered, it will again support read-write operations by the client on the main service.

Issue 6: Request Latency Issues

If you encounter noticeable latency in AMDC requests, you can identify the specific problem with the following methods:

1. slowlog command: This command can query commands with longer execution times (the length of time can be configured via the slowlog parameter in the conf.yaml file), allowing better tracking of the code execution process.
2. –bigkeys: Use the amdc-cli client, connect and add the –bigkeys parameter to query the largest values, thus determining whether the delay is due to excessively large values.
3. –memkeys: Similarly, this parameter queries the total memory usage of key-value pairs to determine whether the delay is due to overly large key-value pairs.

Issue 7: Monitoring Executed Commands

Use amdc-cli to connect, then use the monitor command to monitor the commands continuously received by AMDC. Alternatively, use the following command to record them in the background to the monitor.log file:

	amdc-cli -h [ip] -p [port] [-a password] monitor >monitor.log 2>&1 &

To stop recording to monitor.log, use the following command:

	ps -ef | grep amdc-cli | grep -v grep | awk '{print $2}' | xargs kill -9

Issue 8: Process Running But Unable to Connect

When the client exceeds Linux limits or uses more memory than the license limit, it may appear that the process is running but connection timeouts or connection failures occur.

1. Client exceeding Linux limits: Some Linux systems have a default limit of 1024 for client connections. When there are too many business applications producing over 1024 client connections, the excess will be restricted by Linux and unable to connect further.

   Solution: Permanently set the user-level file descriptor limit by adding the following lines to /etc/security/limits.conf:

* soft nofile 65535
* hard nofile 65535

Note: Different Linux systems may require different methods for modification. If the above does not work, please search for the corresponding system's TCP connection count.

2. Exceeding the license memory limit: The default memory limit for licenses is generally 6GB (for most cases). When the business system writes data exceeding 6GB and continues to write, it causes frequent triggering of the memory eviction algorithm in AMDC, leading to a decrease in AMDC's performance. Under continuous high concurrency, some requests may timeout, causing errors in the business segment.

   Solution: Expanding capacity resolves the issue. There are two methods: one is to apply for a license with greater memory, directly increasing the usable memory per node (not recommended to exceed 16GB, as larger sizes can lead to reduced query efficiency); the second is to implement a distributed cluster mode (corresponding to Redis' cluster mode).

Issue 9: System CPU Continuously Occupied at High Levels

Similar to issue 8, part 2, frequent triggering of the eviction algorithm significantly increases CPU utilization and maintains high levels over long periods. The solution is the same as for issue 8, part 2.