Operations Manual

# Preface

This document is the operations guide for ADMQ for Kafka (Apusic Distributed Message Queue, referred to as ADMQ for Kafka). It aims to provide middleware operations personnel with standardized operational guidelines for ADMQ for Kafka, covering core scenarios such as installation and deployment, configuration optimization, routine operations, troubleshooting, and security hardening, ensuring stable, efficient, and secure operation of the ADMQ for Kafka service.

# Target Audience

This document is intended for IT business leaders, R&D managers, software project managers, software architects, and operations engineers.

For more information about the ADMQ for Kafka product, please refer to the following ADMQ for Kafka product documentation set:

No.	Document	Description
1	ADMQ for Kafka Quick Start Guide	A brief introduction on how to quickly get started with ADMQ for Kafka.
2	ADMQ for Kafka Installation Guide	Detailed instructions on installing ADMQ for Kafka on various operating systems, as well as starting and stopping ADMQ for Kafka services.
3	ADMQ for Kafka Message Engine User Guide	Detailed description of the usage, configuration, and management of ADMQ for Kafka message engine features and related tools.
4	ADMQ for Kafka Console User Guide	Detailed description of the usage and operations of the ADMQ for Kafka management console.
5	ADMQ for Kafka Developer Guide	Detailed instructions on developing ADMQ for Kafka client applications using various programming languages.
6	ADMQ for Kafka Migration Guide	Detailed instructions on migrating from Kafka to ADMQ for Kafka.
7	ADMQ for Kafka Operations Guide	Detailed description of monitoring, operations, security hardening, and other operational aspects of ADMQ for Kafka.
8	ADMQ for Kafka Performance Tuning Guide	Detailed instructions on performance tuning for ADMQ for Kafka.

# Technical Support

The ADMQ for Kafka product provides comprehensive technical support services. You can obtain technical support through the following channels:

Website: www.apusic.com
Phone: 400-855-5800
Email: support@apusic.com
Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN

When contacting technical support, please provide the following information:

Your name
Company information and contact details
Operating system and version
Product version number
Detailed information such as logs and screenshots of exceptions or errors

# Overview

ADMQ Kafka is a high-throughput distributed messaging system commonly used for log collection, data pipelines, and real-time stream processing.

# Basic Concepts

Component	Description
Broker	Kafka service node
Topic	Message category
Partition	Partition of a Topic
Producer	Message producer
Consumer	Message consumer
Consumer Group	Consumer group

Simple data flow: Producer → Broker → Topic → Partition → Consumer Features:

High throughput
Distributed
Sequential disk writes
Replication mechanism ensures reliability

# Common Operations Commands

Process control commands are located in the admq-kafka/bin directory, and the required configuration files are in the admq-kafka/config directory. Open-source Kafka scripts are supported; you can enter the admq-kafka/kafka/bin directory to execute the corresponding scripts.

# Process Management

# Start/Stop Kafka ZK Node

bin/admq-daemon start kafka zk

bin/admq-daemon stop kafka zk

1
2
3

# Start/Stop Kafka Broker Node

bin/admq-daemon start kafka broker 1
bin/admq-daemon stop kafka broker 2

1
2

# Topic Management

# List Topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

# View Topic Details

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test

# Create Topic

kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create \
--topic test \
--partitions 3

1
2
3
4
5

# Delete Topic

kafka-topics.sh \
--bootstrap-server localhost:9092 \
--delete \
--topic test

1
2
3
4

# Consumer Management

# List Consumer Groups

kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--list

1
2
3

# View Consumption Status

kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group test-group

1
2
3
4

Key metrics:

Metric	Meaning
CURRENT-OFFSET	Current consumption offset
LOG-END-OFFSET	Latest message offset
LAG	Backlog

# Produce/Consume

# Produce Messages

kafka-console-producer.sh \
--broker-list localhost:9092 \
--topic test

1
2
3

# Consume Messages

kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test \
--from-beginning

1
2
3
4

# Routine Inspection

# Broker Status

Check if the broker is online:

ps -ef | grep admq-kafka

netstat -ntlp | grep 9092

# ISR Status

Check:

kafka-topics.sh --describe

Judgment:

ISR < Replicas

Indicates abnormal replica synchronization.

# Consumer Lag

Check:

kafka-consumer-groups.sh --describe

Judgment:

LAG > 0

If it continues to grow, investigation is needed.

# Disk Usage

Check:

df -h

Recommendations:

Metric	Recommendation
Disk usage	<70%
IO util	<80%

# Common Troubleshooting

# Consumer Backlog

Symptom:

LAG continues to grow

Causes:

Insufficient consumer capacity
Broker performance bottleneck
Network issues

Solutions:

Method	Description
Add more Consumers	Increase parallelism
Add more Partitions	Improve throughput
Optimize business logic	Reduce processing time

# Broker Down

Symptom:

Leader not available

Check:

ps -ef |grep admq-kafka

View logs:

cat logs/server.log

Resolution:

bin/admq-daemon start kafka broker/standalone

# ISR Not In Sync

Symptom:

ISR < Replicas

Causes:

Cause	Description
High disk IO	Follower sync is slow
Network latency	Replica replication is slow
High CPU load	Broker under pressure

Check:

iostat -x

# Topic Has No Leader

Symptom:

Leader: none

Resolution:

# Execute leader election
kafka-leader-election.sh \
--bootstrap-server localhost:9092 \
--all-topic-partitions \
--election-type preferred

1
2
3
4
5

# Other Issues

ADCC for zk startup fails, log reports Connection refused or Session expired
- Check if the myid file exists and its content is correct.
- Check if the IP addresses in server.x within zookeeper.properties are reachable, and if ports 2888/3888 are blocked by the firewall.
Kafka startup fails, log reports Broker ID 0 is already registered
- Check for residual old ADCC for zk data. For a new deployment, clear the ADCC for zk dataDir and Kafka log.dirs and retry.
- Verify that broker.id across nodes is unique.
ISR set incomplete (Isr: 0,1 instead of 0,1,2)
- Indicates that a Follower is syncing too slowly or the network is unreachable. Check the node's logs and network bandwidth.
Kafka startup fails, log reports Failed to bind to port 9092
- Check if port 9092 is occupied by another process (use netstat -tulpn | grep 9092 or lsof -i:9092).
- Check if the port configured in listeners in server.properties matches the intended port to avoid configuration conflicts.
- Confirm that the firewall is not blocking port 9092, or that the port has been allowed in the firewall rules.
Kafka producer fails to send messages, reports Leader not available
- Check if the corresponding Topic exists; create it if it does not (using the kafka-topics.sh --create command).
- Check if the Kafka cluster is running normally and all Broker nodes are online.
- Check if the Topic's replica configuration is reasonable; if the replication factor exceeds the number of available Brokers, leader election cannot proceed.
Kafka consumer fails to consume, reports No offset found for partition
- For a new consumer group, having no offset on first consumption is normal. Configure auto.offset.reset to earliest or latest.
- For an existing consumer group, check if offset data in ADCC for zk / Kafka has been manually deleted, or if the consumer group has been inactive for so long that offsets have expired (adjust the offsets.retention.minutes configuration).
After ADCC for zk cluster starts, some nodes cannot join the cluster, log reports Cannot open channel to x at election address
- Check that the server.x IP/port configuration in zoo.cfg is consistent across all ADCC for zk nodes with no typographical errors.
- Check that the time is synchronized between nodes (a time difference exceeding 2000ms can cause election failure). Synchronize time using ntpdate.
- Ensure the dataDir directory has permissions allowing the ADCC for zk running user to read and write, preventing permission issues from reading the myid file.
Kafka log reports Not enough replicas, messages cannot be persisted
- Check if the Topic's min.insync.replicas configuration value is greater than the current number of available in-sync replicas (ISR).
- For a single Broker deployment, set min.insync.replicas to 1 (default is 1; adjust if manually changed).
- Check if the Follower nodes are synchronizing normally, and investigate issues such as network latency and high disk IO causing slow synchronization.
After Kafka starts, JVM runs out of memory (OOM), process crashes
- Check the JVM configuration (KAFKA_HEAP_OPTS) in kafka-server-start.sh. The default heap memory may be too large or too small.
- For a single-machine test environment, adjust to -Xmx512M -Xms512M. For production, configure based on server memory (recommended not to exceed 50% of physical memory).
- Check if too many Topics/Partitions are enabled, causing high memory usage. Plan the number of partitions reasonably.
SSL handshake failed when connecting to Kafka
- If SSL authentication is enabled, check that the SSL certificate path and password configuration are correct on both the client and server sides.
- Confirm that the certificate has not expired and that the server's listeners is configured as SSL:// instead of PLAINTEXT://.
- Check that the client has correctly configured SSL-related parameters (e.g., security.protocol=SSL).

# Kafka Scaling and Maintenance

# Adding Partitions

kafka-topics.sh \
--bootstrap-server localhost:9092 \
--alter \
--topic test \
--partitions 6

1
2
3
4
5

Note: After scaling, message ordering may change.

# Partition Reassignment

Generate plan:

kafka-reassign-partitions.sh \
--bootstrap-server localhost:9092 \
--generate

1
2
3

Execute:

kafka-reassign-partitions.sh \
--execute

1
2

# Kafka Performance Tuning

# Broker Parameters

Key parameters:

num.network.threads=8
num.io.threads=16
log.segment.bytes=1GB
log.retention.hours=168

1
2
3
4

# Producer Optimization

Recommended:

acks=all
compression.type=snappy
batch.size=32768
linger.ms=10

1
2
3
4

# Consumer Optimization

fetch.min.bytes=1
fetch.max.wait.ms=500
max.poll.records=500

1
2
3

# Log Management

Log path: admq-kafka/logs

Common errors:

Leader not available
NotEnoughReplicas
Connection refused

1
2
3

# Configuration Management

# Default Configuration

Path: admq-kafka/configs

kafka-zk.conf: Configuration file used by ZK
kafka-broker.conf: Configuration used by broker nodes in cluster deployment
kafka-standalone.conf: Configuration used by broker node in standalone deployment

# Core Configuration Items

listeners=PLAINTEXT://192.168.1.10:9092
# For external access, add: EXTERNAL://your.public.ip:9093
advertised.listeners=PLAINTEXT://192.168.1.10:9092
# Security protocol mapping (e.g., when using multiple listeners)
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
# Log retention time (7 days)
log.retention.hours=168
# Or retain by size (e.g., max 10GB per partition)
# log.retention.bytes=10737418240
# Log segment file size (default 1GB)
log.segment.bytes=1073741824
# Background log flush strategy (rely on OS flush for better efficiency; do not actively flush for HDD)
log.flush.interval.messages=9223372036854775807
log.flush.interval.ms=9223372036854775807

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Security Hardening

# 1. Network Security

# 1.1 Firewall Policy

In production environments, strictly restrict access to Kafka ports:

# Allow only internal network access to Kafka port
iptables -A INPUT -p tcp --dport 9092 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 9092 -j DROP

# Allow only console management IP access
iptables -A INPUT -p tcp --dport 12305 -s 10.0.0.10 -j ACCEPT
iptables -A INPUT -p tcp --dport 12305 -j DROP

1
2
3
4
5
6
7

# 1.2 Network Isolation

Deploy the Kafka cluster in an independent security group/VPC
Prohibit direct exposure of Kafka ports to the internet
Use VPN or dedicated lines for cross-datacenter access

# 2. Authentication and Authorization

# 2.1 Enable SASL Authentication

# Broker configuration
listeners=SASL_PLAINTEXT://0.0.0.0:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512

# Enable ACLs
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
allow.everyone.if.no.acl.found=false

1
2
3
4
5
6
7
8
9

# 2.2 Configure User Authentication

# Create user and set password
kafka-configs.sh --zookeeper localhost:2181 \
  --alter --add-config 'SCRAM-SHA-512=[password=your-secure-password]' \
  --entity-type users --entity-name admin

# List users
kafka-configs.sh --zookeeper localhost:2181 \
  --describe --entity-type users

1
2
3
4
5
6
7
8

# 2.3 Configure ACL Rules

# Production environment access control
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:app-producer \
  --operation Write --topic order-topic

kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --allow-principal User:app-consumer \
  --operation Read --topic order-topic \
  --group order-consumer-group

# Deny anonymous access
kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --add --deny-principal User:ANONYMOUS \
  --operation All --cluster

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# 3. Data Encryption

# 3.1 SSL/TLS Configuration

# Enable SSL
listeners=SSL://0.0.0.0:9093
security.inter.broker.protocol=SSL

# SSL certificate configuration
ssl.keystore.location=/path/to/server.keystore.jks
ssl.keystore.password=your-keystore-password
ssl.key.password=your-key-password
ssl.truststore.location=/path/to/server.truststore.jks
ssl.truststore.password=your-truststore-password
ssl.client.auth=required

1
2
3
4
5
6
7
8
9
10
11

# 3.2 Certificate Management

# Generate self-signed certificate (test environment)
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey

# Export certificate
keytool -keystore server.keystore.jks -alias localhost -export -file cert.crt

# Import truststore
keytool -keystore server.truststore.jks -alias ca-cert -import -file ca-cert.crt

1
2
3
4
5
6
7
8

# 4. Auditing and Monitoring

# 4.1 Enable Audit Log

# log4j configuration
log4j.logger.kafka.authorizer.logger=WARN, authorizerAppender
log4j.additivity.kafka.authorizer.logger=false
log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.authorizerAppender.File=${log.dir}/kafka-authorizer.log
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

1
2
3
4
5
6
7

# 4.2 Monitoring and Alert Items

Monitoring Item	Alert Threshold	Description
Authentication failures	> 10 times/min	Possible brute-force attack
Unauthorized access	> 0	Investigate immediately
SSL certificate expiration	< 30 days	Renew in advance
ACL changes	Any change	Requires audit confirmation

# 5. System Security

# 5.1 Running User

# Create dedicated user
useradd -r -s /bin/false admq-kafka

# Change file owner
chown -R admq-kafka:admq-kafka /apusic/admq-kafka

# Restrict directory permissions
chmod 750 /apusic/admq-kafka
chmod 600 /apusic/admq-kafka/config/*.conf
chmod 600 /apusic/admq-kafka/license.xml

1
2
3
4
5
6
7
8
9
10

# 5.2 systemd Security Options

[Service]
User=admq-kafka
Group=admq-kafka
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

1
2
3
4
5
6
7

# 6. Security Inspection Checklist

Check Item	Check Method	Pass Criteria
Firewall rules	`iptables -L`	Only necessary ports open
Authentication status	View configuration	SASL/SCRAM enabled
ACL configuration	`kafka-acls.sh --list`	No anonymous access permissions
SSL certificate	`keytool -list`	Certificate not expired
File permissions	`ls -la`	Config files 600
Running user	`ps -ef`	Non-root user
Audit log	`ls -la logs/`	Authorizer log exists

# Operations Experience Summary

Kafka failures are mostly concentrated in:

Issue	Percentage
Consumer backlog	50%
Broker down	20%
ISR not in sync	15%
Disk full	10%
Network issues	5%

Operations mantra: First check Broker, then check Leader, then check ISR, then check Lag, finally check IO.

编辑页面

#运维手册

← Migration Manual Performance Tuning→

Operations Manual

# Preface

# Target Audience

# Related Documents

# Technical Support

# Overview

# Basic Concepts

# Common Operations Commands

# Process Management

# Start/Stop Kafka ZK Node

# Start/Stop Kafka Broker Node

# Topic Management

# List Topics

# View Topic Details

# Create Topic

# Delete Topic

# Consumer Management

# List Consumer Groups

# View Consumption Status

# Produce/Consume

# Produce Messages

# Consume Messages

# Routine Inspection

# Broker Status

# ISR Status

# Consumer Lag

# Disk Usage

# Common Troubleshooting

# Consumer Backlog

# Broker Down

# ISR Not In Sync

# Topic Has No Leader

# Other Issues

# Kafka Scaling and Maintenance

# Adding Partitions

# Partition Reassignment

# Kafka Performance Tuning

# Broker Parameters

# Producer Optimization

# Consumer Optimization

# Log Management

# Configuration Management

# Default Configuration

# Core Configuration Items

# Security Hardening

# 1. Network Security

# 1.1 Firewall Policy

# 1.2 Network Isolation

# 2. Authentication and Authorization

# 2.1 Enable SASL Authentication

# 2.2 Configure User Authentication

# 2.3 Configure ACL Rules

# 3. Data Encryption

# 3.1 SSL/TLS Configuration

# 3.2 Certificate Management

# 4. Auditing and Monitoring

# 4.1 Enable Audit Log

# 4.2 Monitoring and Alert Items

# 5. System Security

# 5.1 Running User

# 5.2 systemd Security Options

# 6. Security Inspection Checklist

# Operations Experience Summary