Performance Tuning

# Preface

This document introduces the best practices for performance optimization of ADMQ for Kafka (Apusic Distributed Message Queue for Kafka), helping users fully leverage system performance.

# Target Audience

This document is intended for IT business leaders, R&D managers, software project managers, software architects, and operations engineers.

For more information about ADMQ for Kafka, refer to the following ADMQ for Kafka product documentation set:

No.	Document	Description
1	ADMQ for Kafka Quick Start Guide	Briefly describes how to quickly get started with ADMQ for Kafka.
2	ADMQ for Kafka Installation Guide	Details how to install ADMQ for Kafka on various operating systems, as well as starting and stopping ADMQ for Kafka services.
3	ADMQ for Kafka Message Engine User Guide	Details the usage, configuration, and management of ADMQ for Kafka message engine features and supporting tools.
4	ADMQ for Kafka Console User Guide	Details the usage and operation of the ADMQ for Kafka console features.
5	ADMQ for Kafka Developer Guide	Details how to develop ADMQ for Kafka client applications using various programming languages.
6	ADMQ for Kafka Migration Guide	Details the migration from Kafka to ADMQ for Kafka.
7	ADMQ for Kafka Operations Guide	Details monitoring, operations, and security hardening for ADMQ for Kafka.
8	ADMQ for Kafka Performance Tuning Guide	Details performance tuning for ADMQ for Kafka.

# Technical Support

ADMQ for Kafka provides comprehensive technical support services. You can obtain technical support through the following channels:

Website: www.apusic.com
Phone: 400-855-5800
Email: support@apusic.com
Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN

When contacting technical support, please provide the following information:

Your name
Company information and contact details
Operating system and version
Product version number
Detailed information such as logs and screenshots of anomalies and errors

# Overview

Performance optimization requires a comprehensive approach across multiple layers:

Hardware: CPU, memory, disk, network
OS: Operating system parameters, kernel tuning
Broker: Configuration parameter optimization
Client: Producer and consumer configurations
Architecture: Topic design, partitioning strategy

# Hardware and OS Optimization

# Disk Selection

Disk Type	Recommended Scenario	Description
NVMe SSD	High-throughput scenarios	Best performance, suitable for core business
SSD	Production environments	Better than HDD, recommended
HDD	Low-cost scenarios	Only suitable for log-type scenarios

Optimization Recommendations:

Use dedicated disks for data directories
Enable disk RAID 0/10 for better performance
Avoid using network storage (NFS) as the primary data directory

# OS Parameter Tuning

# File descriptor limits
ulimit -n 1000000

# Network parameter optimization
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_max=67108864
sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"
sysctl -w net.ipv4.tcp_wmem="4096 16384 67108864"
sysctl -w net.core.netdev_max_backlog=250000

# Disable swap
sysctl -w vm.swappiness=0

# File system optimization
mount -o noatime,nodiratime /dev/sda1 /mnt/kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# NIC Optimization

Use multi-queue NICs with CPU affinity
Enable jumbo frames (MTU 9000)
Consider DPDK acceleration (high-throughput scenarios)

# Broker-Side Optimization

# Network and Thread Configuration

Parameter	Default	Recommended	Description
`num.network.threads`	8	CPU cores	Number of threads handling network requests
`num.io.threads`	16	Disks × 2	Number of threads handling disk I/O
`socket.send.buffer.bytes`	102400	204800	Send buffer size
`socket.receive.buffer.bytes`	102400	204800	Receive buffer size
`queued.max.requests`	500	1000	Request queue length waiting for I/O

# Log Storage Configuration

Parameter	Default	Recommended	Description
`log.segment.bytes`	1GB	512MB	Log segment size, smaller segments speed up log cleanup
`log.retention.hours`	168	Adjust per business	Log retention time
`log.retention.check.interval.ms`	300000	60000	Log cleanup check interval
`log.flush.interval.ms`	null	Not recommended to modify	Recommended to rely on PageCache
`log.flush.scheduler.interval.ms`	null	Not recommended to modify	Recommended to rely on PageCache

# Replication and Reliability Configuration

Parameter	Recommended	Description
`default.replication.factor`	3	3 replicas recommended for production
`min.insync.replicas`	2	Use with acks=all
`unclean.leader.election.enable`	false	Disable non-ISR leader election
`replica.lag.time.max.ms`	30000	ISR eviction threshold

# High-Throughput Scenario Configuration

# Network optimization
num.network.threads=16
num.io.threads=32
socket.send.buffer.bytes=307200
socket.receive.buffer.bytes=307200
queued.max.requests=2000

# Log optimization
log.segment.bytes=524288000
log.retention.check.interval.ms=300000

# Compression optimization
compression.type=zstd

# Replica optimization
replica.socket.timeout.ms=30000
replica.fetch.max.bytes=1048576
replica.fetch.min.bytes=1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Producer Optimization

# Core Configuration Parameters

Parameter	Recommended	Description
`acks`	all	Use when reliability is prioritized
`compression.type`	zstd	ZSTD offers the highest compression ratio
`batch.size`	16384-32768	Batch size (bytes)
`linger.ms`	5-20	Batch wait time
`buffer.memory`	64MB	Send buffer
`max.in.flight.requests.per.connection`	5	≤5 when idempotence is enabled
`retries`	Integer.MAX_VALUE	Number of retries
`enable.idempotence`	true	Enable idempotence

# High-Throughput Configuration

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.10:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// Reliability configuration
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

// Throughput optimization
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32768);
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 30000);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000);

// Retry configuration
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 100);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

# Partition Strategy Optimization

Key Partitioning: Use business keys for ordering guarantees
Round-Robin Partitioning: Achieve load balancing
Custom Partitioner: Partition based on business logic

// Custom partitioner example
public class CustomPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, 
                         Object value, byte[] valueBytes, Cluster cluster) {
        // Implement partitioning based on business logic
        return Math.abs(key.hashCode()) % cluster.partitionCountFor(topic);
    }
}

1
2
3
4
5
6
7
8
9

# Consumer Optimization

# Core Configuration Parameters

Parameter	Recommended	Description
`fetch.min.bytes`	1	Minimum fetch bytes
`fetch.max.wait.ms`	500	Maximum wait time
`max.poll.records`	500	Maximum records per poll
`session.timeout.ms`	30000	Session timeout
`heartbeat.interval.ms`	10000	Heartbeat interval
`enable.auto.commit`	false	Manual commit is more reliable
`auto.offset.reset`	earliest	Consume from beginning
`max.partition.fetch.bytes`	1048576	Maximum fetch bytes per partition

# High-Throughput Configuration

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.10:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

// Throughput optimization
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 1);
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 10485760);

// Reliability configuration
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 10000);

// Fetch optimization
props.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 60000);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Consumer Multi-Threading Optimization

// Multi-consumer thread example
ExecutorService executor = Executors.newFixedThreadPool(4);
for (int i = 0; i < 4; i++) {
    final int threadId = i;
    executor.submit(() -> {
        KafkaConsumer<String, String> consumer = createConsumer();
        consumer.subscribe(Arrays.asList("my-topic"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100);
            for (ConsumerRecord<String, String> record : records) {
                processRecord(record);
            }
            consumer.commitSync();
        }
    });
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# JVM Tuning

# Memory Configuration

Scenario	Heap Memory	GC Strategy
Dev/Test	-Xms2g -Xmx2g	G1
Production (small cluster)	-Xms8g -Xmx8g	G1
Production (large cluster)	-Xms16g -Xmx16g	G1

# Recommended GC Tuning Configuration

# G1 GC configuration
KAFKA_HEAP_OPTS="-Xms16g -Xmx16g -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:G1HeapRegionSize=16m \
  -XX:+ParallelRefProcEnabled"

# JMX configuration (for monitoring)
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.authenticate=false"

1
2
3
4
5
6
7
8
9
10
11
12

# Common Problem Diagnosis

Issue	Possible Cause	Solution
Frequent Full GC	Insufficient heap memory	Increase heap memory, optimize consumption logic
Long GC pauses	Improper G1 configuration	Adjust MaxGCPauseMillis
OOM	Memory leak or misconfiguration	Check JVM and client configurations

# Topic Design Optimization

# Partition Count Design

Target Throughput	Recommended Partitions	Description
100K/sec	10-20	~50-100K/sec per partition
500K/sec	50-100	Requires more Brokers
1M/sec	100-200	Requires large cluster

Formula:

Partitions = Target Throughput / Throughput per Partition
Throughput per Partition ≈ 50-100K/sec (depends on message size and hardware)

1
2

# Partition Rebalancing

Adding partitions triggers rebalancing, affecting performance. Recommendations:

Plan sufficient partitions in advance
Use sticky partition strategy (Kafka 2.4+)

# Sticky partition strategy (default)
kafka-topics.sh --alter --topic my-topic --partitions 20

1
2

# Log Compaction Configuration

# Log compaction configuration
cleanup.policy=compact
min.compaction.lag.ms=3600000
delete.retention.ms=86400000
segment.bytes=1073741824

1
2
3
4
5

# Monitoring and Diagnostics

# Key Performance Metrics

Metric	Description	Alert Threshold
MessagesInPerSec	Messages per second	Set per business
BytesInPerSec	Inbound bytes per second	Set per business
BytesOutPerSec	Outbound bytes per second	Set per business
UnderReplicatedPartitions	Under-replicated partitions	> 0
OfflinePartitionsCount	Offline partitions	> 0
RequestLatencyAvg	Average request latency	> 100ms
FetchQueueSize	Request queue size	Continuously growing
PurgatorySize	Pending requests	Continuously growing

# Monitoring with JMX

# Enable JMX port
JMX_PORT=9999 bin/admq-daemon start kafka standalone

# Connect using jconsole
jconsole localhost:9999

1
2
3
4
5

# Monitoring with Prometheus

Configure Prometheus to collect metrics:

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['localhost:12305']

1
2
3
4

# Common Performance Issues

Issue	Symptom	Solution
High producer latency	Increased send time	Increase batch.size, reduce linger.ms
Large consumer lag	Slow consumption	Increase consumer count, optimize processing logic
High disk I/O	Slower writes	Use SSD, optimize log retention policy
Network bottleneck	Low throughput	Increase network bandwidth, use compression
Memory pressure	Frequent GC	Increase heap memory, adjust GC parameters

# Performance Testing Recommendations

# Test Tools

kafka-producer-perf-test.sh: Producer performance testing
kafka-consumer-perf-test.sh: Consumer performance testing
kafka-throttle.sh: Throttle testing

# Test Commands

# Producer performance test
kafka/bin/kafka-producer-perf-test.sh \
  --topic test-topic \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=192.168.1.10:9092

# Consumer performance test
kafka/bin/kafka-consumer-perf-test.sh \
  --topic test-topic \
  --messages 1000000 \
  --group perf-test-group \
  --bootstrap-server 192.168.1.10:9092

1
2
3
4
5
6
7
8
9
10
11
12
13
14

编辑页面

#性能优化手册

← Operations Manual Quick Operation Guide→

Performance Tuning

# Preface

# Target Audience

# Related Documents

# Technical Support

# Overview

# Hardware and OS Optimization

# Disk Selection

# OS Parameter Tuning

# NIC Optimization

# Broker-Side Optimization

# Network and Thread Configuration

# Log Storage Configuration

# Replication and Reliability Configuration

# High-Throughput Scenario Configuration

# Producer Optimization

# Core Configuration Parameters

# High-Throughput Configuration

# Partition Strategy Optimization

# Consumer Optimization

# Core Configuration Parameters

# High-Throughput Configuration

# Consumer Multi-Threading Optimization

# JVM Tuning

# Memory Configuration

# Recommended GC Tuning Configuration

# Common Problem Diagnosis

# Topic Design Optimization

# Partition Count Design

# Partition Rebalancing

# Log Compaction Configuration

# Monitoring and Diagnostics

# Key Performance Metrics

# Monitoring with JMX

# Monitoring with Prometheus

# Common Performance Issues

# Performance Testing Recommendations

# Test Tools

# Test Commands