Migration Manual

# Preface

This document is the migration guide for ADMQ for Kafka (Apusic Distributed Message Queue, referred to as ADMQ for Kafka), detailing how to migrate from Kafka to ADMQ for Kafka.

# Target Audience

This document is intended for IT business leaders, R&D managers, software project managers, software architects, and operations engineers.

For more information about the ADMQ for Kafka product, please refer to the following ADMQ for Kafka product documentation set:

No. Document Description
1 ADMQ for Kafka Quick Start Guide A brief introduction on how to quickly get started with ADMQ for Kafka.
2 ADMQ for Kafka Installation Guide Detailed instructions on installing ADMQ for Kafka on various operating systems, as well as starting and stopping ADMQ for Kafka services.
3 ADMQ for Kafka Message Engine User Guide Detailed description of the usage, configuration, and management of ADMQ for Kafka message engine features and related tools.
4 ADMQ for Kafka Console User Guide Detailed description of the usage and operations of the ADMQ for Kafka management console.
5 ADMQ for Kafka Developer Guide Detailed instructions on developing ADMQ for Kafka client applications using various programming languages.
6 ADMQ for Kafka Migration Guide Detailed instructions on migrating from Kafka to ADMQ for Kafka.
7 ADMQ for Kafka Operations Guide Detailed description of monitoring, operations, security hardening, and other operational aspects of ADMQ for Kafka.
8 ADMQ for Kafka Performance Tuning Guide Detailed instructions on performance tuning for ADMQ for Kafka.

# Technical Support

The ADMQ for Kafka product provides comprehensive technical support services. You can obtain technical support through the following channels:

  • Website: www.apusic.com
  • Phone: 400-855-5800
  • Email: support@apusic.com
  • Kingdee Cloud Community: https://vip.kingdee.com/?productId=73&productLineId=14&lang=zh-CN

When contacting technical support, please provide the following information:

  1. Your name
  2. Company information and contact details
  3. Operating system and version
  4. Product version number
  5. Detailed information such as logs and screenshots of exceptions or errors

# Overview

This document details the best practices for migrating an existing Kafka cluster to ADMQ for Kafka V2.0.6, including compatible versions, configuration migration, metadata migration, and data migration.

# Compatible Kafka Versions

ADMQ for Kafka V2.0.6 is compatible with the following Kafka versions:

Source Kafka Version Compatibility Description
2.4.x Fully Compatible Recommended for migration
2.5.x Fully Compatible Recommended for migration
2.6.x Fully Compatible Recommended for migration
2.7.x Fully Compatible Recommended for migration
2.8.x Fully Compatible Recommended for migration
3.0.x Fully Compatible Recommended for migration
3.1.x Fully Compatible Recommended for migration
3.2.x Fully Compatible Recommended for migration
3.3.x Fully Compatible Recommended for migration
3.4.x Fully Compatible Recommended for migration
3.5.x Fully Compatible Recommended for migration
3.6.x Fully Compatible Recommended for migration
3.7.x Fully Compatible Recommended for migration
3.8.x Fully Compatible Recommended for migration

# Migration Strategies

Strategy Use Case Advantages Disadvantages
Rolling Upgrade Minor version upgrade within the same version No downtime Not suitable for cross-major-version upgrades
Dual Cluster Migration Heterogeneous cluster migration Low migration risk Requires double resources
Progressive Migration Production environment migration Gradual verification Long migration cycle

# Pre-migration Preparation

# 1. Environment Assessment

# Check source cluster status
kafka/bin/kafka-broker-api-versions.sh --bootstrap-server <source-kafka>:9092

# View cluster information
kafka/bin/kafka-topics.sh --list --bootstrap-server <source-kafka>:9092
kafka/bin/kafka-consumer-groups.sh --list --bootstrap-server <source-kafka>:9092
1
2
3
4
5
6

# 2. Data Volume Assessment

# Check Topic count
kafka/bin/kafka-topics.sh --list --bootstrap-server <source-kafka>:9092 | wc -l

# Check total message volume
kafka/bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand \
  --bootstrap-server <source-kafka>:9092 \
  --list
1
2
3
4
5
6
7

# 3. Install ADMQ

Deploy the ADMQ for Kafka cluster according to the installation guide, ensuring:

  • Network connectivity with the source cluster
  • Configuration compatibility with the source cluster parameters

# Configuration Migration

# Broker Configuration Migration

Key configurations to migrate:

Configuration Item Description Migration Suggestion
broker.id Broker ID Keep consistent or reassign
listeners Listener addresses Adjust IP according to the new environment
log.dirs Data directories Reassign directories
zookeeper.connect ADCC for zk address Point to the new ADCC for zk cluster
advertised.listeners Advertised addresses Configure with the new cluster address
compression.type Compression type Keep original configuration
log.retention.* Log retention Adjust as needed
num.partitions Default partition count Keep consistent

# Configuration Mapping Example

# Source cluster configuration (server.properties)
broker.id=0
listeners=PLAINTEXT://192.168.1.10:9092
log.dirs=/data/kafka
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka

# ADMQ configuration (kafka-broker.conf)
broker.id=0
listeners=PLAINTEXT://192.168.1.20:9092
log.dirs=/apusic/admq-kafka/data
zookeeper.connect=192.168.1.20:2181,192.168.1.21:2181,192.168.1.22:2181/admq
1
2
3
4
5
6
7
8
9
10
11

# JAAS Configuration Migration

If using SASL authentication:

# Migrate JAAS configuration (passwords below are examples only; use strong passwords in production)
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="admin-secret"
  user_admin="admin-secret"
  user_producer="producer-secret"
  user_consumer="consumer-secret";
};
1
2
3
4
5
6
7
8
9

# Metadata Migration

# Topic Migration

# 1. Generate migration plan
kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server <source-kafka>:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2" \
  --generate

# topics.json format
{
  "topics": [{"topic": "topic1"}, {"topic": "topic2"}]
}

# 2. Execute migration
kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server <source-kafka>:9092 \
  --reassignment-json-file reassign.json \
  --execute

# 3. Verify migration progress
kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server <source-kafka>:9092 \
  --reassignment-json-file reassign.json \
  --verify
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Method 2: Export and Import Topic Configuration

# Export Topic configuration
kafka/bin/kafka-configs.sh --bootstrap-server <source-kafka>:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --describe > topic-config-backup.txt

# Create the same Topic in ADMQ
kafka/bin/kafka-topics.sh --create \
  --bootstrap-server <admq-kafka>:9092 \
  --topic my-topic \
  --partitions 6 \
  --replication-factor 3
1
2
3
4
5
6
7
8
9
10
11
12

# ACL Migration

# Export ACLs
kafka/bin/kafka-acls.sh --bootstrap-server <source-kafka>:9092 \
  --list > acls-backup.txt

# Create the same ACLs in ADMQ
kafka/bin/kafka-acls.sh --bootstrap-server <admq-kafka>:9092 \
  --add --allow-principal User:producer \
  --operation Write --topic my-topic
1
2
3
4
5
6
7
8

# Consumer Group Offset Migration

# Export consumer group offsets
kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <source-kafka>:9092 \
  --group my-group \
  --describe > offset-backup.txt

# Reset offsets (for scenarios where consumption progress needs to be preserved)
kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <admq-kafka>:9092 \
  --group my-group \
  --topic my-topic \
  --reset-offsets --to-offset <offset-number> \
  --execute
1
2
3
4
5
6
7
8
9
10
11
12
13

# Data Migration

# Using MirrorMaker 2

MirrorMaker 2 is the officially recommended data migration tool for Kafka, supporting:

  • Topic data synchronization
  • ACL synchronization
  • Configuration synchronization
  • Consumer group offset synchronization

# Configuring MirrorMaker 2

# mm2.properties
clusters=source,target
source.bootstrap.servers=<source-kafka>:9092
target.bootstrap.servers=<admq-kafka>:9092

source->target.enabled=true
source->target.topics.regex=.*

# Internal Topic configuration
source->target.emit.syncs.interval.seconds=60
source->target.sync.group.offsets.enabled=true
source->target.sync.group.offsets.interval.seconds=60

# Namespace configuration
source->target.renames=sourcePrefixTransformerClass=org.apache.kafka.connect.transforms.LogRename
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Starting MirrorMaker 2

# Using connect-mirror-maker.sh
kafka/bin/connect-mirror-maker.sh mm2.properties &

# Or using distributed mode
kafka/bin/connect-distributed.sh connect-distributed.properties
1
2
3
4
5

# Using Kafka Connect Migration

// source-connector.json
{
  "name": "source-connector",
  "config": {
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "source.cluster.alias": "source",
    "target.cluster.alias": "target",
    "source.bootstrap.servers": "<source-kafka>:9092",
    "target.bootstrap.servers": "<admq-kafka>:9092",
    "topics": ".*",
    "sync.group.offsets.enabled": "true"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
# Create Connector
curl -X POST -H "Content-Type: application/json" \
  --data @source-connector.json \
  http://localhost:8083/connectors/
1
2
3
4

# Data Verification

# Verify data integrity
kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <admq-kafka>:9092 \
  --group mirror-consumer-group \
  --describe

# Compare message counts between source and target clusters
kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list <source-kafka>:9092 \
  --topic my-topic \
  --time -2 | tail -1

kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list <admq-kafka>:9092 \
  --topic my-topic \
  --time -2 | tail -1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Client Migration

# Switching Client Connection Address

Change the bootstrap.servers in the client configuration from the source cluster address to the ADMQ address:

# Original configuration
bootstrap.servers=192.168.1.10:9092,192.168.1.11:9092

# New configuration (ADMQ)
bootstrap.servers=192.168.1.20:9092,192.168.1.21:9092
1
2
3
4
5

# No Code Changes Required

Since ADMQ is fully compatible with the Kafka protocol, the following aspects require no modification:

  • Producer/consumer code
  • Serialization/deserialization methods
  • Consumer group management approach
  • Topic operation approach

# Verifying Client Migration

# Test producer
kafka/bin/kafka-console-producer.sh \
  --bootstrap-server <admq-kafka>:9092 \
  --topic test-migration

# Test consumer
kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server <admq-kafka>:9092 \
  --topic test-migration \
  --from-beginning
1
2
3
4
5
6
7
8
9
10

# Migration Steps

# Step 1: Deploy ADMQ Cluster

Complete the ADMQ deployment according to the installation guide.

# Step 2: Verify Cluster

# Verify ADMQ cluster health
kafka/bin/kafka-broker-api-versions.sh --bootstrap-server <admq-kafka>:9092
kafka/bin/kafka-topics.sh --create --topic test --partitions 3 --replication-factor 3
1
2
3

# Step 3: Configuration Migration

Migrate the source cluster configuration file parameters to the ADMQ configuration correspondingly.

# Step 4: Metadata Migration

Use tools to synchronize metadata such as Topics, ACLs, and consumer groups.

# Step 5: Data Migration

Use MirrorMaker 2 or Kafka Connect to synchronize data.

# Step 6: Client Switchover

Gradually switch clients to the ADMQ cluster in batches.

# Step 7: Verification

  • Verify message integrity
  • Verify consumption progress
  • Verify business functionality

# Step 8: Cleanup

After confirming the migration is successful, decommission the source cluster.

# Migration Risk Control

# Rollback Plan

  1. Keep the source cluster for a period of time
  2. Client configuration supports fast switching
  3. Record key configurations and states

# Migration Window

Data Volume Estimated Migration Time Description
< 1TB 2-4 hours Small cluster
1-10TB 4-12 hours Medium cluster
> 10TB 1-3 days Large cluster

# Precautions

  1. Network Bandwidth: Ensure sufficient network bandwidth between the source cluster and the ADMQ cluster
  2. Disk Space: The ADMQ cluster needs sufficient disk space to store data
  3. Business Off-peak Hours: It is recommended to perform data migration during business off-peak hours
  4. Monitoring and Alerts: Enable monitoring and alerts to detect anomalies promptly

# Frequently Asked Questions

# Topic Creation Failed

Cause: Topic name conflict or partition count exceeds the number of Brokers Solution: Check the Topic name and ensure the partition count ≤ Broker count

# Data Synchronization Latency

Cause: Insufficient network bandwidth or improper synchronization task configuration Solution: Check the network and increase the MirrorMaker concurrency

# Consumption Progress Lost

Cause: Consumer group offsets were not synchronized Solution: Enable the sync.group.offsets.enabled configuration

# ACL Not Effective

Cause: ACL configuration was not synchronized Solution: Manually synchronize the ACL configuration

编辑页面 (opens new window)
#迁移手册

← Operations Manual→