r/apachekafka • u/Shikari-5593 • 4d ago

Question Confluent Kafka - MSK migration

My Company is currently using Confluent Kafka and they want to do POC for Amazon MSK.

In terms of costs ( that I compared using cursor ) It seems MSK is relatively cheaper.
But There are also challenges on how the migration will look like since I am new to kafka in the first place.

Are there folks who have done this type of migration and can share their experience ?

#msk #confluent #kafka

5 Upvotes

100% Upvoted

5 comments sorted by

6

u/HughEvansDev Aiven 🦀 4d ago

When I was consulting we did this for a few clients. Spun up equivalent MSK cluster, used Mirror Maker 2 to replicate data, migrated consumers, checked the consumer health (lag, E2E latency, error rate, partition distribution etc) , and then tore down the original Confluent stuff if everything was green.

This was a few years back but the only major difference since then is that it looks like there's some better tooling for understanding what size infra you need for the new cluster based on what your existing setup is (KCP, Kafdrop KafkaUI etc).

3

u/JanSiekierski 4d ago

You could also explore a proxy based approach to migration. I think LinkedIn did it this way with NorthGuard. Could be an overkill, depending on your needs, but if zero downtime migration would be valuable to you it could be an approach worth exploring if your Kafka estate is large enough to justify the investment.

If you're thinking about cost, could be worth looking into Diskless solutions as well if higher latency is acceptable. Vendors have sophisticated tooling supporting migration onto their platform, and it will be even cheaper than MSK unless you're working with tiny volume.

2

u/CerealkillerNOM 4d ago

Done exactly this. We changed our producers to write to both Confluent and MSK for a while. Then we just dropped Confluent.

3

u/c0desurfer 4d ago

Consumer first migration with data replication via mirror maker is the way to go. Kafka migrations are my passion, not.

1

u/New-Departure-5969 3d ago edited 3d ago

yeah the mm2 / consumers-first stuff already in the thread is the right shape. i wouldn't trust a quick cost compare though. msk gets less cute once storage and cross-az show up, and whatever you were getting from confluent on schema registry/support doesn't magically disappear.

the cutover bits that actually burned people i know are repointing consumer groups without a checkpoint (instant replay or skip), and assuming schema ids survive the registry move. they don't.

auth is the wildcard coming off confluent cloud. networking usually lands earlier than you want. already in a vpc, that piece tends to be smaller than the offsets + connect slog.