Kafka Migration at Scale - Bikram Chatterjee | ElixirConf EU 2024

Conference: ElixirConf EU 2024

Year: 2024

✨This talk was recorded at ElixirConf EU 2024. If you're curious about our upcoming event, check https://elixirconf.eu ✨ In this talk, we’ll explore how the diverse and dynamic capabilities of the Elixir ecosystem effectively supported the migration of thousands of Kafka topics and clients from self-hosting to cloud at Helvetia on a large corporate scale. Over this year-long migration, we encountered various technical roadblocks due to the volume and complexity involved. However, we successfully overcame these challenges with Elixir to ensure a non-disruptive transition experience for our hundreds of applications processing billions of messages per day. We adopted several useful Elixir language features and patterns to build the required tooling to support this process: • Improving performance with concurrent design • Multiple data transformation and validation scripts • Directed dependency graphs • Excel reporting framework • Macros • Templating with EEx