Spring I/O 2022 - Barcelona, 26-27 May Slides: https://speakerdeck.com/timvanbaarsen/spring-kafka-beyond-the-basics-lessons-learned-on-our-kafka-journey-at-ing-bank GitHub repo: https://github.com/j-tim/spring-io-barcelona-2022-spring-kafka-beyond-the-basics You know the fundamentals of Apache Kafka. You are a Spring Boot developer and working with Apache Kafka. You have chosen Spring Kafka to integrate with Apache Kafka. You implemented your first producer, consumer and maybe some Kafka streams, it’s working… Hurray! You are ready to deploy to production what can possibly go wrong? In this talk, Tim will take you on a journey beyond the basics of Spring Kafka and will share his knowledge, pitfalls, and lessons learned based on real-life Kafka projects that are running in production for many years at ING Bank in the Netherlands. - Can you survive consuming a poison pill from a Kafka topic without writing gigabytes of log lines? - Do you understand how to deal with exceptions in different cases? - Do you validate incoming data consumed from a topic? - Should you do integration testing for Kafka? And what the other options do I have? - Can you incorporate Apache Kafka as part of your distributed tracing? - Are you able to monitor the performance of your Kafka applications to understand whether or not you are lagging behind? In case you answer one or more above questions with ‘NO’ join this talk! ======================================= Video Chapters 00:00:00 Introduction 00:03:47 Kafka in a nutshell 00:07:44 Spring for Apache Kafka 00:13:35 Scenario: Poison Pill (Deserialization exception) 00:25:43 Scenario: Lack of proper exception handling 00:32:18 Testing 00:37:30 Monitoring 00:42:12 Lessons learned