Read the abstract ➤ https://www.conf42.com/Site_Reliability_Engineering_SRE_2024_Pranay_Prateek_scaling_opentelemetry_kafka Other sessions at this event ➤ https://www.conf42.com/sre2024 Support our mission ➤ https://www.conf42.com/support Join Discord ➤ https://discord.gg/DnyHgrC7jC SigNoz on slack ➤ https://signoz-community.slack.com/join/shared_invite/zt-2gag5t3k4-WE5I6xpNbczyDJNdLLJkAg#/shared-invite/email Chapters 0:00 intro 0:26 preamble 0:33 about me 0:51 signoz - open source observability platform 1:46 what is opentelemetry? 2:19 why is opentelemetry important? 4:14 introduction to opentelemetry collector 6:00 architecture of signoz cloud (single tenant) without kafka 7:09 issues with scaling with just opentelemetry collector 8:17 architecture of signoz cloud with kafka 10:01 how kafka can help 12:32 kafka setup, records 13:44 monitoring consumer lag is important 15:03 scaling based on consumer lag 16:15 monitoring producer - consumer latency 16:44 kafka based architecture is working well so far... 17:29 potential improvements 18:51 get involved in a growing community 19:20 thank you