This video was recorded at Code BEAM America 2022 - https://codebeamamerica.com/ Network Observability at LinkedIn | Ananya Shandilya - Staff Software Engineer at LinkedIn ABSTRACT The LinkedIn infrastructure has thousands of services serving millions of requests per second. At this scale, various kinds of data points must be collected, processed and observed to maintain the health of the infrastructure - one of them is network flows. Our infrastructure components export flows at the rate of 2M packets per second. This talk describes a data collection, processing and storage system for network flow data written in Erlang. It gives an overview of the system’s architecture and some of the interesting challenges we faced while scaling this system. OBJECTIVE Share the details about the data processing system we built at LinkedIn using Erlang for collecting and processing high volume and velocity of network flow data AUDIENCE Erlang developers / Observability engineers / Backend engineers • Timecodes 00:00 - 02:02 - Intro and Agenda 02:03 - 07:15 - Intro to InFlow 07:16 - 11:15 - Applications of Flow Data 11:16 - 22:18 - InFlow Collector Architecture 22:19 - 41:22 - InFlow Collector at Scale • Follow us on social: Twitter: https://twitter.com/CodeBEAMio LinkedIn: https://www.linkedin.com/company/27159258 • Looking for a unique learning experience? Attend the next Code Sync conference near you! See what's coming up at: https://codesync.global • SUBSCRIBE TO OUR CHANNEL https://www.youtube.com/channel/UC47eUBNO8KBH_V8AfowOWOw See what's coming up at: https://codesync.global