Read the abstract ➤ https://www.conf42.com/Incident_Management_2023_Christine_Yen_sre_observability_building_llms Other sessions at this event ➤ https://www.conf42.com/im2023 Join Discord ➤ https://discord.gg/DnyHgrC7jC Honeycomb Blogpost about LLMs ➤ https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm Chapters 0:00 intro 2:08 preamble 2:34 magic of llms llms 3:39 - like apis we know and love 7:43 - even more unpredicability 9:03 - how do we define "correct"? observability 10:36 about 11:48 - what's in the box 14:20 - endless feedback loops 16:01 why believe me? query assistant 16:18 - timeline 16:31 - goals 19:14 laws of building on llms 20:06 how do we go forward? instrumentation 21:24 instrumentation for llms 25:01 emerging behaviors 28:06 a truth for llms 28:34 service level objectives 29:00 slos: a quick definition 30:06 slos for developing with llms 32:07 from others in the wild 32:20 duolingo 33:34 intercom 35:01 so in the end: 35:38 thanks!