CNCF-hosted Co-located Events North America 2024

The OpenTelemetry (OTel) Operator is a great tool that helps make your life a little easier by managing OTel for you in your Kubernetes cluster, by: Managing the deployment of the OpenTelemetry Collector Managing the configuration of a fleet of OpenTelemetry Collectors via OpAMP integration Injecting and configuring auto-instrumentation into your pods But what happens when THINGS. DON’T. WORK??

In this talk, Adriana and Reese will cover:

• An overview of the OTel Operator 
• Common installation issues
• Common auto-instrumentation issues
• Common OTel Collector deployment issues
• * …and how to tackle them all

Attendees will walk away from this session with a better understanding of how they can leverage the Operator, and be empowered to use it with confidence.

Continuous profiling is a technique to collect stack trace granularity insight into production resource usage. It is something SREs and other engineers can enable without changes to the app, or knowing how it was compiled. This year, this powerful signal and a polyglot eBPF profiling agent were added to OpenTelemetry. Our talk explores how an existing OpenTelemetry system is better with profiling, specifically how distributed tracing fits into the picture. You'll see both tools in action on Kubernetes, including cross-service requests and how correlation of distributed traces and profiles let you answer more questions, specifically code level causality. We'll show how to leverage this data for resource utilization and even monitoring your carbon footprint. You'll leave with a concrete understanding of continuous profiling, how it relates to OpenTelemetry and how these tools combine to reduce time while adding more understanding of your Kubernetes workloads, from kernel to code.

LLMs are huge to deploy and manage. Have you ever wondered if there's a way to pinpoint exactly which parts of your code are draining resources, causing latency, and hurting performance? In a world where efficiency is crucial, dynamically inspecting application behavior and performance at runtime can be transformative. Join this session to explore how to leverage OpenTelemetry’s profiling feature to optimize LLM code at a much deeper level.

We'll cover how to:
1. Identify specific pieces of code that consume excessive CPU and memory, or cause memory leaks and OOM errors.
2. Improve LLM performance by understanding model behavior, reducing latency, and meeting SLAs and SLOs.
3. Achieve efficient deployments on Kubernetes, ensuring optimal resource utilization and cost savings.

WebAssembly is the next platform for computing, and this time, we can have observability from day one. In building distributed WebAssembly on top of wasmCloud, we built in the full OpenTelemetry ("OTEL") trifecta: traces, metrics and logs.

**Along the way we found a new way to achieve the holy grail of observability — free application & backing service instrumentation.**

This talk will cover how we implemented OTEL in wasmCloud and the benefits and challenges we faced. In a live demo, you will learn how you can trace globally distributed applications written in different programming languages and connected to a variety of backend services.

OTel community has been working on standardizing semantic conventions to correlate telemetry data from various systems. The Messaging SemConv aims to solve it for commonly used queues like Kafka, RabbitMQ, and others systems. OTel instrumentations are adopting these conventions, but the end users still face challenges with async messaging observability at scale. They struggle with questions like "how to trace message flow?", "how to correlate metrics with traces?", "how to do capacity planning and cost optimizations based on telemetry data?". The end-to-end visibility often remains a black box! In this session, through a demo, we'll delve deeper into async architecture to address these questions, demonstrate context propagation within queues, and show how to correlate traces and client or broker-side metrics. Participants will gain hands-on experience with messaging instrumentation, learning how to achieve observability in both simple and complex asynchronous messaging scenarios.

This session provides an in-depth dissection and live demonstration of OpenTelemetry’s build-time auto-instrumentation for Android. Class-loading in the Android runtime poses significant challenges to instrumentation engineers looking for attach or injection points. Fortunately, developers can now use off-the-shelf OpenTelemetry tools to instrument their applications without the need to manually instrument or even alter their application code. We will cover the benefits of front-loading bytecode weaving to build time, the tools used by OpenTelemetry to accomplish this, and the types of rich client-side telemetry created through this approach.

Using Kubernetes as a multi-tenant platform is becoming a general concept. However, the fact that different logs are mixed on the same nodes makes it difficult. Learn how the Telemetry Controller (an abstract layer on top of the OTel Collector) helps you tackle the first problem. Routing from the edges provides isolation for the tenants and simplifies filtering and routing on a tenant level. Sandor will present different scenarios to do intelligent routing and how to build a smart aggregator on top of the Telemetry Controller.

Like many enterprises, we at Dynatrace deal with multiple delivery streams and various tooling to be used to develop, build, test and deliver our product to our customers. To better scale and have a resilient delivery practice, we regularly need to streamline the processes. From a requirement till the running solution, it typically involves many steps, often various people and different systems. Distributed knowledge makes it hard to transparently identify issues and slows things down. Observability in SDLC (Software Delivery Lifecycle) is our key enabler that gives us insights into resources, tools, techniques, staging, quality, rollout, etc. In this session learn how we automatically trace every delivery value flow, identify bottlenecks, wait times, lead times and other characteristics with the help of cloud native observability. Observability tells us where to start with optimization(s) and how to apply a continuous improvement cycle to our delivery chain.

In the course of several SaaS migrations to Otel and Prom-based exporters, the search for the perfect PromQL CPU Usage (and Mem Usage) query ended up with several candidates. This quick talk will walk through these queries, explain their differences, and offer the crowd a way to rate the queries and vote on which ones they think should be used. A quick search on Stack Overflow will also produce quite a few queries to use and prove the need for some crowdsourcing on what queries should be the standard. Let's let the people decide! Or at least let's understand the appropriateness of various queries versus other options; how container and various prometheus exporter metrics work; and talk about the challenges of observability migrations with regards to translating queries from difference vendors to Open Source telemetry. Queries include cadvisor metrics, kubestate metrics, and node exporter metrics -- which also probably adds to the chaos of why there are so many cpu/mem queries.

Reading logs from a file is a very simple concept to understand, but implementing it is a whole other story. This lightning talk will dive into the full process of how Fluent Bit file logging works: detecting file changes, reading new lines from files, tracking state, handling file rotations, and more. This talk will be a brief glimpse into just one of the many deep challenges of implementing efficient observability data collection.