CNCF-hosted Co-located Events North America 2024

Managing large-scale batch workflows efficiently is critical for AI/ML workloads. Data preparation for training or fine tuning models can involve a large number of steps. These make for excellent Argo workflows. But Argo faces the etcd limitation of the 1.5MB object size. This limitation restricts the ability of Argo to run truly large-scale workflows. This talk will delve into the intricacies of this limitation and its impact on AI/ML workflows. We will illustrate with examples how this has been a non-deterministic and frustrating bottleneck for users. To address this challenge, Argo introduced a feature that circumvents the etcd object size restriction. By offloading the bulk of the workflow status to an RDBMS and only storing the reference in etcd, Argo maintains its scaling capabilities still adhering to Kubernetes' limitations. This talk will provide a comprehensive guide on configuring and utilizing the Argo offloading feature in AWS using Aurora Postgres RDS and EKS.

Get ready to supercharge your data science workflow development using the power of Hera, the versatile Python SDK for Argo Workflows.

• Python for Everything: Learn how Hera lets you focus on your business logic and seamlessly integrate it into Argo Workflows – all within your favorite Python environment.
• Effortless Argo with Hera: Use Hera to craft Argo Workflows with ease using simple Python code that handles common tasks such as template parameters, passing data, and fan-out.
• Beyond the Basics: We'll explore how Hera provides a base for your organization to build on, using its advanced capabilities, including pre-build hooks, that empower you to configure Hera for your organization's specific needs.
• Boost your CICD: Learn best practices to build Python workflows efficiently with Hera. Automate developer setups and recurring CICD tasks. 

By the end of this talk, you'll be equipped to supercharge your Argo Workflows with Hera to unlock a new level of automation and efficiency!

Atlan is a collaborative workspace for data teams that offers functionality like metadata cataloging and data lineage amongst others. Atlan provides connector integrations which ingest metadata from various data sources. As the data estate volume hit a massive scale, the platform encountered performance drags with ETL pipelines impacting resiliency, processing runtimes and efficiency. The existing architecture suffered pipeline failures encompassing computation and storage exhaustion, and parallel and concurrent processing pit-falls with troubling spikes in workflow failure rates. In this talk, Satabrata and Nishchith will share how they leveraged Argo’s parallelization techniques with robust re-try mechanisms and effective artifactory loading to ingest 100 Million assets achieving a 450% reduction in processing time. This improvisation also helped them process 3 Million SQL Queries in just 2 hours reducing overall pipeline runtime by 50% and having Argo-powered horizontal scale-out.

Are you new to Argo Events and trying to get your head around how everything works? Join us for an in-depth exploration of the Argo Events architecture, designed to transform you from a novice to a seasoned Argo Events maestro. In this session, we'll embark on a journey through the workings of Argo Events, unraveling the mysteries of the event bus, event source and sensors. We'll delve into their roles and responsibilities, and understand how they go about wrangling events within your Kubernetes cluster. This is the talk that we wished we had heard when we started on our Argo Events journey. Whether you're a seasoned Kubernetes practitioner or a newcomer to the Argo Events realm, this session will provide you with invaluable insights that will elevate your understanding and mastery of this powerful eventing engine.