CNCF-hosted Co-located Events North America 2024

Upstream, Argo Workflow’s CI uses Github Actions. In a true "eating our own dog food" moment, we embarked on a journey to migrate Argo Workflows CI to Argo Workflows itself on our local fork.

Our goals were:
- Boost Efficiency & Reliability: Streamline CI processes for faster development cycles.
- Fork Power: Enable quicker patch releases on our Argo Workflows fork.
- Real-World Inspiration: Provide a practical CI example for Argo Workflows users.

Dive deeper with us:
- Uncover the journey involved in migrating from GitHub Actions to Argo Workflows.
- Explore the challenges and solutions of running Kubernetes inside Kubernetes.
- Learn how we tackled various technical hurdles encountered during the migration.
- Discover the performance, cost, and reliability gains achieved through this self-hosted CI solution.

Managing large-scale batch workflows efficiently is critical for AI/ML workloads. Data preparation for training or fine tuning models can involve a large number of steps. These make for excellent Argo workflows. But Argo faces the etcd limitation of the 1.5MB object size. This limitation restricts the ability of Argo to run truly large-scale workflows. This talk will delve into the intricacies of this limitation and its impact on AI/ML workflows. We will illustrate with examples how this has been a non-deterministic and frustrating bottleneck for users. To address this challenge, Argo introduced a feature that circumvents the etcd object size restriction. By offloading the bulk of the workflow status to an RDBMS and only storing the reference in etcd, Argo maintains its scaling capabilities still adhering to Kubernetes' limitations. This talk will provide a comprehensive guide on configuring and utilizing the Argo offloading feature in AWS using Aurora Postgres RDS and EKS.

IT departments adopt hybrid cloud strategies to leverage the strengths of multiple public clouds like AWS, GCP, and Azure, while also maintaining private on-prem solutions to meet stringent security and regulatory compliance requirements. The challenge, however, lies in managing these diverse infrastructures consistently and efficiently. Complex hybrid environments leads to increased operational costs, security vulnerabilities, and a tangled web of provider-specific management tools that can overwhelm even the most seasoned IT teams. Support becomes fragmented, and the promise of a seamless, agile infrastructure seems out of reach. This session will explore how Argo Workflows can serve as the linchpin in a cloud-agnostic infrastructure strategy, enabling teams to manage resources across platforms and providers with unparalleled ease.

In this presentation we will share our Continuous Integrations (CI) pipeline extensions via Argo Workflows in a pure cloud-native way seamlessly integrated with our code versioning system. We will also describe a large-scale asynchronous event-based bus communication that has accelerated our CI pipelines but also increased their resiliency. Finally, we will cover some Continuous Delivery enhancements via ArgoCD and Argo Rollouts. Actually, even our own infrastructure is operated via GitOps model. This collection of open-source projects and internal tools provides a full cloud-native orchestration and gitops experience for Dev, QA and Performance teams. Our solution extends the existing GitLab CI/CD capabilities by providing the ability to orchestrate, build and manage service interdependencies and integrations with other systems and enhanced metrics-based service rollouts in a multi-region cloud architecture.

Get ready to supercharge your data science workflow development using the power of Hera, the versatile Python SDK for Argo Workflows.

• Python for Everything: Learn how Hera lets you focus on your business logic and seamlessly integrate it into Argo Workflows – all within your favorite Python environment.
• Effortless Argo with Hera: Use Hera to craft Argo Workflows with ease using simple Python code that handles common tasks such as template parameters, passing data, and fan-out.
• Beyond the Basics: We'll explore how Hera provides a base for your organization to build on, using its advanced capabilities, including pre-build hooks, that empower you to configure Hera for your organization's specific needs.
• Boost your CICD: Learn best practices to build Python workflows efficiently with Hera. Automate developer setups and recurring CICD tasks. 

By the end of this talk, you'll be equipped to supercharge your Argo Workflows with Hera to unlock a new level of automation and efficiency!

As organizations increasingly adopt GitOps and infrastructure-as-code, accurately visualizing manifest changes before they merge has become crucial. Mentally parsing Helm templates and Kustomize patches is too unreliable for catching configuration errors. Join us as we review the current landscape of tools and methods used for visualizing code changes in Argo CD, highlight their limitations, and introduce a new method that leverages ephemeral clusters and Argo CD to render accurate diffs of Helm Charts and Kustomize overlays directly on pull requests. The presentation showcases a tool illustrating this new approach and discusses its overall design. We will demonstrate how the approach can be seamlessly integrated into CI/CD pipelines to prevent deployment errors and streamline code reviews, all without access to live infrastructure. Finally, we'll conclude with an honest assessment of the method's capabilities and limitations and discuss potential areas for future development.

Atlan is a collaborative workspace for data teams that offers functionality like metadata cataloging and data lineage amongst others. Atlan provides connector integrations which ingest metadata from various data sources. As the data estate volume hit a massive scale, the platform encountered performance drags with ETL pipelines impacting resiliency, processing runtimes and efficiency. The existing architecture suffered pipeline failures encompassing computation and storage exhaustion, and parallel and concurrent processing pit-falls with troubling spikes in workflow failure rates. In this talk, Satabrata and Nishchith will share how they leveraged Argo’s parallelization techniques with robust re-try mechanisms and effective artifactory loading to ingest 100 Million assets achieving a 450% reduction in processing time. This improvisation also helped them process 3 Million SQL Queries in just 2 hours reducing overall pipeline runtime by 50% and having Argo-powered horizontal scale-out.

ArgoCD has the capability to manage more then one cluster, the question is how to securely connect to those remote clusters? This session will explore how to securely connect your remote clusters, regardless of if they're cloud managed or on-prem, using ArgoCD's own native Kubernetes identity through a token exchange to get an identity for that remote cluster. We'll start with the challenge of connecting to remote clusters securely, detail how token exchange works, then walk through updating the ArgoCD container with custom tools, creating Secrets to represent remote clusters, and ApplicationSets to generate the Application without any static tokens. The session will demo management of cloud hosted clusters, on-prem clusters, and clusters that support Kubernetes' beta of AuthenticationConfiguration in 1.30. By the end of this session you'll see where the configuration points are in ArgoCD to secure your GitOps infrastructure without relying on a single cloud provider's IAM.

Argo Rollouts makes Progressive Delivery easy to adopt, but some times things do not work as expected. Are the steps correctly set? are the analysis metrics right for the workloads? At Adobe Experience Manager we deploy over 10k customer services to Kubernetes. Changes can occur multiple times per day both internal and from code. A new feature can work fine for 99% of customers but still affect the other 1%, and detecting this just from tests is costly. Enter Argo Rollouts, which allows deploying new versions to a subset of users before rolling them to the totality of the users, and rolling them back if not matching some key metrics, using techniques like canary deployments. We will show our learnings deploying Argo Rollouts to manage over 10k workloads using canaries, how do we balance speed and safety for our customers, and some of the issues that we have faced when adopting it.

In modern cloud-native environments, GitOps practitioners face a significant challenge: managing dynamically generated resource values across Infrastructure as Code (IaC) tools like Terraform and Kubernetes manifests. This disconnect often leads to manual interventions, hard-coded values, and cumbersome update processes that compromise the efficiency and security of GitOps workflows. This talk introduces the Terraform Bridge, an innovative solution to seamlessly integrate dynamically generated cloud resource attributes with Kubernetes resources. We present a novel controller that leverages Terraform's output feature to automatically update Kubernetes objects such as ConfigMaps and Deployments. This approach eliminates the need for manual updates and hard-coding of sensitive or randomly generated values, thereby enhancing both security and automation in GitOps pipelines.

Are you new to Argo Events and trying to get your head around how everything works? Join us for an in-depth exploration of the Argo Events architecture, designed to transform you from a novice to a seasoned Argo Events maestro. In this session, we'll embark on a journey through the workings of Argo Events, unraveling the mysteries of the event bus, event source and sensors. We'll delve into their roles and responsibilities, and understand how they go about wrangling events within your Kubernetes cluster. This is the talk that we wished we had heard when we started on our Argo Events journey. Whether you're a seasoned Kubernetes practitioner or a newcomer to the Argo Events realm, this session will provide you with invaluable insights that will elevate your understanding and mastery of this powerful eventing engine.

Argo CD has a lot of settings and this has only proliferated in each new release of Argo CD. Some settings and values are universally useful and should always be enabled while others are only needed for very specific niche use cases. In this talk we will review many of these settings and values using the always fun and informative concept of the Tier List. Tiers for the settings/values include “Make this the default”, “Almost always useful”, “Good in places”, “Meh?” and finally “Nobody needs this”. Audience participation will be encouraged in this session with audience members helping to rank settings and values by show of hands.

Argo Workflows, a K-native DAG-style workflow engine for Kubernetes, is a powerful tool for orchestrating complex workflows. Additionally Java is the most popular language for developers in complex organisations. However, using Argo with the Argo Java SDK can be challenging due to limited examples and unclear documentation. This session aims to provide a comprehensive, step-by-step guide on interacting with Argo Workflows using Fabric8’s Custom Resource Definitions (CRD) in Java. Argo Workflows are implemented as 'Custom Resources' in Kubernetes. By leveraging the Fabric8 library’s Custom Resource Definitions, developers can seamlessly interact with Argo Workflows through the Kubernetes API. Join us to learn how to harness the power of Argo Workflows in Java, simplifying your workflow management on Kubernetes.