CNCF-hosted Co-located Events North America 2024

Atlan is a collaborative workspace for data teams that offers functionality like metadata cataloging and data lineage amongst others. Atlan provides connector integrations which ingest metadata from various data sources. As the data estate volume hit a massive scale, the platform encountered performance drags with ETL pipelines impacting resiliency, processing runtimes and efficiency. The existing architecture suffered pipeline failures encompassing computation and storage exhaustion, and parallel and concurrent processing pit-falls with troubling spikes in workflow failure rates. In this talk, Satabrata and Nishchith will share how they leveraged Argo’s parallelization techniques with robust re-try mechanisms and effective artifactory loading to ingest 100 Million assets achieving a 450% reduction in processing time. This improvisation also helped them process 3 Million SQL Queries in just 2 hours reducing overall pipeline runtime by 50% and having Argo-powered horizontal scale-out.