Loading…
Tuesday November 12, 2024 5:15pm - 5:25pm MST
The interactive nature of Jupyter notebooks has made them indispensable tools for data scientists and AI researchers, facilitating exploratory data analysis, prototyping, and model development. However, managing the cost of resource-intensive computations at different stages of AI/ML lifecycle presents significant challenges. We leveraged Apache YuniKorn to design a resource management system tailored for notebook workloads, which incorporates fair sharing, user-specific policies and budget constraints to allocate computational resources efficiently while adapting for both data preparation and model training stages. And thanks to the extensibility of JupyterLab, we offer rich displays next to the Notebook enabling data scientists to introspect resource usage in real time. This session presents cost saving strategies for interactive development on Jupyter using Kubeflow for model training and Spark for data preparation with YuniKorn scheduler.
Speakers
avatar for Shravan Achar

Shravan Achar

Sr. Software Engineer, Apple
Shravan is a senior software engineer at Apple with a passion for open source technologies. With a background in Mathematics and Computer Science, their current interests include MLOps, Scheduling in AI and Jupyter Notebooks.
Tuesday November 12, 2024 5:15pm - 5:25pm MST
Salt Palace | Level 1 | Grand Ballroom A

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Share Modal

Share this link via

Or copy link