Distributed Computing for Data Scientists

distributed-computingdata-sciencescalingpython

Abstraction: Organizational and technical challenges of scaling Python data science to clusters

Key points:

Three core pain points for data scientists on clusters: software environments (pip/conda churn), resource sharing (bursty workloads vs. batch-oriented schedulers), and data access (S3/HDFS/Snowflake fragmentation)
Data science workloads are bursty — a scientist may need 1000 machines for minutes then idle for an hour; cluster schedulers designed for production jobs handle this poorly
Docker-image workflows designed for data engineering don't suit data scientists who change packages multiple times a day
When cluster access is hard, scientists download data samples locally, creating insecure, disparate copies across the organization
Production clusters are often politically gatekept from data science teams despite shared infrastructure
Written by the Coiled team; Coiled builds managed Dask clusters to address these friction points

Connections: Coiled · Distributed Computing · Data Science · Big Data