The Changing Data Science And Data Engineering Tooling Environment

data-sciencedata-engineeringrolesml-teamstooling

Abstraction: Distinguishing data scientist and data engineer roles, skills, and org structure

Key points:

80% of an AI project involves data preparation and engineering, yet organizations over-hire data scientists and under-hire data engineers
Data scientists: statistics/math background, hypothesis-driven, prefer clean data, typically use R; data engineers: programming/systems background (Java, Python, Scala), build ETL pipelines and distributed systems
The roles are not interchangeable; mixing them leads to inefficient code from scientists doing engineering, or incorrect conclusions from engineers doing science
Most orgs should have more data engineers than data scientists due to the volume of cleaning/moving work vs. modeling work
Data scientists should report to business/line-of-business units, not to the technical team
Tooling gap: most ML tools are actually engineering-centric with a thin science layer; business-centric data science tools (akin to Excel for analysis) are the next frontier

Connections: Jesse Anderson · Data Science · Data Engineering · ML Teams