Don't use Hadoop - your data isn't that big
hadoopbig-datadata-engineeringpandassql
Abstraction: When Hadoop is overkill versus simpler data tools
Key points:
- Hadoop forces every computation into map/groupBy/reduce pattern — a straitjacket unsuitable for most analyses
- 600 MB of data was handled better with
pandas.read_csvthan Hadoop in a real job interview example - Pandas handles hundreds of MB efficiently; 16 GB RAM handles 10 GB CSV files in memory
- Postgres on a 2 TB drive handles 100 GB–2 TB datasets better than Hadoop
- Hadoop has no indexing — only full table scans — making it slower than SQL with proper indexes for most queries
- Hadoop only justified when data exceeds ~5 TB and requires distributed full-table scans
Connections: Hadoop · Pandas · Distributed Computing · Data Engineering
Source: http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html