Adam Drake — Command Line Tools 235x Faster Than Hadoop

Abstraction: Unix command-line tools outperform Hadoop cluster for data processing tasks

Key points:

• Standard Unix command-line tools (awk, grep, sort, etc.) can process data 235x faster than a Hadoop cluster for certain workloads
• Hadoop overhead (job scheduling, HDFS I/O, JVM startup) dominates for datasets that fit in local disk or memory
• Single-machine processing with optimized CLI tools is appropriate before reaching true big-data scale
• Snapshot captured only the author's current bio page; original benchmark article content was not preserved

Connections: Data Engineering · High Performance Computing · Unix Tools

Source: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html