Writing An Hadoop MapReduce Program In Python

Abstraction: Tutorial for writing Hadoop MapReduce word-count jobs in Python via Streaming API

Key points:

• Uses Hadoop Streaming API to pass data between Python mapper/reducer via STDIN/STDOUT — no Jython/Java jar needed
• mapper.py reads lines from STDIN, emits <word>\t1 pairs; reducer.py sums counts exploiting Hadoop's pre-sort of map output by key
• Local testing pattern: cat data | mapper.py | sort -k1,1 | reducer.py before submitting to cluster
• Job submitted with hadoop jar hadoop-streaming.jar -mapper mapper.py -reducer reducer.py -input ... -output ...
• mapred.reduce.tasks can be set with -D flag; mapred.map.tasks is treated as a hint only
• Advanced version uses Python iterators/generators with itertools.groupby for cleaner reducer code

Connections: Hadoop · Mapreduce · Distributed Computing · Big Data

Source: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/