Useful Unix commands for exploring data

Abstraction: Unix command-line tools for quick data file exploration and transformation

Key points:

• head -1 to check column names; wc -l to count records; head -N | tail -1 to inspect a specific row
• uniq -d finds duplicate records; uniq -u extracts unique records — requires adjacent duplicates (sort first)
• sort -n -t',' -k3 for numerical sort on a specific CSV column; -r for descending order
• awk -F "," '{print NF}' counts columns; NF = number of fields, NR = number of rows
• awk can split files by column value: '{print > $4"_output.csv"}' creates one file per unique value
• awk computes running aggregates: '{colSum+=$2} END {print colSum/NR}' for column averages

Connections: Data Exploration · Command Line Tools · Unix

Source: http://datavu.blogspot.com/2014/08/useful-unix-commands-for-exploring-data.html