7 Command-Line Tools for Data Science | Jeroen Janssens
command-linedata-scienceunix-toolscsvjson
Abstraction: Seven Unix command-line tools for OSEMN data science workflow
Key points:
- OSEMN framework: Obtain, Scrub, Explore, Model, iNterpret — command line excels at first three stages
- jq: sed-equivalent for JSON — filters, selects, and formats JSON from APIs
- json2csv: converts JSON to CSV for use with classic tools like cut, awk, grep
- csvkit: full suite for CSV (csvsort, csvlook, csvjoin, in2csv, csvgrep) with sqlite import support
- scrape + xml2json: XPath/CSS extraction from HTML piped to JSON for jq processing
- sample: outputs a percentage of stdin lines with optional delay and time limit — useful for pipeline debugging
- Rio: bash wrapper making R/Rscript part of Unix pipelines, supports ggplot2 PNG output via
-gflag
Connections: Jeroen Janssens · Csvkit · Data Engineering · Data Science
Source: http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html