GitHub - Factual/drake: Data workflow tool, like a "Make for data

data-pipelineworkflow-automationdata-engineeringbuild-tools

Abstraction: GNU Make-inspired data workflow tool with dependency resolution

Key points:

Drake organizes command execution around data and dependencies; resolves which commands to run (based on file timestamps) and in what order
Similar to GNU Make but designed for data workflows: supports multiple inputs/outputs, HDFS, Amazon S3, and incremental execution
Workflow file is called a Drakefile; targets built with drake mytarget or forced with drake +mytarget
Written in Clojure; installable via Homebrew on Mac or a single shell script; supports Clojure REPL usage and a plugin mechanism
Plugin system allows custom extensions; Vim syntax support and Python utilities available as community add-ons
Open-sourced by Factual in 2013; distributed under Eclipse Public License (same as Clojure)

Connections: Factual · Drake · Data Pipeline · Workflow Automation · Data Engineering