Architecting a Machine Learning System for Risk

Abstraction: Airbnb's real-time fraud detection ML pipeline architecture

Key points:

• Architecture must balance two goals: fast/robust (near-realtime scoring under 10ms) and agile (rapid model iteration and deployment)
• Uses Openscoring — a Java REST service backed by JPMML — to serve PMML-encoded models; random forests chosen for appropriate precision-recall
• PMML format enables model portability across R, Python, Java; disadvantage is no support for bleeding-edge models or online learning
• Model-building pipeline evolved from R to Python/scikit-learn for better automation, feature extraction, deployment, and testing in a single script
• Key lesson: ground truth quality matters more than features or algorithm choice; inaccurate labels set a hard ceiling on precision-recall
• Recommendation: log everything to HDFS; data logged now may become essential for responding to future novel attack vectors

Connections: Airbnb · Scikit Learn · Machine Learning · Fraud Detection · Production ML Systems

Source: http://nerds.airbnb.com/architecting-machine-learning-system-risk/