Why a Hedge Fund Built Its Own Database

Abstraction: Man Group's ArcticDB serverless DataFrame database on object storage

Key points:

• Man Group ($160B AUM, $6T/yr trading) needed a high-performance Python-native time-series database for quant research; built Arctic in 2011 on MongoDB, open sourced 2015, then rewrote core in C++ against S3/object stores as ArcticDB for 10x+ performance
• Key design: give up ACID "isolation" (coordinated multi-writer transactions) to avoid database servers entirely — all logic runs client-side against shared object storage (S3, flash)
• Supports wide DataFrames with 100K+ columns (one user has 1M columns) and ragged/sparse data — matches the shapes quants actually work with, not normalized SQL schemas
• Immutable versioned data structures: every write creates a new version; symbolic links atomically update the "current" pointer; full time-travel to any past version or timestamp
• Achieves 40 GB/s sustained throughput from local flash storage; used by Bloomberg (BQuant) and other finance firms; most popular open source tool is D-Tale (pandas visualizer)
• Research productivity — not low-latency execution — is the main competitive constraint in quant; market edges erode as data volume and competition grow

Connections: Man Group · Arcticdb · Bloomberg · Time Series Database · Columnar Storage · Serverless Architecture

Source: https://www.infoq.com/presentations/arcticdb/