Diagnosing Memory "Leaks" in Python

Abstraction: Systematic approach to diagnosing Python process memory growth

Key points:

• Django's DEBUG mode stores every query in memory, mimicking a leak — always check this first
• Toolchain: htop for observation, GNU time -v for peak usage, resource module for inline measurement, objgraph for object counts, guppy/heapy for heap diffs
• GDB with gdb-heap can reveal allocations invisible to Python's GC (e.g., C-level fragmentation)
• Python's VM retains peak memory indefinitely even after GC; processes show "high watermark" behavior that is not a true leak — known as memory fragmentation
• Celery-specific fix: CELERYD_MAX_TASKS_PER_CHILD rolls worker processes; general fix is multiprocessing isolation via processify
• Best solution is using less memory; breaking work into smaller chunks avoids the watermark problem entirely

Connections: Celery · Django · Memory Management · Debugging · Python

Source: http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html