91视频

Python鈥檚 garbage collector handles most memory management automatically 鈥� until it doesn鈥檛. In long-running production services, memory usage creeps up over hours or days, OOM kills start hitting your containers, and tracemalloc tells you 鈥測eah, there are allocations鈥� without giving you the full picture. We needed something better.

So we built a C-based memory profiler that intercepts allocations below the Python interpreter, uses sampling to keep overhead minimal, attributes every allocation back to the originating Python function, and ships the data to Elasticsearch in real-time. No code changes. No restarts. Just set an environment variable:

This post covers how we built it, the Linux dynamic linking tricks that make it possible, what we actually found in production, and why this approach catches leaks that pure-Python profilers miss.

The Problem: Python Memory Leaks Are Harder Than They Look

Python鈥檚 reference counting and cyclic garbage collector are great for typical application code. But in production services 鈥� think microservices processing millions of tracking updates per day 鈥� memory leaks happen for reasons invisible to Python-level tools:

• C extension leaks: Libraries like lxml, numpy, or database drivers allocate memory through C鈥檚 malloc. Python鈥檚 tracemalloc can鈥檛 see these at all.

• Reference cycles involving __del__: Objects with destructors can鈥檛 be collected by the cyclic GC, silently leaking.

• Arena fragmentation: CPython鈥檚 memory allocator (pymalloc) uses arenas. Even when objects are freed, the arena may not be returned to the OS.

• Hidden allocations in standard library code: Things like warnings.warn() or time.strftime() that allocate strings on every call 鈥� invisible unless you鈥檙e watching at the C level.

Why Not tracemalloc or Memray?

We evaluated every major option. Here鈥檚 the comparison that drove our decision:

tracemalloc: High Memory Cost, Blind to C

It eats memory. tracemalloc stores a full Python traceback for every live allocation. We saw 200鈥�400MB of overhead on services already tight on memory. You鈥檙e trying to diagnose an OOM, and the profiler makes it happen faster.

It can鈥檛 see C malloc. When lxml calls xmlMalloc, when your database driver allocates result buffers in C 鈥� tracemalloc sees none of it. In our services, ~30鈥�40% of memory was allocated through C extensions.

It requires code changes. You need to call tracemalloc.start(nframes) and decide the frame depth upfront.

Memray: Not Built for Production Monitoring

叠濒辞辞尘产别谤驳鈥檚 is impressive for debugging. But it needs PYTHONMALLOC=malloc for accurate leak detection (changing program behavior), generates gigabyte trace files, and memray attach can crash your process. There鈥檚 no way to stream data to Elasticsearch or set up Kibana dashboards.

Understanding Dynamic Linking: The Foundation

To understand why our profiler works, you need to understand how programs find their functions at runtime.

Static vs Dynamic: A Live Demo

Take a simple C program and compile it two ways:

The static binary is 70x larger because it has malloc baked in at a fixed address. The dynamic binary just says 鈥淚 need malloc from somewhere.鈥�

Static vs Dynamic Linking

How Python鈥檚 Memory Functions Are Linked

Where Python鈥檚 allocation functions live determines what we can intercept.

Reading Python鈥檚 Symbol Table

U (undefined) = always interceptable via LD_PRELOAD. T (defined) = depends on how the binary was built.

Reading Python鈥檚 Symbol Table

The Two Python Builds

Python can be built with libpython statically baked in (Debian/Ubuntu) or dynamically linked as a shared library (Fedora, custom builds). This changes interception behavior:

Build 1: Static libpython (Debian/Ubuntu aarch64)

Internal calls are hardcoded direct jumps 鈥� the dynamic linker is never involved.

Build 2: Dynamic libpython (custom build / Fedora)

All calls to PyMem_Malloc go through the PLT 鈥� LD_PRELOAD intercepts them.

Two Python Builds 鈥� Two Interception Behaviors

Proving It with LD_DEBUG

Watch the dynamic linker resolve symbols with the profiler loaded:

Two lines 鈥� the complete chain:

1. libpython calls PyMem_Malloc 鈫� lands in trackleak.so (our profiler intercepts it)

2. trackleak.so calls the real one via dlsym(RTLD_NEXT) 鈫� lands in libpython (we pass it through)

Why the Profiler Works on Both Builds

The malloc symbol is U (undefined) on every build. It always goes through the PLT. Even when PyMem_Malloc is baked into the binary, it eventually calls malloc when it needs a new arena 鈥� and that call goes through the dynamic linker.

On static-libpython builds, you catch everything through the malloc safety net. On dynamic-libpython builds, you catch both levels. Either way, nothing escapes.

The Allocation Flow: Normal vs Intercepted

Normal flow:

With LD_PRELOAD=trackleak.so 鈥� double interception:

Python Memory Allocation Flow 鈥� Normal vs Intercepted

Symbol Interposition: How LD_PRELOAD Works

The dynamic linker resolves function symbols at runtime by searching shared libraries in order. LD_PRELOAD lets you jump the queue. dlsym(RTLD_NEXT, "malloc") means 鈥渟kip me, find the next definition鈥� 鈥� giving us a pointer to libc鈥檚 real malloc.

A Minimal Example

Every malloc call 鈥� from CPython鈥檚 interpreter loop, from NumPy鈥檚 array allocations, from OpenSSL inside your HTTP client 鈥� now prints to stderr. No recompilation, no code changes.

How LD_PRELOAD Intercepts malloc

Architecture of the Profiler

Profiler Architecture 鈥� From malloc to Elasticsearch

Functions We Intercept

We intercept six functions total:

On dynamic-libpython builds, all six are intercepted. On static-libpython builds, malloc/free are always intercepted, and PyMem_* are intercepted only from C extensions.

Avoiding Infinite Recursion

Our profiler uses functions that call malloc. We use a thread-local guard (__thread int in_malloc) to detect re-entry:

Walking the Python Stack from C

From inside our C malloc wrapper, we acquire the GIL, walk the frame chain, extract function name / file path / line number, then release the GIL:

Why the GIL is necessary: This isn鈥檛 about protecting your globals 鈥� it鈥檚 about protecting CPython鈥檚 internal heap objects. PyUnicode_AsUTF8() may write to the object (lazy UTF-8 cache creation). PyFrame_GetCode() does Py_INCREF (a refcount write). Frames can be freed by other threads at any moment. Without the GIL: corrupted refcounts, dangling pointers, segfaults.

Leak Detection Through Retention Rate

The key metric:

On malloc: sample 1-in-50 and track the pointer in a hash table.
On free: look up the pointer 鈥� if it was sampled, credit the freed bytes to the originating function.

High retention = memory leak candidate. A function with 95% retention is allocating memory that nobody is freeing.

The pymalloc Blind Spot

CPython doesn鈥檛 call malloc for every small object. It grabs 256KB arenas and carves them internally. We filter for allocations above 500 bytes 鈥� the leaks that matter (large buffers, HTTP bodies, deserialized data) bypass pymalloc and hit malloc directly.

Why pymalloc Arenas Are Opaque to malloc-level Profilers

What We Found in Production

On our tracking and shipment update services 鈥� high-throughput consumers processing millions of messages 鈥� we found two memory leaks hiding in plain sight.

Bug #1: Django鈥檚 warnings.warn() on Every Database Write

Every database write with a naive datetime triggered warnings.warn(). Under the hood, that builds formatted strings, creates WarningMessage objects, and walks the entire call stack. Dozens of string allocations per call, thousands of times per second.

Fix: Comment out the warning. We know our datetimes are naive 鈥� no need for Python to warn us on every single database write.

Bug #2: Python鈥檚 Logging formatTime on Every Log Line

Two string allocations per log line via time.strftime(). Millions of string objects per hour on a heavily-logging shipment tracking service.

Fix: Return the epoch timestamp as a string and parse it in the ELK pipeline instead. One tiny allocation instead of two formatted strings.

How We Found These

Both bugs share a pattern: death by a thousand cuts. Each individual allocation is tiny. No single call leaks memory. But at thousands of calls per second, allocations accumulate faster than the garbage collector can clean them up.

The profiler鈥檚 Kibana dashboards showed these functions with high allocation volume and elevated retention rates. Neither would have been caught by code review, tracemalloc, or Memray trace analysis.

Deployment

One environment variable:

Auto-detects NOMAD_JOB_NAME for tagging. 10-second startup delay to skip noisy initialization. Ships to Elasticsearch every 5 minutes via async bulk API. ~3鈥�5% CPU overhead. Kibana dashboards show top allocators, retention trends, and per-service memory profiles.

Lessons Learned

• Sampling is non-negotiable. Our early prototype tracked every allocation. It worked in testing and fell over under 10K req/s. 1-in-50 gives valid retention rates at <5% overhead.

• Hash table resets beat resizes. When the table fills up, wipe it clean. A crash in the profiler is worse than losing data.

• GIL acquisition from C is tricky. Calling PyGILState_Ensure() from inside a malloc wrapper that might be called from any thread requires a recursion guard.

• Know your Python build. Whether PyMem_Malloc is interceptable depends on static vs dynamic libpython. Use ldd $(which python3) to check. Either way, malloc interception is your safety net.

Wrapping Up

The code is straightforward C 鈥� LD_PRELOAD, dlsym, and the Python C API. One .so file, one environment variable. The bugs it found 鈥� warnings.warn() firing on every DB write, time.strftime() allocating on every log line 鈥� were invisible to every other tool we tried.

Sometimes the leak isn鈥檛 in your code. It鈥檚 in the code your code calls, at a layer below where Python can see.

馃敆 TrackLeak is open source!

Get the code, file issues, and contribute: