Fixing Bugs in CPython #1: A Use-After-Free in Context.__eq__

How comparing two contextvars.Context objects could crash the Python interpreter, and how the fix works. First in a series for anyone who wants to understand the CPython codebase or make their first contribution to it.
This post is part of a series where I walk through bugs I’ve fixed in CPython, from crash to merged PR. If something is unclear or there’s a topic you’d like covered, my inbox is open: hello at abdull dot dev.
A while back I was browsing the CPython issue tracker looking for my next bug when I found gh-142829: a report that comparing two Context objects can crash the interpreter. The report came with AddressSanitizer traces showing the crash on every maintained version of Python, from 3.9 to a 3.15 alpha.
Crash reports like this are a good way into the codebase. The reproducer is small, the failure is unambiguous, and the fix teaches you a corner of the interpreter you’d never have read otherwise. This one taught me how contextvars stores its data. Let’s walk through it together, from the crash to the merged fix.
First, what even is a context variable?
contextvars was added in Python 3.7 PEP 567. A ContextVar is a variable whose value depends on the current execution context: set it inside one async task and other tasks don’t see the change. This is how frameworks like FastAPI and Starlette track request-scoped state without passing it through every function call. The module docs cover the details.
A Context is a snapshot of all such variables. You can run code inside one with ctx.run(fn), and you can compare two contexts with ==. That comparison is where our bug lives.
The bug
The bug reproducer is an eleven lines of plain Python:
import contextvars
var = contextvars.ContextVar("v")
ctx1 = contextvars.Context()
ctx2 = contextvars.Context()
class Boom:
def __eq__(self, other):
ctx1.run(lambda: var.set(object()))
return True
ctx1.run(var.set, Boom())
ctx2.run(var.set, object())
ctx1 == ctx2
Run this against a build with AddressSanitizer and the interpreter dies inside ctx1 == ctx2 with a use-after-free, meaning some code read memory that had already been freed. In a normal build the corruption often goes unnoticed, or crashes somewhere unrelated later on, which is how a bug like this survives for years.
So what can possibly go wrong inside an innocent-looking ==? To answer that, we need one piece of background.
How Context stores its data
Contexts need cheap copies. Entering ctx.run() or calling ctx.copy() shouldn’t cost O(N), so a plain dict won’t cut it. Instead, CPython keeps a context’s variables in a HAMT (Hash Array Mapped Trie), an immutable tree structure. Immutable is the important part: var.set() never modifies the tree in place. It builds a new tree that shares most of its nodes with the old one, and the Context swaps its pointer over to the new tree. If you want to go deeper, this YouTube video is a good visual explanation, and the comment at the top of Python/hamt.c walks through the design with diagrams.
So far, we know:
- A
Contextwraps one of these trees. var.set()swaps in a new tree and lets go of the old one.
So why does it crash?
ctx1 == ctx2 ends up in _PyHamt_Eq() in hamt.c. Stripped down, it looks like this:
hamt_iterator_init(&iter, v->h_root);
do {
iter_res = hamt_iterator_next(&iter, &v_key, &v_val);
if (iter_res == I_ITEM) {
find_res = hamt_find(w, v_key, &w_val);
...
case F_FOUND: {
int cmp = PyObject_RichCompareBool(v_val, w_val, Py_EQ);
In words: walk every key/value pair in the first tree, look each key up in the second tree, and when it’s found, compare the two values. Reasonable enough. But two things in this loop deserve a closer look.
- Everything here is borrowed: Quick detour, because this idea explains a whole family of CPython bugs. Every Python object carries a reference count:
Py_INCREFadds one,Py_DECREFremoves one, and when the count hits zero the object is freed. An owned reference keeps the object alive. A borrowed one is only visiting: the object survives only as long as somebody else’s reference does. In the loop above, the iterator points into the tree’s internal nodes, andv_key,v_val,w_valare all borrowed. Nothing in the function keeps the trees alive. - The loop calls back into Python: PyObject_RichCompareBool(
v_val,w_val, Py_EQ) runs the value’s__eq__method. That’s arbitrary Python code, and it can do anything, including destroying the objects this function is borrowing.
Now watch the reproducer again with those two facts in mind:
_PyHamt_Eqstarts walkingctx1’s tree.- It reaches the
Boomvalue and callsBoom.__eq__. __eq__callsctx1.run(…), and inside thatrun, the current context isctx1itself.- The
var.set(…)inside does the following (incontextvar_set()inPython/context.c):
PyContext *ctx = context_get(); // the current context: ctx1
PyHamtObject *new_vars = _PyHamt_Assoc(
context_get_current_vars(ctx), (PyObject *)var, val);
context_set_vars(ctx, new_vars); // swaps in the new tree
- The swap drops
ctx1’s reference to the old tree. Since_PyHamt_Eqonly borrowed it, that was the last reference. The old root and the nodes on the replaced path get freed. (The rest survive, shared with the new tree.) __eq__returnsTrue, the loop asks the iterator for the next item… and the iterator is standing on freed memory. AddressSanitizer reports a use-after-free, and now we know exactly whose memory it was.
The comparison/check can’t rely on its caller to keep the trees alive, because the code being compared can swap the caller’s reference away mid-comparison.
The fix
The patch is about 30 lines, and it comes down to one rule: across any call that can run Python code, hold owned references, not borrowed ones.
- At the top of the function,
Py_INCREFboth trees. The old tree now survives even ifctx1swaps it mid-comparison. The comparison runs against a snapshot, which is the only sensible meaning of==when mutation can happen during it. - Around
PyObject_RichCompareBool,Py_INCREFthe key and both values, andPy_DECREFthem right after. Whatever__eq__does, the pointers being compared stay valid. - Every exit path goes through a single
done:label that drops the two tree references. Thisgotocleanup pattern shows up all over the C codebase once a function has more than one way out.
Plus regression tests and a NEWS entry, which every CPython fix needs.
The PR was later merged into main in January, 26. The backport to 3.14 went through automatically.
If you want to try something like this yourself: the issue tracker has crashes labeled type-crash with reproducers attached (sometimes), and the devguide walks you through building CPython with sanitizers. That’s how I started, and it gets you to a first real bug faster than reading the codebase.
Next episode: a race condition in threading, where Thread.is_alive() could report a thread as finished while the OS thread was still running.
References
- Issue: gh-142829
- Fix: PR #142905
- Source:
Python/hamt.c,Python/context.c - PEP 567
- Contextvars Docs
- CPython DevGuide