ImageGrab.grab() on Windows leaks a GDI bitmap on every failed grab

[Ptozin]

What did you do?

Called ImageGrab.grab() repeatedly on Windows while no interactive desktop was available. For example, an RDP session was disconnected without returning it to the console, or the process ran in session 0 over SSH. Every call raised OSError: screen grab failed, and the application caught the error and retried.

Minimal reproducer (run in a process with no interactive desktop, e.g. over SSH):

import ctypes
from PIL import ImageGrab

def gdi_objects():
    return ctypes.windll.user32.GetGuiResources(ctypes.windll.kernel32.GetCurrentProcess(), 0)

before = gdi_objects()
for _ in range(50):
    try:
        ImageGrab.grab()
    except OSError:
        pass
print(gdi_objects() - before)

What did you expect to happen?

Each failed call raises OSError and releases every GDI object it created, so the printed number is 0.

What actually happened?

It prints 50: one GDI object leaks per failed call.

python 3.12.10 64bit pillow 11.3.0 session 0 Windows-2022Server-10.0.20348-SP0
calls=50 failed=50 err='screen grab failed'
gdi_objects 0 -> 50 (+50, 1.00/fail)

In production this ran every ~10 s on VMs whose RDP session had been disconnected. Each leaked object is a full-screen bitmap (about 8 MB at 1920x1080). The system commit charge grew until the VMs ran out of memory: about 5 GiB in about 1 hour on an 8 GB VM, and about 12 GiB in about 2.5 hours on a 16 GB VM. Windows then reported "LogonUI.exe: could not allocate additional memory". grab(bbox=...) captures the full screen and crops afterwards, so a small bbox does not reduce the leak.

Cause

In src/display.c, PyImaging_GrabScreenWin32 creates the bitmap with CreateCompatibleBitmap but releases it with DeleteObject(bitmap) only on the success path. Every goto error after the bitmap exists skips that call. That covers the SelectObject, BitBlt and GetDIBits failures. DeleteDC(screen_copy) does not delete the bitmap that is selected into it.

error:
    PyErr_SetString(PyExc_OSError, "screen grab failed");

    DeleteDC(screen_copy);
    if (screens == -1) {
        ReleaseDC(wnd, screen);
    } else {
        DeleteDC(screen);
    }

    return NULL;

Related gaps in the same function, still on main:

• If PyBytes_FromStringAndSize fails, the function returns NULL without releasing bitmap, screen_copy or screen.
• On the GetDIBits failure path, buffer is not released.
• The width == -1 path jumps to error before bitmap is assigned, so a fix that deletes bitmap in error: must initialise it to NULL.

This was noted in passing in #5536 ("The bitmap is also not deleted on error cases"), but that report was about something else, so it was never fixed.

Suggested fix

Initialise bitmap = NULL and buffer = NULL. In error:, add if (bitmap) { DeleteObject(bitmap); } and Py_XDECREF(buffer);. Send the PyBytes_FromStringAndSize failure through the same cleanup, without replacing its MemoryError with OSError.

What are your OS, Python and Pillow versions?

• OS: Windows Server 2022 Datacenter, 10.0.20348 (GCE VM)
• Python: 3.12.10, 64-bit
• Pillow: 11.3.0 (also seen with 11.2.1; the same code is on main)

[radarhere]

I've created #10094 with your suggested fix.

[radarhere]

Be aware that any fixes we release will be part of future versions of Pillow. I don't know why you are still on 11.3.0, but in the best case scenario, this will be part of Pillow 13.0.0.