avoid reusing the image buffer across frames of a different mode

[insaf021]

ImageFile.load_prepare() only allocates a new backing image when self._im is None, so a multi-frame plugin that re-opens on seek() reuses the previous frame's buffer. If the next frame keeps the same dimensions but widens the mode (an L frame followed by RGB, say), the tile still passes the setimage() bounds check, and the unpacker writes pixelsize bytes per pixel into a buffer sized for the narrower mode, overrunning each row.

With libgmalloc guard pages this faults deterministically in the decode loop:

Fatal Python error: Segmentation fault
  File "src/PIL/ImageFile.py", line 417 in load   # decoder.decode(b)

TiffImageFile.seek() already drops the buffer when its size or mode changes. DCX, MPO and MIC re-run another plugin's _open() on seek(), and PSD swaps the layer mode, none of them clearing _im.

Changes proposed in this pull request:

• Apply the same size/mode guard as TiffImageFile.seek to DcxImageFile.seek, MpoImageFile.seek, MicImageFile.seek and PsdImageFile.seek, so the backing image is reused only when its geometry still matches.
• Add regression tests for DCX, MPO and PSD seeking to a frame or layer with a different mode.

[radarhere]

Thanks very much.

However, it turns out that we've been preparing a similar change internally. I've created #10105. If you think that we've missed anything in our version of this, please let us know.

[insaf021]

Had a look through #10105. It covers the same four plugins (DCX, MPO, MIC, PSD) and centralising the guard in Image.Image.seek() is tidier than repeating it in each seek(). Nothing missing that I can see. Pulling it up also lets the old size-only check in JpegImagePlugin.SOF() go, since the MPO re-open path now gets size and mode compared together in one place. Looks solid.

[radarhere]

For the record, when I said "we've been preparing a similar change internally", that was code for "this is a security problem that's already been reported to us".

Please consider https://pillow.readthedocs.io/en/stable/handbook/security.html#security-reporting moving forward.