Pillow does not respect compiler library path

I have a setup with Python 3.9 installed in /usr/lib for historical reasons, and zlib and libjpeg installed /usr/lib64.

When I attempt to build Pillow, it will fail since it cannot find zlib/libjpeg, even if the compiler will happlly link to both libs without issue.

In my case, I could work around this by explicitly setting LIBRARY_PATH=/usr/lib64.

This issue for me seems to be due to two reasons:

pkg-config will strip -L linker flags that are part of the system library directory (/usr/lib64 in my case).
distutils sysconfig.get_config_var("LIBDIR") will return /usr/lib/, which will be set to self.compiler.library_dirs in setup.py.

This is almost related to #7634, but not really the same issue, since LIBDIR in my case is /usr/lib, which doesn't fix the issue.

I tried commenting out the dependency checks and hardcoding the library dirs manually in setup.py:

# for f in feature:
#   if not getattr(feature, f) and feature.require(f):
#      if f in ("jpeg", "zlib"):
            #raise RequiredDependencyException(f)
        #raise DependencyException(f)

# core library

libs = self.add_imaging_libs.split()
defs = []
feature.zlib="/usr/lib64"
libs.append(feature.zlib)
defs.append(("HAVE_LIBZ", None))

feature.jpeg="/usr/lib64"
libs.append(feature.jpeg)
defs.append(("HAVE_LIBJPEG", None))

With this, I could successfully build Pillow with jpeg and zlib support.

I'm not sure what a robust solution for this would be. Other build tools like autoconf works by attempting to compile a program to, for example, see if a library exists. This is maybe not suitable for this project, but maybe adding /usr/lib64 to the default library_dirs would make sense?

I'm slightly confused. In your first comment, you said

In my case, I could work around this by explicitly setting LIBRARY_PATH=/usr/lib64.

but then you put that aside and modified Pillow instead? Could you explain what was wrong with your initial workaround?

The modifications in the second comment were just to show that the dependency checks are insufficient.

It was enough to do this to get the build to work with zlib/jpeg:

libs = self.add_imaging_libs.split()
defs = []
feature.zlib="foo"
defs.append(("HAVE_LIBZ", None))

feature.jpeg="bar"
defs.append(("HAVE_LIBJPEG", None))

# if-statements below for jpeg and zlib are commented out

Could you explain what was wrong with your initial workaround?

There's not really anything wrong with it, I just think it's odd that the build works by commenting out some lines and setting some variables to junk. It would be cool if it just worked™ 😄, my compiler already knows to look in /usr/lib64.

Pillow used to explicitly search /usr/lib64 in some Linux environments, until #3245 switched to using ldconfig instead.

https://man7.org/linux/man-pages/man8/ldconfig.8.html

On some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries.

So it would seem unexpected that ldconfig isn't solving this automatically for you.

What operating system are you using?
How did you install the libraries?
What does ldconfig -p give you? Or, if you're on FreeBSD, ldconfig -r?

I'm using a custom Linux-based build system which is RedHat/Fedora based. My libraries are built/installed as RPM's in /usr/lib64.

Looks like we didn't have the file /etc/ld.so.cache, which seems to require ldconfig to be run once as root to generate. After running sudo ldconfig once, the build worked without issue.

Doesn't this mean that you need to run ldconfig as root once before every build?

It would seem the normal behaviour is to run ldconfig after a new library is installed.

https://unix.stackexchange.com/questions/256893/relationship-between-ldconfig-and-ld-so-cache

If I am installing a program, how does my computer know to use the new libraries that had been added? After apt-get install is ldconfig run?
...
Yes
...
apt-get and dpkg both invoke ldconfig to rebuild the cache.

https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

When you install a new version of a library, you install it in one of a few special directories and then run the program ldconfig

Ah I see, looks like Fedora/RHEL has some RPM macro magic to run ldconfig when shared libraries are installed.

So, would you consider this to be resolved then?

I think the issue still persists in some cross-compilation setups, but the workaround works good enough for me.

Navigation Menu

Repository navigation

Pillow does not respect compiler library path #8542

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Pillow does not respect compiler library path #8542

Description

Activity

CendioHalim commented on Nov 8, 2024

radarhere commented on Nov 9, 2024

CendioHalim commented on Nov 11, 2024

radarhere commented on Nov 11, 2024

CendioHalim commented on Nov 11, 2024

CendioHalim commented on Nov 11, 2024

radarhere commented on Nov 11, 2024

CendioHalim commented on Nov 14, 2024

radarhere commented on Nov 14, 2024

CendioHalim commented on Nov 20, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions