PKCS#11 signing: argument pData was unexpectedly null on RHEL 9.6 with OpenSSL 3.2.2

[stanhu]

On RedHat Linux 9.6 (via Docker image registry.access.redhat.com/ubi9/ubi-minimal:9.6-1755695350 with OpenSSL 3.2.2, I see a warning message when using the PKCS#11 FIPS library to sign a binary with Google Cloud HSM:

I20250911 05:40:42.692146 140020334341184 logging.cc:185] returning 0x7 from C_Sign due to status INVALID_ARGUMENT: at bridge.cc:697: argument pData was unexpectedly null [type.googleapis.com/kmsp11.StatusDetails='CK_RV=0x7']

The binary seems to have been signed just fine, however.

This is the invocation of osslsigncode:

osslsigncode sign -h sha256 -pkcs11engine /usr/lib64/engines-3/pkcs11.so -pkcs11module /usr/local/lib/libkmsp11.so -certs chain.crt -key pkcs11:object=my-key  -in work/test.exe -out work/new.exe -ts http://ts.ssl.com

When I downgraded to RedHat Linux 8.10 with OpenSSL 1.1, I did not see this warning.

[tdbhacks]

Just to confirm, I see -in work/test.exee (with two 'e's) in the osslsigncode command you posted: I assume that's a typo in the issue description and not that it couldn't find the file, subsequently triggering the null pData argument? 🙂

I'd guess that kind of issue hopefully gets blocked in some layer above our library, without getting all the way down into the stack. Other than that, I'm not really sure, errors happening at the bridge conversion layer usually indicate issues with the inputs we're provided with, because we haven't really done any processing internally at that point. Our library is essentially just complaining that it couldn't find input data, so I'm also very surprised that you're observing a correct signature regardless of that error

[tdbhacks]

Side note: we have #28 open for openssl3 compatibility

[stanhu]

@tdbhacks Thanks, that was a typo.

I should have mentioned: it's possible this warning occurred because there was a destroyed key in the keyring:

I20250911 05:40:42.266561 140020334341184 object_loader.cc:75] INFO: version projects/my-project/locations/us-east1/keyRings/my-project/cryptoKeys/my-key/cryptoKeyVersions/1 is not loadable due to unsupported state DESTROYED
I20250911 05:40:42.692146 140020334341184 logging.cc:185] returning 0x7 from C_Sign due to status INVALID_ARGUMENT: at bridge.cc:697: argument pData was unexpectedly null [type.googleapis.com/kmsp11.StatusDetails='CK_RV=0x7']
Engine "pkcs11" set.

But I wasn't sure if this was related because the warning happens 0.4 seconds later, and it doesn't show up when I downgraded.