fix: disable AWS IoT SDK metrics to avoid awscrt _certificate_source crash - #71
Merged
Merged
Conversation
…crash
awsiot's mqtt_connection_builder defaults enable_metrics_collection to
True, so it builds an AWS IoT SDK metrics string and passes it to
awscrt.mqtt.Connection with disable_metrics=False. awscrt then reads
private ClientTlsContext internals to encode the feature list:
# awscrt/aws_iot_metrics.py
val = _certificate_source_metrics_value(tls_ctx._certificate_source)
_certificate_source was only added to ClientTlsContext in awscrt 0.35.0.
On installs where awscrt's modules are not all from the same version --
which is what several Home Assistant 2026.7 / Python 3.14 users are
hitting -- that attribute is missing and get_rest_devices dies with:
AttributeError: 'ClientTlsContext' object has no attribute
'_certificate_source'
The connection is never even attempted, so setup fails and the retry
loop hammers the Hatch login endpoint until it returns HTTP 429.
These metrics only report AWS SDK name/version/platform to AWS in the
CONNECT packet username; nothing in this library consumes them. Opting
out skips the offending code path entirely and leaves the connection
otherwise unchanged.
Fixes dahlb/ha_hatch#323
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the crash in dahlb/ha_hatch#323 (also reported independently at mmalolepszy/hon-revived#58, which isn't a Hatch integration — it just also builds an AWS IoT MQTT connection through
awsiot).Correction to an earlier version of this description: I originally wrote that affected machines have a mixed awscrt install on disk. That was wrong, and I've since disproved it by instrumenting an affected instance. The real cause is an in-process module skew. Details below, because it changes what the actual cure is.
The failure
On Home Assistant 2026.7 / Python 3.14,
get_rest_devicesdies before the MQTT connection is ever attempted:Setup then enters
setup_retry, and the retry loop keeps hittinghttps://data.hatchbaby.com/public/v1/loginuntil Hatch returns HTTP 429.Why it happens
awsiot.mqtt_connection_builderdefaultsenable_metrics_collectiontoTrue:Because we never pass that kwarg,
metricsis non-None, sodisable_metrics=Falseandawscrtgoes on to encode an IoT-SDK feature list. Doing so reads a private attribute of the TLS context —tls_ctx._certificate_source— which only exists inawscrt>= 0.35.0 (added in awslabs/aws-crt-python#747).The reason that attribute goes missing is that
awscrtgets upgraded in place underneath a running Home Assistant process.awsiotsdkpinsawscrtto an exact version (1.21.1 →awscrt==0.20.5, 1.28.0 →awscrt==0.31.1, 1.31.0 →awscrt==0.36.1), so installing this library's requirements at config-entry setup time can replace awscrt on disk while the interpreter already has part of it imported. Then:awscrt/__init__.pyandawscrt/io.pyare already insys.modulesfrom the old awscrt, so they are never re-imported. The residentClientTlsContextclass object has no_certificate_sourceslot.awscrt/mqtt.py,mqtt5.pyandaws_iot_metrics.pyare imported after the swap, so they load from the new awscrt — and the new metrics code reads a slot the old resident class doesn't have.Timeline from an affected instance (HAOS, HA 2026.7.4, Python 3.14.6), which is what convinced me:
uvreplaces awscrt's sources with 0.36.1 while HA is runningmqtt/mqtt5/aws_iot_metrics.pycwritten — imported fresh, so from 0.36.1.io.pyand__init__.pyget no new.pyc: still resident from the old versionAttributeErrorio/__init__.pycfinally recompiled → config entryloadedOn disk that machine is completely clean: awscrt 0.36.1 throughout, a single
.dist-info, oneawscrtdir onsys.path,ClientTlsContext.__slots__does contain_certificate_source, and every.pycis timestamp-validated and current. The corrupt state existed only in the live process. (The unqualified'ClientTlsContext'in the error does correctly indicate the loaded class lacks the slot — my error was inferring from that anything about the files on disk.)This also looks like the same underlying cause as ha_hatch#293 (
function takes exactly 17 arguments (18 given)) — same in-place swap, but hitting the Python↔native_awscrtboundary rather than the metrics code.What this change does, and what it doesn't
Passing
enable_metrics_collection=Falsestops us opting into the metrics path, so the private-attribute read never happens. Those metrics only append AWS SDK name/version/platform to the MQTT CONNECT username for AWS's own telemetry; nothing here consumes them and Hatch's broker doesn't care. The connection is otherwise unchanged, and it drops a needless dependency on awscrt private internals.To be straight about the limits: this is not a general cure for the module skew — restarting Home Assistant is. It removes this specific crash, and it is worth having regardless (we shouldn't be reaching into awscrt privates for telemetry we don't want), but a process that has half-old awscrt in memory can still fail at other boundaries, as #293 suggests. Users hitting either error should restart HA; this change means a subsequent in-place awscrt upgrade won't reintroduce this particular failure.
Pinning
awscrt/awsiotsdkwouldn't help here either: the disk is already consistent, and the damage is in the live interpreter.Verification
Simulating the resident-old-class condition (unsetting the slot in a
ClientTlsContext.__init__wrapper) against realawscrt0.36.1 +awsiotsdk1.31.0 on Python 3.14.6:Added
tests/test_util_bootstrap.py, which stubs the REST/AWS surface and assertsget_rest_devicespassesenable_metrics_collection=False. Confirmed it fails without the fix (KeyError: 'enable_metrics_collection') and passes with it.python -m unittest discover -s tests→ 25 tests OK.python -m ruff check .→ All checks passed.