Skip to content

fix: disable AWS IoT SDK metrics to avoid awscrt _certificate_source crash - #71

Merged
dahlb merged 1 commit into
dahlb:mainfrom
chasebolt:fix/disable-aws-iot-sdk-metrics
Jul 28, 2026
Merged

fix: disable AWS IoT SDK metrics to avoid awscrt _certificate_source crash#71
dahlb merged 1 commit into
dahlb:mainfrom
chasebolt:fix/disable-aws-iot-sdk-metrics

Conversation

@chasebolt

@chasebolt chasebolt commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Addresses the crash in dahlb/ha_hatch#323 (also reported independently at mmalolepszy/hon-revived#58, which isn't a Hatch integration — it just also builds an AWS IoT MQTT connection through awsiot).

Correction to an earlier version of this description: I originally wrote that affected machines have a mixed awscrt install on disk. That was wrong, and I've since disproved it by instrumenting an affected instance. The real cause is an in-process module skew. Details below, because it changes what the actual cure is.

The failure

On Home Assistant 2026.7 / Python 3.14, get_rest_devices dies before the MQTT connection is ever attempted:

File "hatch_rest_api/util_bootstrap.py", in get_rest_devices
File "awscrt/mqtt.py", line 421, in __init__
    self._metrics = _create_metrics_mqtt3(metrics, self.proxy_options, self.client.tls_ctx)
File "awscrt/aws_iot_metrics.py", line 352, in _get_encoded_feature_list_mqtt3
    val = _certificate_source_metrics_value(tls_ctx._certificate_source)
AttributeError: 'ClientTlsContext' object has no attribute '_certificate_source'

Setup then enters setup_retry, and the retry loop keeps hitting https://data.hatchbaby.com/public/v1/login until Hatch returns HTTP 429.

Why it happens

awsiot.mqtt_connection_builder defaults enable_metrics_collection to True:

# awsiot/mqtt_connection_builder.py
metrics = _build_sdk_metrics() if _get(kwargs, 'enable_metrics_collection', True) else None
...
disable_metrics=metrics is None,
metrics=metrics

Because we never pass that kwarg, metrics is non-None, so disable_metrics=False and awscrt goes on to encode an IoT-SDK feature list. Doing so reads a private attribute of the TLS context — tls_ctx._certificate_source — which only exists in awscrt >= 0.35.0 (added in awslabs/aws-crt-python#747).

The reason that attribute goes missing is that awscrt gets upgraded in place underneath a running Home Assistant process. awsiotsdk pins awscrt to an exact version (1.21.1 → awscrt==0.20.5, 1.28.0 → awscrt==0.31.1, 1.31.0 → awscrt==0.36.1), so installing this library's requirements at config-entry setup time can replace awscrt on disk while the interpreter already has part of it imported. Then:

  • awscrt/__init__.py and awscrt/io.py are already in sys.modules from the old awscrt, so they are never re-imported. The resident ClientTlsContext class object has no _certificate_source slot.
  • awscrt/mqtt.py, mqtt5.py and aws_iot_metrics.py are imported after the swap, so they load from the new awscrt — and the new metrics code reads a slot the old resident class doesn't have.

Timeline from an affected instance (HAOS, HA 2026.7.4, Python 3.14.6), which is what convinced me:

time event
10:03:01 uv replaces awscrt's sources with 0.36.1 while HA is running
10:03:11 mqtt/mqtt5/aws_iot_metrics .pyc written — imported fresh, so from 0.36.1. io.py and __init__.py get no new .pyc: still resident from the old version
10:52:15 AttributeError
17:10:01 HA restarted → io/__init__ .pyc finally recompiled → config entry loaded

On disk that machine is completely clean: awscrt 0.36.1 throughout, a single .dist-info, one awscrt dir on sys.path, ClientTlsContext.__slots__ does contain _certificate_source, and every .pyc is timestamp-validated and current. The corrupt state existed only in the live process. (The unqualified 'ClientTlsContext' in the error does correctly indicate the loaded class lacks the slot — my error was inferring from that anything about the files on disk.)

This also looks like the same underlying cause as ha_hatch#293 (function takes exactly 17 arguments (18 given)) — same in-place swap, but hitting the Python↔native _awscrt boundary rather than the metrics code.

What this change does, and what it doesn't

Passing enable_metrics_collection=False stops us opting into the metrics path, so the private-attribute read never happens. Those metrics only append AWS SDK name/version/platform to the MQTT CONNECT username for AWS's own telemetry; nothing here consumes them and Hatch's broker doesn't care. The connection is otherwise unchanged, and it drops a needless dependency on awscrt private internals.

To be straight about the limits: this is not a general cure for the module skew — restarting Home Assistant is. It removes this specific crash, and it is worth having regardless (we shouldn't be reaching into awscrt privates for telemetry we don't want), but a process that has half-old awscrt in memory can still fail at other boundaries, as #293 suggests. Users hitting either error should restart HA; this change means a subsequent in-place awscrt upgrade won't reintroduce this particular failure.

Pinning awscrt/awsiotsdk wouldn't help here either: the disk is already consistent, and the damage is in the live interpreter.

Verification

Simulating the resident-old-class condition (unsetting the slot in a ClientTlsContext.__init__ wrapper) against real awscrt 0.36.1 + awsiotsdk 1.31.0 on Python 3.14.6:

--- current behaviour (no enable_metrics_collection) ---
REPRODUCED AttributeError: 'ClientTlsContext' object has no attribute '_certificate_source'

--- with enable_metrics_collection=False ---
OK: built Connection, metrics disabled -> True

Added tests/test_util_bootstrap.py, which stubs the REST/AWS surface and asserts get_rest_devices passes enable_metrics_collection=False. Confirmed it fails without the fix (KeyError: 'enable_metrics_collection') and passes with it.

python -m unittest discover -s tests → 25 tests OK. python -m ruff check . → All checks passed.

…crash

awsiot's mqtt_connection_builder defaults enable_metrics_collection to
True, so it builds an AWS IoT SDK metrics string and passes it to
awscrt.mqtt.Connection with disable_metrics=False. awscrt then reads
private ClientTlsContext internals to encode the feature list:

    # awscrt/aws_iot_metrics.py
    val = _certificate_source_metrics_value(tls_ctx._certificate_source)

_certificate_source was only added to ClientTlsContext in awscrt 0.35.0.
On installs where awscrt's modules are not all from the same version --
which is what several Home Assistant 2026.7 / Python 3.14 users are
hitting -- that attribute is missing and get_rest_devices dies with:

    AttributeError: 'ClientTlsContext' object has no attribute
    '_certificate_source'

The connection is never even attempted, so setup fails and the retry
loop hammers the Hatch login endpoint until it returns HTTP 429.

These metrics only report AWS SDK name/version/platform to AWS in the
CONNECT packet username; nothing in this library consumes them. Opting
out skips the offending code path entirely and leaves the connection
otherwise unchanged.

Fixes dahlb/ha_hatch#323
@dahlb
dahlb merged commit 586c3a6 into dahlb:main Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants