Connection retries stop too quickly

[kylegordon]

Summary: everything is Docker containers. Predbat communicates within the 'homeautomation' docker network and the web UI is exposed through Traefik.

I was doing some maintenance and stopped home-assistant for a few minutes, leaving other containers running. An hour or so after restarting home-assistant I noticed all the Predbat entities were dead.

I can't recall how long home-assistant was down for, but the Predbat logs indicate that it suffered a variety of connect call failures and name or service not known failures for only 3 minutes before stopping completely.

Can Predbat be configured to retry indefinitely?

[gcoan]

Please include logfile

Predbat normally retries every 5 minutes

[kylegordon]

predbat.log

Certainly, attached

[springfall2008]

Predbat doesn't retry forever but it will then exit, and the addon is set to try restarting itself again so it should have just rebooted after 30 seconds.

[springfall2008]

Oh wait you are in docker, so whoever wrote the docker container missed the restart wrapper perhaps? When you edit a file e.g. apps.yaml does it restart okay?

[kylegordon]

Heya, yes, in docker. I don't recall where I obtained the link, but the docker image is from https://hub.docker.com/r/nipar44/predbat_addon.

Editing apps.yaml does indeed cause a restart and successful run.

I'm wondering if there's a healthcheck I can apply to interrogate the latest log line to determine if it's crashed or not, and use docker to restart the task. Or can predbat be configured just to exit on failure?

[nipar4]

Hey Guys

I only change the initial start-up files and make as few modifications to these as possible. However there is a slight issue I have noticed, that if Home assistant is restarted (Or stopped) just as Predbat does a run it will crash and not recover.

All other restarts due to file modification etc. work

My assumption up is this would normally be handled by the HA management system so would be invisible to anyone who uses the addon as everything would be restarted or stopped together.

I am currently using a crude health check and an HA automation script to identify a problem and restart but this is not too reliable and a bit complex

Based on the suggestion above can predbat be made to exit on failure back to the start-up script which would then automatically cause the restart?

[gcoan]

I'm wondering if there's a healthcheck I can apply to interrogate the latest log line to determine if it's crashed or not

there is an automation in the predbat documentation that will detect if predbat is erroring (error status) or if it has stopped executing (plan stale) https://springfall2008.github.io/batpred/output-data/#predbat-error-monitor

At the moment this restarts the predbat addon but could be modified to call the OS command to restart the predbat docker container

[nipar4]

I've tried this but some of the processes are still running. For example preheat is still running, or appears to be, the web interface is still active therefore predbat.status is still updated and the state is still showing as running so the automation will not kick in for a restart

2025-02-12 21:45:00.973125: Predheat: update at 2025-02-12 21:45:00.973076+00:00
2025-02-12 21:45:10.873854: Predheat: has 8 days of historical data
2025-02-12 21:45:10.873935: Predheat: Hysteresis On 0.5 off 0.1 heat loss 120 W heat loss 0.03 W per degree
2025-02-12 21:45:10.873961: Predheat: Mode gas Heating active False Heat loss watts 120 degrees 0.03 watts per degree 4000.0 heating energy so far 110.83 volume temp 24.812 internal temp 19.767 external temp 5.0
2025-02-12 21:45:10.888721: Predheat: Todays energy import 110.83 kwh cost 0 p
2025-02-12 21:45:11.023105: Predheat: Completed run status idle
2025-02-12 21:50:00.321317: Predheat: update at 2025-02-12 21:50:00.321191+00:00
2025-02-12 21:50:02.848981: Predheat: has 8 days of historical data
2025-02-12 21:50:02.849062: Predheat: Hysteresis On 0.5 off 0.1 heat loss 120 W heat loss 0.03 W per degree
2025-02-12 21:50:02.849087: Predheat: Mode gas Heating active False Heat loss watts 120 degrees 0.03 watts per degree 4000.0 heating energy so far 110.83 volume temp 23.998 internal temp 19.767 external temp 5.0
2025-02-12 21:50:02.862188: Predheat: Todays energy import 110.83 kwh cost 0 p
2025-02-12 21:50:02.991243: Predheat: Completed run status idle
2025-02-12 21:55:00.290501: Predheat: update at 2025-02-12 21:55:00.290368+00:00
2025-02-12 21:55:03.155061: Predheat: has 8 days of historical data
2025-02-12 21:55:03.155144: Predheat: Hysteresis On 0.5 off 0.1 heat loss 120 W heat loss 0.03 W per degree
2025-02-12 21:55:03.155173: Predheat: Mode gas Heating active False Heat loss watts 120 degrees 0.03 watts per degree 4000.0 heating energy so far 110.83 volume temp 23.347 internal temp 19.767 external temp 5.0
2025-02-12 21:55:03.168571: Predheat: Todays energy import 110.83 kwh cost 0 p
2025-02-12 21:55:03.296545: Predheat: Completed run status idle

[nipar4]

Here's the docker log of the error if it helps....

_predbat_logs.txt

[kylegordon]

I'm wondering if there's a healthcheck I can apply to interrogate the latest log line to determine if it's crashed or not

there is an automation in the predbat documentation that will detect if predbat is erroring (error status) or if it has stopped executing (plan stale) https://springfall2008.github.io/batpred/output-data/#predbat-error-monitor

At the moment this restarts the predbat addon but could be modified to call the OS command to restart the predbat docker container

Without privileged access and/or access to the Docker socket, HA will not be able to instruct the OS/Docker/system to restart the container. Whilst it's technically possible to provide all this, granting such powers wouldn't be appropriate. Maybe ok in a dedicated system such as HAOS, but I would say otherwise in a generic containerized system.

I don't think even a HA shell command would work as that would be running inside the context of the container, not the OS.

Usually in these cases we rely on Docker healthchecks or something further upstream. The processes should fail-fast or fail-early, and then let the docker restart policy take over

[nipar4]

Not seen any discussion on the latest modifications on main for restarting so thought I would comment here as this is relevant to the docker use case

I have tested this and for me it works as required, predbat will perform a number of retries then fall back to restarting. Although I haven't left this for hours, for the time I have been testing it successfully restarts when Home assistant eventually comes back online.

Hopefully this will also resolve the other use cases and means I can remove my automations which is good

[kylegordon]

I still find that predbat fails unrecoverably after all the docker containers are brought up simultaneously, such as from a cold boot or a docker daemon upgrade/restart. It's been over 5 minutes now with no output from docker logs -f predbat, and whilst HA has finished booting, along with Z2M, Node-Red, etc, Predbat has tried, failed and gotten stuck.

Using latest (19/02/25 nipar44/predbat_addon@sha256:21ecbf99ad571289d0bc6931fb249a98d018665bdedf5c52c4140527cd4a8e2a) but I'd be interested to know if this is reproducible elsewhere.

2025-02-26 19:13:46.971401: Warn: Regular expression argument: geserial unable to match re:sensor.givtcp_(.+)soc_kwh, now will disable
2025-02-26 19:13:46.971959: Warn: Regular expression argument: geserial2 unable to match re:sensor.givtcp2(.+)soc_kwh, now will disable
2025-02-26 19:13:46.972561: Warn: Regular expression argument: pv_forecast_today unable to match re:(sensor.(solcast|)(pv_forecast_|)forecast_today), now will disable
2025-02-26 19:13:46.973251: Warn: Regular expression argument: pv_forecast_tomorrow unable to match re:(sensor.(solcast_|)(pv_forecast_|)forecast_tomorrow), now will disable
2025-02-26 19:13:46.973825: Warn: Regular expression argument: pv_forecast_d3 unable to match re:(sensor.(solcast_|)(pv_forecast_|)forecast_(day_3|d3)), now will disable
2025-02-26 19:13:46.974374: Warn: Regular expression argument: pv_forecast_d4 unable to match re:(sensor.(solcast_|)(pv_forecast_|)forecast_(day_4|d4)), now will disable
2025-02-26 19:13:46.974881: Warn: Regular expression argument: car_charging_energy unable to match re:(sensor.myenergi_zappi_[0-9a-z]+charge_added_session|sensor.wallbox_portal_added_energy), now will disable
2025-02-26 19:13:46.975384: Warn: Regular argument car_charging_planned expression re:(sensor.wallbox_portal_status_description|sensor.myenergi_zappi[0-9a-z]+plug_status) failed to match - disabling this item
2025-02-26 19:13:46.975799: Warn: Regular expression argument: octopus_intelligent_slot unable to match re:(binary_sensor.octopus_energy([0-9a-z]+|)intelligent_dispatching), now will disable
2025-02-26 19:13:46.976249: Warn: Regular expression argument: octopus_ready_time unable to match re:(time.octopus_energy([0-9a-z]+|)intelligent_ready_time), now will disable
2025-02-26 19:13:46.976973: Warn: Regular expression argument: octopus_charge_limit unable to match re:(number.octopus_energy([0-9a-z]+|)intelligent_charge_limit), now will disable
2025-02-26 19:13:46.977546: Warn: Regular expression argument: carbon_intensity unable to match re:(sensor.carbon_intensity_uk), now will disable
2025-02-26 19:13:46.978198: Warn: Regular expression argument: octopus_saving_session unable to match re:(binary_sensor.octopus_energy([0-9a-z]+|)saving_session(s|)), now will disable
2025-02-26 19:13:46.978621: Warn: Regular expression argument: octopus_free_session unable to match re:(event.octopus_energy([0-9a-z_]+|)octoplus_free_electricity_session_events), now will disable
2025-02-26 19:13:46.979115: Warn: Regular expression argument: metric_octopus_import unable to match re:(sensor.(octopus_energy|)electricity_[0-9a-z]+[0-9a-z]+current_rate), now will disable
2025-02-26 19:13:46.979579: Warn: Regular expression argument: metric_octopus_export unable to match re:(sensor.(octopus_energy|)electricity[0-9a-z]+[0-9a-z]+export_current_rate), now will disable
2025-02-26 19:13:46.980018: Warn: Regular expression argument: metric_standing_charge unable to match re:(sensor.(octopus_energy|)electricity[0-9a-z]+_[0-9a-z]+current_standing_charge), now will disable
Watching ['./execute.py', './apps.yaml', './ha.py', './solcast.py', './web.py', './futurerate.py', './octopus.py', './predbat.py', './alertfeed.py', './energydataservice.py', './inverter.py', './userinterface.py', './unit_test.py', './predheat.py', './compare.py', './fetch.py', './startup.py', './utils.py', './prediction.py', './plan.py', './config.py', './hass.py', './output.py', './download.py', './gecloud.py'] for changes
2025-02-26 19:13:58.108721: Warn: Unable to load data from Github URL: https://api.github.com/repos/springfall2008/batpred/releases
2025-02-26 19:13:58.108783: Warn: Unable to download Predbat version information from github, return code: []
2025-02-26 19:13:58.146145: Warn: can not resolve load_today value sensor.givtcp{geserial}load_energy_today_kwh
2025-02-26 19:13:58.148659: Warn: record_status Warn: can not resolve load_today value sensor.givtcp{geserial}_load_energy_today_kwh
2025-02-26 19:13:58.148689: Error: Failure to fetch history for None
2025-02-26 19:13:58.148695: Error: Unable to fetch history for None
2025-02-26 19:13:58.150781: Warn: record_status Error: Unable to fetch history from None
2025-02-26 19:13:58.150806: Error: Exception raised
2025-02-26 19:13:58.214118: Error: Traceback (most recent call last):
File "/config/predbat.py", line 970, in update_time_loop
self.update_pred(scheduled=False)
File "/config/predbat.py", line 573, in update_pred
self.fetch_sensor_data()
File "/config/fetch.py", line 816, in fetch_sensor_data
self.load_minutes, self.load_minutes_age = self.minute_data_load(self.now_utc, "load_today", self.max_days_previous, required_unit="kWh", load_scaling=self.load_scaling)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/fetch.py", line 407, in minute_data_load
raise ValueError
ValueError

2025-02-26 19:13:58.220678: Info: record_status Error: Exception raised
2025-02-26 19:13:58.220709: Error:
Traceback (most recent call last):
File "/config/hass.py", line 173, in timer_tick
item"callback"
File "/config/predbat.py", line 976, in update_time_loop
raise e
File "/config/predbat.py", line 970, in update_time_loop
self.update_pred(scheduled=False)
File "/config/predbat.py", line 573, in update_pred
self.fetch_sensor_data()
File "/config/fetch.py", line 816, in fetch_sensor_data
self.load_minutes, self.load_minutes_age = self.minute_data_load(self.now_utc, "load_today", self.max_days_previous, required_unit="kWh", load_scaling=self.load_scaling)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/fetch.py", line 407, in minute_data_load
raise ValueError
ValueError

2025-02-26 19:15:01.613236: Warn: can not resolve load_today value sensor.givtcp_{geserial}load_energy_today_kwh
2025-02-26 19:15:01.617903: Warn: record_status Warn: can not resolve load_today value sensor.givtcp{geserial}_load_energy_today_kwh
2025-02-26 19:15:01.617927: Error: Failure to fetch history for None
2025-02-26 19:15:01.617933: Error: Unable to fetch history for None
2025-02-26 19:15:01.621381: Warn: record_status Error: Unable to fetch history from None
2025-02-26 19:15:01.621401: Error: Exception raised
2025-02-26 19:15:01.624262: Error: Traceback (most recent call last):
File "/config/predbat.py", line 1024, in run_time_loop
self.update_pred(scheduled=True)
File "/config/predbat.py", line 573, in update_pred
self.fetch_sensor_data()
File "/config/fetch.py", line 816, in fetch_sensor_data
self.load_minutes, self.load_minutes_age = self.minute_data_load(self.now_utc, "load_today", self.max_days_previous, required_unit="kWh", load_scaling=self.load_scaling)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/fetch.py", line 407, in minute_data_load
raise ValueError
ValueError

2025-02-26 19:15:01.649103: Info: record_status Error: Exception raised
2025-02-26 19:15:01.649135: Error:

[kylegordon]

fwiw, I have the following in docker-compose in an attempt to mitigate this, but to no avail

depends_on:
  GivTCP:
      condition: service_started
  home-assistant:
      condition: service_healthy

I see the failures are probably related to DNS (Pihole is containerized), but Predbat remains wedged long after DNS comes up. Will add a dependency on PiHole in this case

[gcoan]

I see the failures are probably related to DNS

If you use the IP address of your HA server in apps.yaml instead of the URL you will avoid making DNS calls

[kylegordon]

I see the failures are probably related to DNS

If you use the IP address of your HA server in apps.yaml instead of the URL you will avoid making DNS calls

Predbat, GivTCP and HA are all within the same Compose stack. They are relying on Dockers own internal DNS service to communicate with each other. The failures regarding Github are just external lookups failing