Domoticz hangs after full upgrade

[lost]

Hello,

Run into this issue since a week after making a new 64bit raspberry PI OS (debian 13 based/latest), reinstalling domoticz/scripts/plugin (only one, Z4D now using python venv).

My HW is zwave (mostly, still using legacy OZW as I only have legacy hardware there) and zigbee (recent additions after OZW being deprecated, using Z4D python plugin)

This new SD was prepared on a 2nd exact same PI3B, all that could be tested (without both zwave and zigbee adapters plugged, so related support not fully started) looked OK, even after restoring a DB: Domoticz starts, just complain about missing z-stick gen5 and zi-stick, web UI OK.

So this was time to swap SD on the 1st PI to be upgraded: This immediately hanged, no logs as they were in a tmp but moves in /var/log no file after a 2nd hang.

Starting Domoticz from CLI with wiki troubleshooting instructions shows this:

Code: Select all

~/domoticz $ sudo ./domoticz -www 8080 -sslwww 443 -log "/var/log/domoticz.log" -loglevel all -debuglevel normal,auth,hardware,received,webserver,eventsystem,python,thread_id
2025-12-29 14:14:06.914  [7fb0ad8de0] Status: Domoticz V2025.2 (c)2012-2025 GizMoCuz
2025-12-29 14:14:06.914  [7fb0ad8de0] Status: Build Hash: e63981b18, Date: 2025-10-13 10:42:57
2025-12-29 14:14:06.914  [7fb0ad8de0] Status: Startup Path: /home/domo/domoticz/
2025-12-29 14:14:07.118  [7fb0ad8de0] Sunrise: 07:45:00 SunSet: 16:03:00
2025-12-29 14:14:07.118  [7fb0ad8de0] Day length: 08:19:00 Sun at south: 11:54:00
2025-12-29 14:14:07.118  [7fb0ad8de0] Civil twilight start: 07:08:00 Civil twilight end: 16:40:00
2025-12-29 14:14:07.118  [7fb0ad8de0] Nautical twilight start: 06:27:00 Nautical twilight end: 17:20:00
2025-12-29 14:14:07.118  [7fb0ad8de0] Astronomical twilight start: 05:49:00 Astronomical twilight end: 17:59:00
2025-12-29 14:14:07.328  [7fb0ad8de0] Status: PluginSystem: Started, Python version '3.13.5', 2 plugin definitions loaded.
2025-12-29 14:14:07.342  [7fb0ad8de0] Active notification Subsystems: email (1/13)
2025-12-29 14:14:07.346  [7fb0ad8de0] Debug: CWebServer::StartServer() : settings : 'server_settings[is_secure_=false, www_root='/home/domo/domoticz/www', listening_address='::', listening_port='8080', vhostname='', php_cgi_path='']'
2025-12-29 14:14:07.346  [7fb0ad8de0] Debug: CWebServer::StartServer() : IAM settings : 'iam_settings[is_enabled_=true, auth_url='/oauth2/v1/authorize', token_url='/oauth2/v1/token', discovery_url='/.well-known/openid-configuration', code_expire_seconds='60', token_expire_seconds='3600', refresh_expire_seconds='86400']'
2025-12-29 14:14:07.354  [7fb0ad8de0] Status: WebServer(HTTP) started on address: :: with port 8080
2025-12-29 14:14:07.365  [7fb0ad8de0] Debug: [web:8080] Adding IPv4 network (127.0.0.1) to list of trusted networks.
2025-12-29 14:14:07.366  [7fb0ad8de0] Debug: [web:8080] Adding IPv4 network (192.168.1.31) to list of trusted networks.
2025-12-29 14:14:07.366  [7fb0ad8de0] Debug: [web:8080] Adding IPv4 network (192.168.1.100) to list of trusted networks.
2025-12-29 14:14:07.366  [7fb0ad8de0] Debug: [web:8080] Adding IPv4 network (192.168.1.101) to list of trusted networks.
2025-12-29 14:14:07.367  [7fb0ad8de0] Debug: WebServer(HTTP) started with 278 Registered Commands
2025-12-29 14:14:07.367  [7fb0ad8de0] Debug: cWebEm Registration: 23 pages, 9 actions, 4 whitelist urls, 11 whitelist commands
2025-12-29 14:14:07.369  [7fb0ad8de0] Debug: CWebServer::StartServer() : settings : ssl_server_settings['server_settings[is_secure_=true, www_root='/home/domo/domoticz/www', listening_address='::', listening_port='443', vhostname='', php_cgi_path='']', ssl_method='tls', certificate_chain_file_path='./server_cert.pem', ca_cert_file_path='./server_cert.pem', cert_file_path=./server_cert.pem', private_key_file_path='./server_cert.pem', private_key_pass_phrase='', ssl_options='single_dh_use', tmp_dh_file_path='./server_cert.pem', verify_peer=false, verify_fail_if_no_peer_cert=false, verify_file_path='']
2025-12-29 14:14:07.369  [7fb0ad8de0] Debug: CWebServer::StartServer() : IAM settings : 'iam_settings[is_enabled_=true, auth_url='/oauth2/v1/authorize', token_url='/oauth2/v1/token', discovery_url='/.well-known/openid-configuration', code_expire_seconds='60', token_expire_seconds='3600', refresh_expire_seconds='86400']'
2025-12-29 14:14:07.385  [7fb0ad8de0] Debug: [web:443] Enabled ciphers (TLSv1.2) ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
2025-12-29 14:14:07.398  [7fb0ad8de0] Debug: [web:443] 'DH PARAMETERS' found in file ./server_cert.pem
2025-12-29 14:14:07.406  [7fb0ad8de0] Status: WebServer(SSL) started on address: :: with port 443
2025-12-29 14:14:07.407  [7fb0ad8de0] Debug: [web:443] Adding IPv4 network (127.0.0.1) to list of trusted networks.
2025-12-29 14:14:07.407  [7fb0ad8de0] Debug: [web:443] Adding IPv4 network (192.168.1.31) to list of trusted networks.
2025-12-29 14:14:07.408  [7fb0ad8de0] Debug: [web:443] Adding IPv4 network (192.168.1.100) to list of trusted networks.
2025-12-29 14:14:07.408  [7fb0ad8de0] Debug: [web:443] Adding IPv4 network (192.168.1.101) to list of trusted networks.
2025-12-29 14:14:07.409  [7fb0ad8de0] Debug: WebServer(SSL) started with 278 Registered Commands
2025-12-29 14:14:07.409  [7fb0ad8de0] Debug: cWebEm Registration: 23 pages, 9 actions, 4 whitelist urls, 11 whitelist commands
2025-12-29 14:14:07.411  [7fb0ad8de0] Status: Camera: settings (re)loaded
2025-12-29 14:14:07.412  [7fb0ad8de0] Starting shared server on: :::6144
2025-12-29 14:14:07.413  [7fa9a90f20] Status: TCPServer: shared server started...
2025-12-29 14:14:07.414  [7fb0ad8de0] Status: mDNS: Service started
2025-12-29 14:14:07.415  [7fa9280f20] Debug: mDNS: Local IPv4 address: 0.0.0.0
2025-12-29 14:14:07.415  [7fa9280f20] Debug: mDNS: Local IPv6 address: ::
2025-12-29 14:14:07.415  [7fa9280f20] mDNS: Service: _http._tcp.local.:443 for Hostname: domoticz (2 sockets)
2025-12-29 14:14:07.415  [7fa9280f20] Debug: mDNS: Sending mDNS announce
2025-12-29 14:14:07.416  [7f9b7eef20] Status: RxQueue: queue worker started...
2025-12-29 14:14:09.417  [7f9a7cef20] Status: Z-Stick: ZWave: This hardware type is Deprecated! Please try to move to MQTT Auto Discovery and ZWave JS UI
2025-12-29 14:14:09.417  [7f9bffef20] Debug: RasPiMobo: Checking for onboard sensors
2025-12-29 14:14:09.457  [7f9bffef20] Status: RasPiMobo: System: ODroid/Raspberry
2025-12-29 14:14:09.458  [7f99fbef20] Status: RasPiMobo: Hardware Monitor: Started (OStype Linux)
2025-12-29 14:14:09.459  [7f997aef20] Status: Météo Guyancourt VC: Started...
2025-12-29 14:14:09.460  [7f9bffef20] Zi-Stick: Worker thread started.
2025-12-29 14:14:09.460  [7f98f9ef20] Status: Zi-Stick: Entering work loop.
2025-12-29 14:14:09.460  [7f9bffef20] Status: Zi-Stick: Started.
2025-12-29 14:14:09.461  [7f7fffef20] Status: NotificationSystem: thread started...
2025-12-29 14:14:09.463  [7f9bffef20] Status: EventSystem: reset all events...
2025-12-29 14:14:09.466  [7f9bffef20] Debug: EventSystem: Events (re)loaded
2025-12-29 14:14:09.466  [7f9bffef20] Status: EventSystem: reset all device statuses...
2025-12-29 14:14:09.637  [7fae05ef20] Status: PluginSystem: 1 plugins started.
2025-12-29 14:14:09.769  [7f98f9ef20] Error: Zi-Stick: (Zigate) failed to load 'plugin.py', Python Path used was '/home/domo/domoticz/plugins/Domoticz-Zigbee/:/usr/lib/python313.zip:/usr/lib/python3.13:/usr/lib/python3.13/lib-dynload:/usr/local/lib/python3.13/dist-packages:/usr/lib/python3/dist-packages:/usr/lib/python3.13/dist-packages'.
2025-12-29 14:14:09.839  [7f98f9ef20] Error: Zi-Stick: Traceback (most recent call last):
2025-12-29 14:14:09.840  [7f98f9ef20] Error: Zi-Stick:   File "/home/domo/domoticz/plugins/Domoticz-Zigbee/plugin.py", line 134, in <module>
2025-12-29 14:14:09.840  [7f98f9ef20] Error: Zi-Stick:     import z4d_certified_devices
2025-12-29 14:14:09.840  [7f98f9ef20] Error: Zi-Stick: ModuleNotFoundError: No module named 'z4d_certified_devices'
2025-12-29 14:14:09.917  [7f9a7cef20] Status: Z-Stick: using config in: /home/domo/domoticz/Config/
2025-12-29 14:14:09.923 Always, OpenZwave Version 1.6-1967-g5550fa34 Starting Up
2025-12-29 14:14:09.963 Always, Using Language Localization 
2025-12-29 14:14:09.983  [7f9a7cef20] Status: Z-Stick: Starting...
2025-12-29 14:14:09.984  [7f9a7cef20] Status: Z-Stick: Version: 1.6-1967-g5550fa34
2025-12-29 14:14:10.042  [7f9bffef20] Status: Python EventSystem: Initializing event module.
2025-12-29 14:14:10.042  [7f9bffef20] Status: Python EventSystem: Initializing event module.
2025-12-29 14:14:10.043  [7f7efdef20] Status: EventSystem: Started
2025-12-29 14:14:10.043  [7f7e7cef20] Status: EventSystem: Queue thread started...
2025-12-29 14:14:11.459  [7f99fbef20] Debug: RasPiMobo: Fetching *NIX sensor data (System sensors)
2025-12-29 14:14:11.459  [7f99fbef20] Debug: RasPiMobo: Getting Internal Temperature
2025-12-29 14:14:11.471  [7f9b7eef20] Debug: SQLH UpdateValueInt PI CPU Temp HwID:6  DevID:1 Type:80  sType:5 nValue:0 sValue:48.9 IDX: 48
2025-12-29 14:14:11.480  [7f9b7eef20] subtype       = TEMP5 - LaCrosse TX2, TX3, TX4, TX17
Sequence nbr  = 0
ID            = 1
Temperature   = 48.9 C
Signal level  = 12
Battery       = OK
2025-12-29 14:14:11.480  [7f9b7eef20] RasPiMobo: Temp (PI CPU Temp)
2025-12-29 14:14:11.480  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_alarmPanel.lua)
2025-12-29 14:14:11.493  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_tempoWarn.lua)
2025-12-29 14:14:11.505  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_heaterOnMov.lua)
2025-12-29 14:14:11.515  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_flood.lua)
2025-12-29 14:14:11.526  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_alarmTriger.lua)
2025-12-29 14:14:11.538  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_smartMeter.lua)
2025-12-29 14:14:11.549  [7f7e7cef20] Debug: EventSystem: script device trigger (/home/domo/domoticz/scripts/lua/script_device_plugsMgt.lua)

So we have some Z4D errors that looked to be linked with z4d_certified_devices not present, but I follow the Z4D wiki procedure and this indeed looks to hang a bit later, plugin not started so don't think this is the RC for the crash, as last log is about a Lua script started by event system...

As this abrubtly crash the PI with full HW support (half of the times up to need mounting the SD on a Linux PC so fsck can be run on boot partition and ext4 journal being replayed on rootfs, something the boot loader of the PI3 seems unable to do from it's probably minimal ext support), that's not easy to get more clues.

On top of that, with logging now not in files but managed by journald, looks default on raspberry PI OS 13 is to have this non persistent, needing a change in /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf (=> Storage=persistent).

But on top of this, I now need to learn a bit how to check those journald system logs no more stored as clear text.... I'll let know if I can find something there.

Quite unusual to have some applicative SW abruptly crashing Linux like this. Do someone have seen this or debug tips to propose?

I for now did revert old SD (Debian 11 and Domoticz stable 2025.1 based)...

[waltervl]

Seems the installation of z4D was not correct (with python venv)
And yes, python plugins can crash Domoticz. But there should be a crashlog file in the domoticz folder showing where it went wrong.
But concentrate on the z4D plugin, that should run first (or disable it for now).

[lost]

Yes, I asked Pipiche on the french forum about this... but IMO, the problem is most probably not there because as a result the plugin don't start at all... and as I said, very strange to have such abrupt crash starting some applicative SW: This looks kernel related but why? Without both zwave/zigbee controllers on the spare PI domoticz starts. And these are both serial-usb based (so only serial-usb kernel module involved between spare/production PIs) and if there was a serial-usb issue in the kernel I for sure would not be the 1st to notice!

So I'm a bit stuck for now...

[lost]

Can't find anything from journald logs using the SD mounted on a Linux PC and using it's var/log/journal. I have the line used to start somotice manually with debug options from wiki, pam message as started as root.... and nothing more, system logs stops as abruptly as the crash occurs:

Code: Select all

(...)
déc. 29 15:14:06 rasDom sudo[1448]:     domo : TTY=pts/0 ; PWD=/home/domo/domoticz ; USER=root ; COMMAND=./domoticz -www 8080 -sslwww 443 -log /var/log/domoticz.log -loglevel all -debuglevel normal,auth,hardware,received,webserver,eventsystem,python,thread_id
déc. 29 15:14:06 rasDom sudo[1448]: pam_unix(sudo:session): session opened for user root(uid=0) by domo(uid=1000)