2026-01-11 05:10:05.523 Error: main thread seems to have ended or hung unexpectedly (last update 301.000000 seconds ago

[akamming]

Hi,

i am running on the latest beta in docker on a pi 4. about everymonth i do an update (so also update to latest beta)

Since the last few weeks i have an issue that sometimes domoticz crashed. Pattern is:

It stats with the a slimme meter fault and the the mainworker stopping:

Code: Select all

2026-01-11 05:00:46.615  Slimme Meter: Dismiss incoming - value is not delimited in line "1-0:1.8.1(0"
2026-01-11 05:00:46.826  Slimme Meter: Dismiss incoming - value is not delimited in line "1-0:1.8.1(0"
2026-01-11 05:04:53.867  Error: mainworker seems to have ended or hung unexpectedly (last update 301.000000 seconds ago)
2026-01-11 05:05:03.646  Error: Domoticz(1) is exiting due to watchdog triggered...
2026-01-11 05:05:04.072  Error: Failed to start gdb, will use backtrace() for printing stack frame

2026-01-11 05:05:04.788  Status: Closing application!...
2026-01-11 05:05:04.811  Status: Stopping worker...
2026-01-11 05:05:10.073  Error: mainworker seems to have ended or hung unexpectedly (last update 318.000000 seconds ago)
2026-01-11 05:05:16.491  Error: mainworker seems to have ended or hung unexpectedly (last update 324.000000 seconds ago)
2026-01-11 05:05:18.507  Error: mainworker seems to have ended or hung unexpectedly (last update 326.000000 seconds ago)
2026-01-11 05:05:19.508  Error: mainworker seems to have ended or hung unexpectedly (last update 327.000000 seconds ago)
2026-01-11 05:05:20.508  Error: mainworker seems to have ended or hung unexpectedly (last update 328.000000 seconds ago)
2026-01-11 05:05:21.509  Error: mainworker seems to have ended or hung unexpectedly (last update 329.000000 seconds ago)
2026-01-11 05:05:22.509  Error: mainworker seems to have ended or hung unexpectedly (last update 330.000000 seconds ago)
2026-01-11 05:05:23.510  Error: mainworker seems to have ended or hung unexpectedly (last update 331.000000 seconds ago)
2026-01-11 05:05:24.510  Error: mainworker seems to have ended or hung unexpectedly (last update 332.000000 seconds ago)
2026-01-11 05:05:25.511  Error: mainworker seems to have ended or hung unexpectedly (last update 333.000000 seconds ago)

I think it has to do with the database backup taking too much time, cause when that end, several hardware items start to work again (but also some more functions of domoticz stopping threads):

Code: Select all

 2026-01-11 05:09:22.602  Error: mainworker seems to have ended or hung unexpectedly (last update 570.000000 seconds ago)
2026-01-11 05:09:23.602  Error: mainworker seems to have ended or hung unexpectedly (last update 571.000000 seconds ago)
2026-01-11 05:09:24.603  Error: mainworker seems to have ended or hung unexpectedly (last update 572.000000 seconds ago)
2026-01-11 05:09:25.603  Error: mainworker seems to have ended or hung unexpectedly (last update 573.000000 seconds ago)
2026-01-11 05:09:38.905  Error: mainworker seems to have ended or hung unexpectedly (last update 586.000000 seconds ago)
2026-01-11 05:09:39.158  Status: Ending automatic database backup procedure...
2026-01-11 05:09:39.488  XC40: General/Custom Sensor (XC40-fullRange)
2026-01-11 05:09:39.615  Slimme Meter: Usage (Delivery L1)
2026-01-11 05:09:39.616  Status: RxQueue: queue worker stopped...
2026-01-11 05:09:39.618  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:09:39.618  Status: Stopping all hardware...
2026-01-11 05:09:39.620  XC40: General/Custom Sensor (XC40-estimatedEfficiency)
2026-01-11 05:09:39.635  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:09:39.636  XC40: TextSensor (XC40-Last Known Location)
2026-01-11 05:09:39.640  MQTT AD: Temp/THR128/138, THC138 (domesphelper (Outside_Temperature))
2026-01-11 05:09:39.640  XC40: KWH Meter (Volvo-chargedTotal)
2026-01-11 05:09:39.643  XC40: KWH Meter (Volvo-usedKWH)
2026-01-11 05:09:39.645  MQTT AD: Temp/THR128/138, THC138 (domesphelper (OT_Outside_Temperature))
2026-01-11 05:09:39.648  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:09:39.648  XC40: KWH Meter (Volvo Thuis geladen)
2026-01-11 05:09:39.651  XC40: KWH Meter (Charged Public)
2026-01-11 05:09:39.652  MQTT AD: Temp/THR128/138, THC138 (domesphelper (Outside_Temperature))
2026-01-11 05:09:39.653  XC40: KWH Meter (Charged Public AC)
 

And then at 05:10 it get's into a complete non functional mode:

Code: Select all

2026-01-11 05:10:23.079  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:10:23.096  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:10:23.120  Status: XC40: Stopping threads.
2026-01-11 05:10:23.121  Status: XC40: Stopped.
2026-01-11 05:10:23.121  Status: buienradar: Worker stopped...
2026-01-11 05:10:23.172  Status: Cpu: Hardware Monitor: Stopped...
2026-01-11 05:10:23.243  MQTT AD: General/Text (domesphelper (Uptime))
2026-01-11 05:10:23.272  Error: main thread seems to have ended or hung unexpectedly (last update 319.000000 seconds ago)
2026-01-11 05:10:23.408  Status: MQTT AD: Worker stopped...
2026-01-11 05:10:24.273  Error: main thread seems to have ended or hung unexpectedly (last update 320.000000 seconds ago)
2026-01-11 05:10:25.273  Error: main thread seems to have ended or hung unexpectedly (last update 321.000000 seconds ago)
2026-01-11 05:10:26.274  Error: main thread seems to have ended or hung unexpectedly (last update 322.000000 seconds ago)
2026-01-11 05:10:27.274  Error: main thread seems to have ended or hung unexpectedly (last update 323.000000 seconds ago)
2026-01-11 05:10:41.472  Error: main thread seems to have ended or hung unexpectedly (last update 337.000000 seconds ago)
2026-01-11 05:10:42.472  Error: main thread seems to have ended or hung unexpectedly (last update 338.000000 seconds ago)
2026-01-11 05:10:43.473  Error: main thread seems to have ended or hung unexpectedly (last update 339.000000 seconds ago)
2026-01-11 05:10:44.473  Error: main thread seems to have ended or hung unexpectedly (last update 340.000000 seconds ago)
2026-01-11 05:10:45.473  Error: main thread seems to have ended or hung unexpectedly (last update 341.000000 seconds ago)
2026-01-11 05:10:46.474  Error: main thread seems to have ended or hung unexpectedly (last update 342.000000 seconds ago)
2026-01-11 05:10:47.474  Error: main thread seems to have ended or hung unexpectedly (last update 343.000000 seconds ago)
2026-01-11 05:10:48.474  Error: main thread seems to have ended or hung unexpectedly (last update 344.000000 seconds ago)
2026-01-11 05:10:49.475  Error: main thread seems to have ended or hung unexpectedly (last update 345.000000 seconds ago)
2026-01-11 05:10:50.475  Error: main thread seems to have ended or hung unexpectedly (last update 346.000000 seconds ago)
2026-01-11 05:10:51.475  Error: main thread seems to have ended or hung unexpectedly (last update 347.000000 seconds ago)
2026-01-11 05:10:52.476  Error: main thread seems to have ended or hung unexpectedly (last update 348.000000 seconds ago)
2026-01-11 05:10:53.476  Error: main thread seems to have ended or hung unexpectedly (last update 349.000000 seconds ago)

Never had these issues until about 4 weeks ago, now this happens about once a week. And because domoticz does not fully stop, it is also not automatically restarted, so my domoticz remains inresponsive.

How to fix this?

BTW: This logging is from Version: 2025.2 (build 17003)

[waltervl]

How long does it take to create the backup? Just fo a reference, in my production environment it takes 2 seconds for my 30 MB database.

What if you switch the backup off temporary?

[akamming]

How long does it take to create the backup? Just fo a reference, in my production environment it takes 2 seconds for my 30 MB database.

What if you switch the backup off temporary?

normally it's about 9 seconds (probably takes longer than your case cause it's going to a network drive).

This longer time only happened in this case where domoticz got stuck . I don't want to switch off the backup, cause it's not an issue i can reproduce very easily. Probably it's now running for a long time and then after a week or so it crashes again... (and don't want to be without backup for such a long time)

[waltervl]

Going to a network drive can give issues when the network has issues. Better copy the backups from you local disk to the network disk in a separate process if you want them there.... Then if there are network issues Domoticz is not influenced by it.

[akamming]

Going to a network drive can give issues when the network has issues. Better copy the backups from you local disk to the network disk in a separate process if you want them there.... Then if there are network issues Domoticz is not influenced by it.

I tried. This morning I had the same issue on the local fs

I am not convinced the backup is causing it. More likely the very slow backup is also a symptom of a deeper issue which causes the mainworker and the backup to block

[waltervl]

Then check your event scripts (lua/dzvents) as those, when not using asynchronuos external commands can lead into delays.

Or perhaps your server is getting slow due to other applications running on the server.

[akamming]

Then check your event scripts (lua/dzvents) as those, when not using asynchronuos external commands can lead into delays.

Or perhaps your server is getting slow due to other applications running on the server.

Dzvents is disabled. I have no lua scripts. Domoticz is the only application on this pi

[waltervl]

Perhaps the error starts with this message

Code: Select all

 Slimme Meter: Dismiss incoming - value is not delimited in line "1-0:1.8.1(0"

This looks like a malformed P1 telegram. What could be the reason for that?

[akamming]

Perhaps the error starts with this message

Code: Select all

 Slimme Meter: Dismiss incoming - value is not delimited in line "1-0:1.8.1(0"

This looks like a malformed P1 telegram. What could be the reason for that?

This error indeed only happened at the time of this crash. No clue what the cause is. It normally does not happen