How to debug python "thread seems to have ended unexpectedly"?

Python and python framework

Moderator: leecollings

Post Reply
User avatar
heggink
Posts: 994
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi / ODroid
Domoticz version: 12451
Location: NL
Contact:

How to debug python "thread seems to have ended unexpectedly"?

Post by heggink »

I use a bunch of python plugins (buienradar, kodi, macpres, NUT, Tradfri, harmony) some of which use urllib. My suspicion is that this is the cause of getting regular "thread seems to have ended unexpectedly" errors in the log. I struggle to identify which plugin may be the cause. Anyone (@dnpwwo ?) any ideas on how to debug effectively? Every now and then, the whole python plugin system goes haywire and starts spitting these out for all my plugins every other second completely flooding the log with errors. At that point, only a restart seems to work.
Thanks in advance!
Herman
Docker in Truenas scale, close to latest beta
RFXCOM 433 and 868, zigbee2mqtt,
P1 meter & solar panel
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: How to debug python "thread seems to have ended unexpectedly"?

Post by Dnpwwo »

@heggink,

The "thread seems to have ended unexpectedly" messages do not come from the plugin framework itself, they come from the Domoticz 'Watchdog' process that is supposed to log when a piece of hardware (plugin or otherwise) has been unresponsive for greater than 30 seconds.

Sounds like one of your plugins is never returning from a callback.

Determining which one is not easy (unless you upgraded one recently and the problem started). If it happens regularly you can shut them down one at a time using a process of elimination.

If you can send the output to a file you could add a

Code: Select all

Domoticz.Debugging(128)
line to all the onStart functions and the framework will log the calls to all callbacks so you can see the last one that is called before the other messages start.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
User avatar
heggink
Posts: 994
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi / ODroid
Domoticz version: 12451
Location: NL
Contact:

Re: How to debug python "thread seems to have ended unexpectedly"?

Post by heggink »

If I add the debugging to every plugin and restart the lot then I would see which ones are executed and finish and don't finish, before the warning kicks in, correct? Since I use multiple instances of the same plugin in a couple of cases, the elimination process is quite tedious.
My logging is in the syslog anyway (ie a file) so that's already taken care of.
Thanks for the quick reply btw. Much appreciated.
Docker in Truenas scale, close to latest beta
RFXCOM 433 and 868, zigbee2mqtt,
P1 meter & solar panel
dheuts
Posts: 65
Joined: Monday 25 March 2019 15:14
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: Brunssum, Netherlands
Contact:

Re: How to debug python "thread seems to have ended unexpectedly"?

Post by dheuts »

Did you figuren out what the problem was? I have it every full hour
Raspberry Pi 4 With Domoticz - RFXCom - Tuya Wifi LED lights - Chuango Alarm - Zwave+ - Zigbee2MQTT - Anna Thermostat - Broadlink IR, P1 - Eufy Robo Vacuum - Worx Robo Mower
Post Reply