Error: Domoticz received fatal signal 11 when executing plugin code

Python and python framework

Moderator: leecollings

Post Reply
User avatar
jorgh
Posts: 124
Joined: Friday 27 June 2014 23:19
Target OS: Raspberry Pi / ODroid
Domoticz version: 3.8224
Location: Netherlands
Contact:

Error: Domoticz received fatal signal 11 when executing plugin code

Post by jorgh »

All,

On two occasions I've noticed that Domoticz crashes reporting signal 11, (segmentation fault) when executing python code using the plugin framework.

As far as I can tell, there is no error in the Python code. In both occasions, it happens if I pass data, returned from a Domoticz function to another function. So there appears to be something gone wrong with memory allocation on the returned data. I've put the easiest to understand example below.

In this scenario, I add a new level to a multilevel switch. After this, if you request the Level options, Domoticz crashes when you try to access the data. After I restart domoticz, I can handle the level options, until I add a new level. This is 100% reproducable for me.

Code to add a level:

Code: Select all

def addListeningMode(strCode):
  nValue = Devices[MAINLISTENINGMODE].nValue
  sValue = Devices[MAINLISTENINGMODE].sValue
  dictOptions = Devices[MAINLISTENINGMODE].Options
  dictOptions["LevelNames"] = dictOptions["LevelNames"]+'|['+strCode+']'+' New'
  dictOptions["LevelActions"] = dictOptions["LevelActions"]+'|'

  Devices[MAINLISTENINGMODE].Update(nValue = nValue, sValue = sValue, Options = dictOptions)
After a level has been added, the following code (created just for reproduction) crashes Domoticz, when no new levels are added, the code does not cause Domoticz to crash.

Code: Select all

def setSelectorByCode(intId, strCode):
  Domoticz.Log("Onkyo: setSelectorByCode code: "+strCode)
  dictOptions = Devices[intId].Options
  Domoticz.Log("Onkyo: Fetched Options")
  Domoticz.Log("Onkyo: options: "+dictOptions['LevelNames'])

The following is logged:

Code: Select all

2017-05-12 11:21:55.270  (Onkyo) Onkyo: setSelectorByCode code: 80
2017-05-12 11:21:55.272  (Onkyo) Onkyo: Fetched Options
2017-05-12 11:21:55.273  Error: Domoticz received fatal signal 11 !...
2017-05-12 11:21:55.282  Error:   /home/pi/dev-domoticz/domoticz() [0x1c25c0]
2017-05-12 11:21:55.284  Error:   /home/pi/dev-domoticz/domoticz(_Z14signal_handleri+0x58) [0x1c2678]
2017-05-12 11:21:55.285  Error:   /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer_v2+0) [0xb6b2a180]
In the other case, I pass code received from onMessage to the python XML library. The Error log is 100% the same in that case.

Is there someone that can shed some light on this? Do I need to file a bug report somewhere? If any additional data or logging is needed, I'm willing to assist to further debug this issue.

Kind regards,

Jorg
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Dnpwwo »

@jorgh,

I'm not familiar with the device Options code, it was changed to a dictionary by @oohlaf.

From a quick look, both the device 'insert' and 'update' functions use the same internal SetDeviceOptions but the 'update' decrements the Python object reference counter afterwards while the 'insert' does not. This would suggest that one of these functions is incorrect.

When an object's reference count drops to zero Python will garbage collect the memory which could easily lead to kind of seg fault you are seeing.

I can't look at this right now but if you are building from source on Windows set a break on line 807 of PythonObjects.cpp and have a look at at pOptionsDict's ob_refcnt when you run your test code. If it goes to zero you've found the problem.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
User avatar
jorgh
Posts: 124
Joined: Friday 27 June 2014 23:19
Target OS: Raspberry Pi / ODroid
Domoticz version: 3.8224
Location: Netherlands
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by jorgh »

In search of the cause of Domoticz crashing (signal 11) I've compiled Domoticz from source with 'Debug' target.
However, I was unable to reproduce the crash. Subsequenly I compiled with 'Release' target, however that also did not reproduce the crash.
When running the version that was updated (so basically downloading the compiled version) Domoticz allways crashes running my plugin.

All versions (downloaded, compiled release, compiled debug) are identical (V3.8072)

What could be a possible explanation for this, might there be something 'wrong' with the pre-compiled versions, as I read Domoticz crashes in multiple plugin topics? It seems to be an 'generic' issue that more people are facing and for me, compiling from source I do not face the issue anymore.

The point that Domoticz crashes within the plugin, is (in my case) always when updating switches, although the number of successful updates before Domoticz crashes varies across versions, and is not limited to adding levels to selector switches, but any update, to any switch type.

Regards,

Jorg
Logread
Posts: 228
Joined: Sunday 28 August 2016 7:48
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: France
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Logread »

+1
All the 5 python plugins I wrote and am using started generating various errors (code 6 or 11, device creation/updating errors) for the past couple of weeks using the pre-built binaries (from version 80xx more or less I think)... I found out thanks to one user of my plugins that self compiled builds are ok... my head stopped banging the wall :oops:
User avatar
jorgh
Posts: 124
Joined: Friday 27 June 2014 23:19
Target OS: Raspberry Pi / ODroid
Domoticz version: 3.8224
Location: Netherlands
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by jorgh »

Well in that case, there might be something wrong with the environment that creates the binary builds. I don't know how/what or who creates these builds, but someone should take a look at that. Most people use the binary builds and as there seems to be an issue with the generated builds that a lot of people have issues with.

Regards,

Jorg
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Dnpwwo »

@jorgh & @Logread,

Thanks for the feedback. I never get crashes and have suspected for a while that this could be related to the RPi build itself rather than the plugin code itself (my current main system has been running for over 13 days with 7 plugins).

From what I remember, @Gizmocuz says that the RPi build is done using the same mechanism as when building from source so maybe its the versions of libraries.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
User avatar
G3rard
Posts: 669
Joined: Wednesday 04 March 2015 22:15
Target OS: -
Domoticz version: No
Location: The Netherlands
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by G3rard »

And on my Ubuntu server the plugins are also running without any problems (Domoticz v3.8025).
So seems indeed related to RPi build.
Not using Domoticz anymore
Electrocut
Posts: 22
Joined: Sunday 04 May 2014 21:52
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Electrocut »

Same for me.

Software and Hardware situation:
- RaspberryPi 2
- Rapsbian Jessie
- Domoticz stable 3.8153, built on-board from git source

When I add a Python plugin, domoticz crashed on RaspberryPi first boot:

Code: Select all

2017-09-28 18:25:11.844  OpenZWave: Value_Added: Node: 14 (0x0e), CommandClass: SENSOR ALARM, Label: Heat, Instance: 1
2017-09-28 18:25:11.988  MySensors: Using serial port: /dev/ttyUSB1
2017-09-28 18:25:12.137  RFLink: Controller Initialized!...
2017-09-28 18:25:12.141  (Thermostat) Initialized version 0.3.3, author 'logread'
2017-09-28 18:25:12.197  Error: Domoticz received fatal signal 11 !...
2017-09-28 18:25:12.205  Error:   /home/pi/domoticz/domoticz() [0x1ce184]
2017-09-28 18:25:12.205  Error:   /home/pi/domoticz/domoticz(_Z14signal_handleri+0x58) [0x1ce23c]
2017-09-28 18:25:12.205  Error:   /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer_v2+0) [0x76ad0180]
Then, I call:
sudo /etc/init.d/domoticz.sh stop
sudo /etc/init.d/domoticz.sh start

... and domoticz starts successfully:

Code: Select all

2017-09-28 18:26:43.737  OpenZWave: Value_Added: Node: 14 (0x0e), CommandClass: SENSOR ALARM, Label: General, Instance: 1
2017-09-28 18:26:43.738  OpenZWave: Value_Added: Node: 14 (0x0e), CommandClass: SENSOR ALARM, Label: Smoke, Instance: 1
2017-09-28 18:26:43.738  OpenZWave: Value_Added: Node: 14 (0x0e), CommandClass: SENSOR ALARM, Label: Heat, Instance: 1
2017-09-28 18:26:43.918  MySensors: Using serial port: /dev/ttyUSB1
2017-09-28 18:26:44.062  RFLink: Controller Initialized!...
2017-09-28 18:26:44.068  (Thermostat) Initialized version 0.3.3, author 'logread'
2017-09-28 18:26:44.126  (Thermostat) Debug log level set to: 'false'.
2017-09-28 18:26:44.137  RFLink Detected, Version: 1.1 Revision: 45 Build: 9
2017-09-28 18:26:44.172  Incoming connection from: 127.0.0.1
2017-09-28 18:26:44.184  User: Admin initiated a switch command (129/Chaudière test/Off)
2017-09-28 18:26:44.220  (Dummy) Light/Switch (Chaudière test)
2017-09-28 18:26:44.414  subtype       = TH5 - WTGR800
Sequence nbr  = 0
ID            = 23544
Temperature   = 21.7 C
Humidity      = 70 %
Status        = Wet
Signal level  = 12
Battery       = OK
2017-09-28 18:26:44.415  (RFLink) Temp + Humidity (Température Chambre)
2017-09-28 18:26:45.332  subtype       = TEMP5 - LaCrosse TX2, TX3, TX4, TX17
Sequence nbr  = 0
ID            = 29441
Temperature   = 19.8 C
Signal level  = 12
Battery       = OK
...
Until next reboot or power failure.

I will try the binary version built for Arm, provided on domoticz website.
Electrocut
Posts: 22
Joined: Sunday 04 May 2014 21:52
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Electrocut »

With 3.8153 version downloaded from domoticz.com, it crashes the same way, but log is more verbose!

Code: Select all

2017-09-28 18:41:36.158  (Thermostat) Initialized version 0.3.3, author 'logread'
2017-09-28 18:41:36.217  (Thermostat) Debug log level set to: 'false'.
2017-09-28 18:41:36.217  Error: (Thermostat) 'onStart' failed 'AttributeError':''method' object has no attribute 'InTempSensors''.
2017-09-28 18:41:36.218  Error: (Thermostat) ----> Line 479 in /home/pi/domoticz/plugins/SmartVirtualThermostat/plugin.py, function onStart
2017-09-28 18:41:36.218  Error: (Thermostat) ----> Line 150 in /home/pi/domoticz/plugins/SmartVirtualThermostat/plugin.py, function onStart
2017-09-28 18:41:36.218  Error: Domoticz received fatal signal 6 !...
2017-09-28 18:41:36.225  Error:   /home/pi/domoticz/domoticz() [0x1cdc48]
2017-09-28 18:41:36.225  Error:   /home/pi/domoticz/domoticz(_Z14signal_handleri+0x58) [0x1cdd00]
2017-09-28 18:41:36.225  Error:   /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer_v2+0) [0x76b3d180]
2017-09-28 18:41:36.225  Error:   /lib/arm-linux-gnueabihf/libc.so.6(gsignal+0x38) [0x76b3bf70]
Then, I call:
sudo /etc/init.d/domoticz.sh stop
sudo /etc/init.d/domoticz.sh start

... and domoticz starts successfully:

Code: Select all

2017-09-28 18:43:21.241  RxQueue: queue worker started...
2017-09-28 18:43:23.202  Wunderground: Worker started...
2017-09-28 18:43:23.203  (Compteur EDF) Teleinfo device uses serial port: /dev/ttyAMA0 at 1200 bauds
2017-09-28 18:43:23.204  (Compteur EDF) CRC checks will be performed on incoming data
2017-09-28 18:43:23.207  Hardware Monitor: Started
2017-09-28 18:43:23.228  EventSystem: reset all events...
2017-09-28 18:43:23.229  EventSystem: reset all device statuses...
2017-09-28 18:43:23.426  (Compteur EDF) Teleinfo CRC check mode set to 1
2017-09-28 18:43:23.430  RFLink: Using serial port: /dev/ttyUSB0
2017-09-28 18:43:23.512  PluginSystem: Entering work loop.
2017-09-28 18:43:23.530  Python EventSystem: Module not found - Trying to initialize.
2017-09-28 18:43:23.532  Python EventSystem: Initalizing event module.
2017-09-28 18:43:23.533  EventSystem: Started
2017-09-28 18:43:23.702  OpenZWave: using config in: /home/pi/domoticz/Config/
2017-09-28 18:43:23.707  OpenZWave: Starting...
2017-09-28 18:43:23.707  OpenZWave: Version: 1.4-2501-g5af6780-dirty
2017-09-28 18:43:23.887  OpenZWave: Driver Ready
2017-09-28 18:43:24.207  MySensors: Using serial port: /dev/ttyUSB1
2017-09-28 18:43:24.290  (Thermostat) Initialized version 0.3.3, author 'logread'
2017-09-28 18:43:24.347  (Thermostat) Debug log level set to: 'false'.
2017-09-28 18:43:24.355  RFLink: Controller Initialized!...
-> It looks like in the first start, Python EventSystem is not initialized. This 2 lines are only visible in the second log:

Code: Select all

2017-09-28 18:43:23.530  Python EventSystem: Module not found - Trying to initialize.
2017-09-28 18:43:23.532  Python EventSystem: Initalizing event module.
Logread
Posts: 228
Joined: Sunday 28 August 2016 7:48
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: France
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Logread »

I noted that a few days ago python plugins have been re-enabled in domoticz betas, including some exceptions (read error handling) improvements. Thanks to the developpers. Curious to hear feedback from users of python plugins on the most reçent betas...
Electrocut
Posts: 22
Joined: Sunday 04 May 2014 21:52
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by Electrocut »

I have just updated to current (= 3.8450) beta (on my RaspberryPi), using the ARMv7 version.
It seems to be much better than the stable (= 3.8153), regarding the crash describe above.

- For the moment, no SEGFAULT on the first start
- No visible regression for the features I use (Lua, Python plugins, MySensors, RFLink, and "Teleinfo EDF" meter).
rimram31
Posts: 26
Joined: Sunday 04 October 2015 11:25
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: France
Contact:

Re: Error: Domoticz received fatal signal 11 when executing plugin code

Post by rimram31 »

I'm posting this comment to help anyone who will encounter the same issue.

I've an installation which work perfectly now for near ten years ! with an old domoticz version I'm no more updating for a long time (everything is froozen, including the sdcard in ro only mode) as this handle some heaters at home.

And suddenly yesterday domoticz was down and crash after restarting after a few seconds. Looking at logs (reactivated to be able to debug), it shows exactly the same error as here, the reason why I've found this thread with some additionnals lines about Weather Underground and some json parsing error.

The issue appear because I've left an old Weather Underground device/plugin which I know is obsolete for years but now fail in some python issue and crash domoticz ! and something seems to be changed recently, I've been a victim of something changed in the internet :D

As you have no way to remove the plugin as dz don't work anymore, the solution I've found was to stop my internet connection and restart dz which now don't crash, removing Weather Underground and restore my internet connection and restart.

Problem solved, if this may help.
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests