chasing segfault in domoticz with domoticz-solaredge-modbustcp-plugin

[brjhaverkamp]

Hello all,

This Christmas break I decided to upgrade my domoticz setup.
I moved from a raspberry pi 2 with buster to a raspberry pi 3 with bullseye.
I though it best to start from scratch, so I installed a fresh image on a new SD card.
I moved over the domoticz.db and started it up.
Installed all plugins from the git repositories (Domoticz-iDetect, domoticz-solaredge-modbustcp-plugin, domoticz-zigbee2mqtt-plugin
And I installed all required python module dependencies with pip3.

Now I have the problem that about 3 out of 4 times it crashes during startup. It seems to start somewhere in the initialization phase of the solaredge-modbustcp plugin.
So it doesn't crash all the time!?!, and if it is up, it says up. It has been running nicely for a week or so now. But I fear to reboot it. And of course I want to solve this. I worked through this with the author, Addie, but we could'nt get to the root cause of this. So I ask here, in the hope of your advice on what else I can look at?
I added the domoticz_crash.log and the terminal output, which has slightly different info.
These are both from the same crash and were created with all other plugins disabled, just to make sure.
If any more info is needed, please let me know.

Greets,

Bert

domoticz segfault 2023-01-08.txt

(286.17 KiB) Downloaded 29 times

domoticz_crash.log

(54.22 KiB) Downloaded 28 times

[boum]

Hello,
Not an expert of that, I see there are 2 python threads in the output. Are there 2 plugins? "Solarpanels att" and "Solarpanels din"? Or is the python plugin creating 2 threads? Both are in the middle of some "import" stack trace. It might be some race condition in some module dependency (or the importer is not completely thread safe).

[brjhaverkamp]

Hi Boum,

Yes, I have to sets of SolareEdge panels. One on the diningroom and one on the attic. So I have two instances of the plugin running.
But also with one of them disabled, I have the same issue.
On the console the traceback looks like the issue is in the initialization already, when looking at the lines indicated in the different files. But maybe I am interpreting the below list wrongly?

I read about possible python module version incompatibilities when installing some via apt and others via pip. Could that be the cause?



Current thread 0x0000007f9dffaf80 (most recent call first):
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1108 in create_module
File "<frozen importlib._bootstrap>", line 565 in module_from_spec
File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/usr/lib/python3.9/asyncio/events.py", line 786 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
File "/usr/lib/python3.9/asyncio/base_events.py", line 40 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/usr/lib/python3.9/asyncio/__init__.py", line 8 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/usr/local/lib/python3.9/dist-packages/pymodbus/client/base.py", line 4 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/usr/local/lib/python3.9/dist-packages/pymodbus/client/__init__.py", line 5 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/usr/local/lib/python3.9/dist-packages/solaredge_modbus/__init__.py", line 7 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/www/domoticz/plugins/domoticz-solaredge-modbustcp-plugin/plugin.py", line 54 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 790 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
domoticz: Domoticz(pid:2178, tid:2193('Solarpanels att')) received fatal signal 11 (Segmentation fault)

[boum]

I run the plugin with Python 3.8 and no issue there.
Maybe you're right, I don't know much about it, but looking at your logs, the modbus module is loaded from /usr/local/lib/ but asyncio (probably a dependency of this last one) is loaded from /usr/lib/. Maybe one location is pip the other is apt.
Anyway check for updates on the asyncio python module since the crash happens there.

[brjhaverkamp]

New observation.

Only if I have them both enabled, I observe the segfaults. If I have only one plugin enabled, and the other disabled, I could start domoticz 10 times without observing a crash.

Could this be a bottleneck issue or overload issue on the modbus side? or are messages confused? (I have no idea how modbus works to be honest, never dug in.)

[waltervl]

Multiple python plugins using modbus gives issues in stable 2022.2 due to multithreading. See viewtopic.php?t=39645

[brjhaverkamp]

Hi Watervl

Yes, I read the thread. It could be related, although I didn't see any mention of a hard crash there.
I wonder if this can be solved on the domoticz level. Does it make sense to start modules (or only identical modules) with a certain interval in between?

Or Addie, if you read this, can you add a parameter to regulate the loading of multiple module with a certain delay?

[brjhaverkamp]

OK, I tried the delayed startup of the solaredge module by adding a delay parameter and executing a sleep right at the beginning of onStart(self).
This doesn't have the desired effect.
Also from the traces I sent, it looks like the crash happens right at the initiation of the module, possibly even in the include solaredge-modbus line.

I also discovered that I can reproduce the crash by creating two instances of the plugin towards the same inverter. This means that it is also testable for people with only one solar-edge inverter.

So my request is if you can try to reproduce this. Assuming you have one inverter, copy the hardware twice in domoticz and try to start domoticz a few times in a row. For reference: use the command /var/www/domoticz/domoticz -www 8080 -sslwww 443 -syslog -loglevel status,error
In my case it crashes about 3 out of 5 times.

Bert

[voyo]

Hi!
I have same issues with mine domoticz instance (v2024.7 ) , fortunately it is running on docker thus it is respawning automatically.
Beside of that - exactly same symptoms. And I also have few modules which are communicating over TCP Modbus.
I'm quite sure I didnt had this problem with some old Domoticz version, it started only after I upgraded.
It looks to me like issues with multithreading and the way how Domoticz is handling python plugins, some race condition maybe ?
Brjhaverkamp, did you manage somehow to solve this problem ?

[voyo]

I had a free weekend , and started to implement some kind of locking mechanism for my Modbus plugins, hopping that it should help with these crashes related to my modbus-python-plugins. In order to to not break my production deployment, I installed fresh domoticz instance in docker,
shortly speaking - for my testing I actually tried domoticz-beta ( Version: 2025.1 (build 16611) ), and apparently it was not crashing , what I confirmed on my production configuration and all plugins.
Since it was newly created docker I also compared python and its modules - both environments have same
pymodbus==3.8.6
pyModbusTCP==0.3.0

So... this issue seems solved in new Domoticz version. Whatever it was. Probably I'v spent few hours completely unnecessary implementing threading-safe locking mechanism for my modbus plugins. Such as life
Unfortunately I was not able to find related topic on forum or issue on github to confirm my observations.