Systematiq crash of Python Plugin

[pipiche]

Domoticz 2022.1 build (14275)

Code: Select all

Mar 15 09:29:33 rasp domoticz[1681]: Thread 0xab10a330 (most recent call first):
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 320 in wait
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/queue.py", line 171 in get
Mar 15 09:29:33 rasp domoticz[1681]:   File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigateTransport/forwarderThread.py", line 31 in forwarder_thread
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 946 in run
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 966 in _bootstrap
Mar 15 09:29:33 rasp domoticz[1681]: Thread 0xab90b330 (most recent call first):
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 320 in wait
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/queue.py", line 171 in get
Mar 15 09:29:33 rasp domoticz[1681]:   File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigateTransport/writerThread.py", line 56 in writer_thread
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 946 in run
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 966 in _bootstrap
Mar 15 09:29:33 rasp domoticz[1681]: Thread 0xac10c330 (most recent call first):
Mar 15 09:29:33 rasp domoticz[1681]:   File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigateTransport/readSerial.py", line 129 in serial_read_write_from_zigate
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 946 in run
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 966 in _bootstrap
Mar 15 09:29:33 rasp domoticz[1681]: Thread 0xacae2330 (most recent call first):
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 320 in wait
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/queue.py", line 171 in get
Mar 15 09:29:33 rasp domoticz[1681]:   File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/LoggingManagement.py", line 441 in logging_thread
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 946 in run
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
Mar 15 09:29:33 rasp domoticz[1681]:   File "/usr/lib/python3.10/threading.py", line 966 in _bootstrap
Mar 15 09:29:33 rasp domoticz[1681]: Current thread 0xafcfe330 (most recent call first):
Mar 15 09:29:33 rasp domoticz[1681]:   <no Python frame>
Mar 15 09:29:33 rasp domoticz[1681]: Extension modules: Domoticz (total: 1)
Mar 15 09:29:33 rasp domoticz[1681]: 2022-03-15 09:29:33.762  Error: Domoticz(pid:1681, tid:1703('Zigate V2')) received fatal signal 11 (Segmentation fault)
Mar 15 09:29:33 rasp domoticz[1681]: 2022-03-15 09:29:33.763  Error: siginfo address=0x691, address=0xaf339c3c
Mar 15 09:29:39 rasp kernel: i2c-bcm2835 3f805000.i2c: i2c transfer timed out

pp.zip

(8.42 KiB) Downloaded 31 times

[Dnpwwo]

@pipiche,

Couple of things:

• I see you are using Python 3.10. I didn't know (for sure) that this worked so thanks for testing it
• Your Python threads look fine btw although its good to see that they are being dumped

The actual crash of course, is caused by an incoming HTTP message:

Code: Select all

Error: > #8  0xb5fd5f48 in PyDict_SetItemString () from /lib/libpython3.10.so
Error: > #9  0x00c015ec in Plugins::CPluginProtocolHTTP::ProcessInbound(Plugins::ReadEvent const*) ()
Error: > #10 0x00be6eec in Plugins::CPlugin::ConnectionRead(Plugins::CPluginMessageBase*) ()
Error: > #11 0x00c0df40 in Plugins::ReadEvent::ProcessLocked(Plugins::CPlugin*) ()
Error: > #12 0x005afc6c in Plugins::CPluginMessageBase::Process(Plugins::CPlugin*) ()
Error: > #13 0x00be1e7c in Plugins::CPlugin::Do_Work() ()

This is most likely due to the content of the message rather than a cross thread locking issue because 'ProcessLocked' will ensure that the thread has the Python GIL and should be able to allocate memory (in PyDict_SetItemString) without issue.

If this is a response from a web server can you check to see if there are any really large cookies? There was a report a while ago of an HTTP response that crashed Domoticz every time but I couldn't reproduce it. From memory it was a Proximus plugin.

[pipiche]

Hi mate,

Surprising indeed to see python3.10
This is on a rpi3b+ with Fedora 34.

In any case domoticz is entirely build on this platform


I'll investigate the HTTP request as this is coming from a client we developed ourselves.

What is strange is that I don't get this issue on other platform. But will investigate further.

Thanks for looking after the crash.
Is there any limitation in term of size for the HTTP request ?

[pipiche]

I did investigate and don't find any issue. As the same plugin works really great on an other platform.

the setup is the following

the plugin runs a thread which is the WebServer listening on HTTP. And that is my understanding of thread 13

here is how the connection is open

Code: Select all

    # self.httpPort = '9440'
    self.httpServerConn = Domoticz.Connection(
        Name="Zigate Server Connection", Transport="TCP/IP", Protocol="HTTP", Port=self.httpPort
    )
    self.httpServerConn.Listen()
    self.logging("Status", "Web backend for Web User Interface started on port: %s" % self.httpPort)
 

Here are the header of the PUT request send via a browser connected to port 9440

Code: Select all

PUT /rest-z4d/1/setting-debug undefined
Host: rasp.local:9441
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json
Content-Length: 2321
Origin: http://rasp.local:9441
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

And here is the POST request

Code: Select all

{"debugLQI":{"current":0},"debugMatchId":{"current":"ffff"},"enablePluginLogging":{"current":1},"logDeviceUpdate":{"current":1},"loggingBackupCount":{"current":7},"loggingMaxMegaBytes":{"current":0},"debugBasicOutput":{"current":0},"debugBinding":{"current":0},"debugCasaIA":{"current":0},"debugCluster":{"current":1},"debugCommand":{"current":1},"debugConfigureReporting":{"current":0},"debugDanfoss":{"current":0},"debugDatabase":{"current":0},"debugDZDB":{"current":0},"debugGroups":{"current":0},"debugHeartbeat":{"current":0},"debugIAS":{"current":0},"debugInput":{"current":1},"debuginRawAPS":{"current":0},"debugLegrand":{"current":0},"debugLivolo":{"current":0},"debugLumi":{"current":0},"debugNetworkEnergy":{"current":0},"debugNetworkMap":{"current":0},"debugOTA":{"current":0},"debugoutRawAPS":{"current":0},"debugPairing":{"current":0},"debugPDM":{"current":0},"debugPhilips":{"current":0},"debugPlugin":{"current":0},"debugProfalux":{"current":0},"debugReadAttributes":{"current":0},"debugSchneider":{"current":0},"debugThermostats":{"current":0},"debugThreadCommunication":{"current":0},"debugThreadDomoticz":{"current":0},"debugThreadForwarder":{"current":0},"debugThreadWriter":{"current":0},"debugTransport":{"current":0},"debugTransport8000":{"current":0},"debugTransport8002":{"current":0},"debugTransport8011":{"current":0},"debugTransport8012":{"current":0},"debugTransportFrwder":{"current":0},"debugTransportPluginEncoder":{"current":0},"debugTransportProto":{"current":0},"debugTransportRder":{"current":0},"debugTransportSerial":{"current":0},"debugTransportTcpip":{"current":0},"debugTransportWrter":{"current":0},"debugTransportZigpy":{"current":0},"debugTransportZigpydeCONZ":{"current":0},"debugTransportZigpyEZSP":{"current":0},"debugTransportZigpyZigate":{"current":0},"debugTransportZigpyZNP":{"current":0},"debugTuya":{"current":0},"debugWebServer":{"current":1},"debugWidget":{"current":0},"debugWriteAttributes":{"current":0},"debugzclCommand":{"current":0},"debugzclDecoder":{"current":0},"debugzdpCommand":{"current":0},"debugzdpDecoder":{"current":0},"debugzigateCmd":{"current":1},"debugzigateCommand":{"current":1},"logThreadName":{"current":0},"NXPExtendedErrorCode":{"current":0},"showTimeOutMsg":{"current":0},"trackTransportError":{"current":0},"ZiGateReactTime":{"current":0}}

[freetribes]

Hello,

I have the same problem with a respberrypi4;
with any plug-in configuration. As soon as I validate the plug-in configuration, domoticz crashes.

Can you help me ?



my config:

raspberrypi4, raspbian64,

plugin version: beta6

Domoticz.log

(39.4 KiB) Downloaded 32 times

[pipiche]

Which Domoticz version are you using ? Which python3 version ?

If you are running bulleyes and having python3.9 you must use a recent beta version of Domoticz or downgrade the python3 to 3.8