ZigbeeForDomoticz plugin: failed to load 'plugin.py'

[Tuinfluiter]

Trying to get a new ConBee3 stick working with the ZigbeeForDomoticz plugin, on:
Pi4, headless, Domoticz 2024.4, 3 docker containers for Domoticz, MQTT and Zwavejs-UI
System is running well.

I followed the procedures on:
https://zigbeefordomoticz.github.io/wik ... ocker.html and
https://zigbeefordomoticz.github.io/wik ... tings.html
I have several troubles and I have a problem seeing what is in fact cause and was is effect.
But I think the first one to solve is this, what I find in the domoticz logfile:

2024-02-07 19:50:16.514 Error: Z4D-ConBee3: (Zigate) failed to load 'plugin.py', Python Path used was '/opt/domoticz/userdata/plugins/Domoticz-Zigbee/:/usr/lib/python39.zip:/usr/lib/python3.9:/usr/lib/python3.9/lib-dynload:/usr/local/lib/python3.9/dist-packages:/usr/lib/python3/dist-packages:/usr/lib/python3.9/dist-packages'.

My docker-compose.yml on /opt/domoticz contains:
volumes:
- ./config:/opt/domoticz/userdata

which, as far as I understand, means:
/opt/domoticz/config on the host is mapped on:
/opt/domoticz/userdata in the docker container
Apart from "config maps to userdata", the further directory tree is the same.
See:

rpi4@raspi4:~ $ ls -al /opt/domoticz/config/plugins/Domoticz-Zigbee/plugin.py
-rwxr-xr-x 1 root root 75218 Feb 6 21:08 /opt/domoticz/config/plugins/Domoticz-Zigbee/plugin.py

rpi4@raspi4:~ $ docker exec -it domoticz sh
# ls -al /opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py
-rwxr-xr-x 1 root root 75218 Feb 6 21:08 /opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py

The path: /opt/domoticz/userdata/plugins/Domoticz-Zigbee/ definitely exists, with plugin.py in it.
The plugin.py has execute permission.
In docker-speak this path exists. But, of course, in host-speak (in the domoticz.log file) indeed it doesn't exist.

To me this is pretty confusing, but it is now also confusing for the Pi?

What is wrong here?

Furthermore I found this (already earlier) in the domoticz logfile:

2024-02-07 12:52:17.892 Error: Z4D-ConBee3: Cannot access path: /opt/domoticz/userdata/plugins/Domoticz-Zigbee/Conf/PluginConf.txt

Indeed, regardless of the previous discussion on config vs userdata, the path exists, there are files in it, but this file is simply not present!

Any thoughts?

[pipiche]

Unfortunatly due to zigpy lib issues, the ConBee III is not yet supported on the stable branch of the plugin.
you can switch

Code: Select all

git checkout wip-develop
sudo python3 -m pip install -r requirements.txt --upgrade

Then for the rest, I think this is more docker related issues than the plugin, maybe someone with docker know-how could help

[Tuinfluiter]

OK, that is a bit of a surprise, but thanks for this good advice.
I got this working; simply the git checkout command and in docker:
delete the FIRSTRUN file, docker down and up; then the customstartup.sh script performs the update.
I observed that the requirements.txt was updated with libraries with higher version numbers.
Now the intitial startup went pretty good:
I can see the two Zigate devices:
Zigate status: has an extra entry: Ready, and
Zigate Notifications: has two extra entries: coordinator IEEEE and channel
The problems with the path to the plugin directories (config vs userdata) don't occur anymore.

The logfiles ends with:

2024-02-08 16:45:15,095 INFO :b'Switch Blue Led On'
2024-02-08 16:45:15,096 INFO :b'coordinator set to Certification : 1/CE -> CE'
2024-02-08 16:45:15,839 INFO :b'ota_scan_folder Following Firmware have been loaded '
2024-02-08 16:45:15,840 INFO :b'Plugin with Zigpy, Coordinator Conbee III firmware 0x264d0900 correctly initialized'

But what is left now at this point is:
1. I cannot access port 9440 (cannot make a connection), and in the PluginZigbee_10.log file,
directly at the second line:
2. HTTP Error 401:

2024-02-08 16:44:29,160 INFO :b'Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started'
2024-02-08 16:44:29,171 ERROR : [ MainThread] Urlopen to http://127.0.0.1:8080/json.htm?type=com ... ethardware rejected. Error: HTTP Error 401: Unauthorized

As the error message is about opening a URL, I suspect that there may be one root cause for both problems.
Maybe this has something to do with the API base URL setting (http://127.0.0.1:8080) ?
I have in Settings > Security the "Basic-Auth over HTTP" paramter enabled.

Then as a next step according to the wiki I needed to disable the interface, change "Initialize Coordinator" from True to False and enable the interface again. Then I ran into additional errors, but it is not unlikely they may be the result of the same problem mentioned above. So I'd like to get that problem resolved first and than look further.

[pipiche]

On the fact that you cannot access 9440, don't you have a firewall in place and you need to open the port ?

may be something like

Code: Select all

sudo ufw allow from 192.168.1.0/24 to any port 9440

[Tuinfluiter]

On the fact that you cannot access 9440, don't you have a firewall in place and you need to open the port ?

No. The installation is just a week old and it is not yet connected outside the LAN, so I didn't install a firewall yet.
Furthermore for the WebUI of Zwave-js I access port 8091; I didn't have to do anything to make this work.

Could is possibly be that the API base URL, which is now http://127.0.0.1:8080 should have a username / password?
Something like: http://username:[email protected]:8080 ?
Or: http://<ip-address>:8080 ?

[pipiche]

you need to check the Domoticz security wiki, Domoticz add such requirements (which on a local network doesn't make sense for me).

[Tuinfluiter]

I tried the URL with an extra username:password and also I added 127.0.* to the trusted networks list.
Both had the same result:
There are no http-errors or urlopen-errors anymore.

2024-02-10 20:44:18.649 Status: Z4D-ConBee3: Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started
2024-02-10 20:44:18.653 Status: [web:8080] Incoming connection from: 127.0.0.1

There is an incoming connection; the plugin is started and it does a bunch of initialisations
So far, so good.
But then there is a stackdump:

2024-02-10 20:44:18.977 Status: Z4D-ConBee3: 1 Entries loaded from /opt/domoticz/userdata/plugins/Domoticz-Zigbee/Data/DeviceList-10.txt
2024-02-10 20:44:18.981 Status: Z4D-ConBee3: Transport mode: ZigpydeCONZ
2024-02-10 20:44:19.119 Error: Z4D-ConBee3: Call to function 'onStart' failed, exception details:
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: Traceback (most recent call last):
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: File "/opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py", line 1515, in onStart
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: _plugin.onStart()
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: File "/opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py", line 548, in onStart
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: start_zigbee_transport(self )
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: File "/opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py", line 970, in start_zigbee_transport
2024-02-10 20:44:19.123 Error: Z4D-ConBee3: _start_zigpy_deConz(self)
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/opt/domoticz/userdata/plugins/Domoticz-Zigbee/plugin.py", line 1075, in _start_zigpy_deConz
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: from zigpy.config import (CONF_DEVICE, CONF_DEVICE_PATH, CONFIG_SCHEMA, SCHEMA_DEVICE)
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/zigpy/config/__init__.py", line 37, in <module>
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: from zigpy.config.validators import (
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/zigpy/config/validators.py", line 9, in <module>
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: import zigpy.zdo.types as zdo_t
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/zigpy/zdo/__init__.py", line 10, in <module>
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: import zigpy.util
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/zigpy/util.py", line 13, in <module>
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: from cryptography.hazmat.primitives.ciphers import Cipher
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in <module>
2024-02-10 20:44:19.124 Error: Z4D-ConBee3: from cryptography.hazmat.primitives.ciphers.base import (
2024-02-10 20:44:19.125 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in <module>
2024-02-10 20:44:19.125 Error: Z4D-ConBee3: from cryptography.exceptions import (
2024-02-10 20:44:19.125 Error: Z4D-ConBee3: File "/usr/local/lib/python3.9/dist-packages/cryptography/exceptions.py", line 9, in <module>
2024-02-10 20:44:19.125 Error: Z4D-ConBee3: from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2024-02-10 20:44:19.125 Error: Z4D-ConBee3: ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576
2024-02-10 20:44:30.858 RFXcom: Temp + Humidity (Badkamer temp)

Then this error, keeps repeating:

2024-02-10 20:45:28.076 Error: Z4D-ConBee3: [ 70] I have hard time to get Coordinator Version. Mostlikly there is a communication issue

It looks like the plugin automatically stops:

2024-02-10 20:46:19.160 Status: Z4D-ConBee3: Stop directive received.
2024-02-10 20:46:19.160 Z4D-ConBee3: onStop()

and starts again:

2024-02-10 20:46:19.361 Status: Z4D-ConBee3: Started.
2024-02-10 20:46:20.028 Status: Z4D-ConBee3: Initialized version 7.2, author 'pipiche38'
2024-02-10 20:46:20.030 Status: Z4D-ConBee3: Zigbee for Domoticz plugin starting
2024-02-10 20:46:20.082 Status: Z4D-ConBee3: Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started

It looks like every two minutes the plugin is started automatically again:

2024-02-10 20:44:18,648 INFO :b'Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started'
2024-02-10 20:46:20,082 INFO :b'Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started'
2024-02-10 20:48:21,071 INFO :b'Zigbee for Domoticz (z4d) plugin wip-develop-7.2.005 started

I do see the two devices being added to the utility page.
But I still can't access the port 9440.
Any thoughts on what's wrong now?

[pipiche]

can you do that ? This look similar to https://zigbeefordomoticz.github.io/wik ... er-process

Code: Select all

sudo python3 -m pip install cryptography==40.0.2 --upgrade

[Tuinfluiter]

Well, that did the trick! Thanks a lot!
It all started up smoothly now. Not one error message anymore.
The log looks clean.
The two widgets look good:
Zigate status: ready
Zigate notifications: channel 25

There is one thing left now:
I still cannot access port 9440 to get to the configuration page.
The log says:

2024-02-11 11:54:11.461 Status: Z4D-ConBee3: Web backend for Web User Interface started on port: 9440

So that's OK
But neststat says:

rpi4@raspi4:~ $ sudo netstat -tulpn | grep :8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1265/docker-proxy
tcp6 0 0 :::8080 :::* LISTEN 1289/docker-proxy
rpi4@raspi4:~ $ sudo netstat -tulpn | grep :8091
tcp 0 0 0.0.0.0:8091 0.0.0.0:* LISTEN 1263/docker-proxy
tcp6 0 0 :::8091 :::* LISTEN 1290/docker-proxy
rpi4@raspi4:~ $ sudo netstat -tulpn | grep :9440
rpi4@raspi4:~ $ sudo netstat -tulpn | grep :443

Port 8091 is used for Zwavejs-UI and I can access that. But according to neststat port 9440 is not open.
Can't find anything about this.
Do I need do something special for this?
A port number in the 8000-8099 range?

[Tuinfluiter]

I just found something:
The port needs to be specified in de docker-compose.yml file.
Now it only contains port 8080.
The port for the Zwavejs-UI config page is in another docker.compose.yml, so when looking in the file for domoticz you don't get triggered that multiple ports may be needed for the domoticz docker application.
Edit: corrected typo of the word port

[pipiche]

You means that the wiki here needs also to be updated https://zigbeefordomoticz.github.io/wik ... ocker.html

[Tuinfluiter]

Not sure yet, I didn't get it working yet

[Tuinfluiter]

At the moment that I tried to add the port 9440 to the docker-compose.yml file, I ran in an other problem that the plugin.py file could not be found. Again something to do with paths that are different from the host and inside the docker container (like I had before). Couldn't find what caused this.
But at that moment I decided to stop using domoticz in docker. So I restored a backup with only the OS and started all over again with bare metal Domoticz. (I still have docker for MQTT and Zwavejs-UI). I'm now at the point that I can install the Z4D plugin.
But it means that I cannot answer the question if the port 9440 needs to be added and the docker-compse.yml file and thus that an update of the wiki is needed.

[Tuinfluiter]

I performed the bare metal installation of domoticz in stead of the previous installation of domoticz in docker.
Pi4, Bookworm, Domoticz 2024.4
Then installed the Zigbee-for-Domoticz plugin. Used the wiki procedure, with the following modifications as proposed by pipiche:
after the git clone command, switch to the development version with the command:

git checkout wip-develop

With:

sudo pip3 install -r requirements.txt --break-system-packages

the cryptography-42.0.2 library is installed. I ran into the similar stackdump as discussed above.
To force downgrade to cryptography-40.0.2 library I needed to add the --break-system-packages switch:

sudo python3 -m pip install cryptography==40.0.2 --upgrade --break-system-packages

After that the enabling of Z4D went smooth; no errors anymore.
The webUI with configuration was available on port 9440 right away.
I could pair an IKEA Tradfri wallplug.
The first modification was necessary for support of the ConBee III stick.
The second modififaction is probably necessary due the Bookwork OS version.

[Tuinfluiter]

@pipiche:
Thanks to your advice, I'm running Z4D for a year now without problems
Because of support of the Conbee3 stick, I installed the plugin in februari 2024 from the development branch, with:

Code: Select all

$ git clone https://github.com/zigbeefordomoticz/Domoticz-Zigbee.git
$ git checkout wip-develop
$ sudo python3 -m pip install -r requirements.txt --upgrade

I didn't do any maintenance in the mean time.
This means I'm still on version 7.2.006 of the plugin.
I'd like to update the plugin the the latest verion now and move to the stable branch.
So that's why my request some advice on how to move foreward:

First of all: I assume the Conbee3 stick is supported in the stable version of the plugin now?

1) am I correct that if I want to update to the current stable version, I can do:

Code: Select all

$ git checkout stable7
$ git pull
$ sudo python3 -m pip install -r requirements.txt --upgrade

2) When I installed the development version, the last command installed the cryptography-42.0.2 library.
I think that was because the OS I'm running is Bookworm.
This crypto library caused a stackdump. That problem was then solved by downgrading this library to version 40, with:

Code: Select all

$ sudo python3 -m pip install cryptography==40.0.2 --upgrade --break-system-packages

So my question is: when I do step (1), will I run into the problem with the crypto library again? In other words: do I need to downgrade from version 42 to version 40 then again with step (2)?

[Rembo]

@pipiche:
Thanks to your advice, I'm running Z4D for a year now without problems
Because of support of the Conbee3 stick, I installed the plugin in februari 2024 from the development branch, with:

Code: Select all

$ git clone https://github.com/zigbeefordomoticz/Domoticz-Zigbee.git
$ git checkout wip-develop
$ sudo python3 -m pip install -r requirements.txt --upgrade

I didn't do any maintenance in the mean time.
This means I'm still on version 7.2.006 of the plugin.
I'd like to update the plugin the the latest verion now and move to the stable branch.
So that's why my request some advice on how to move foreward:

First of all: I assume the Conbee3 stick is supported in the stable version of the plugin now?

1) am I correct that if I want to update to the current stable version, I can do:

Code: Select all

$ git checkout stable7
$ git pull
$ sudo python3 -m pip install -r requirements.txt --upgrade

2) When I installed the development version, the last command installed the cryptography-42.0.2 library.
I think that was because the OS I'm running is Bookworm.
This crypto library caused a stackdump. That problem was then solved by downgrading this library to version 40, with:

Code: Select all

$ sudo python3 -m pip install cryptography==40.0.2 --upgrade --break-system-packages

So my question is: when I do step (1), will I run into the problem with the crypto library again? In other words: do I need to downgrade from version 42 to version 40 then again with step (2)?

Yes, you can update to the stable version by following the steps you mentioned. As for the cryptography library issue, there's a chance you might encounter the same problem after switching to the stable version, especially if the system still uses version 42. If that's the case, it's best to downgrade it to version 40 again, just like you did before. It’s worth checking after the update.

[Tuinfluiter]

I tried to perform the steps; I succeeded, but with some "hurdles":

Code: Select all

$ git checkout stable7
error: Your local changes to the following files would be overwritten by checkout:
	plugin.py
Please commit your changes or stash them before you switch branches.

Apparently git thinks that I modified the plugin.py file, which I didn't (never even opened the file).
Anyway, I "stashed" it:

Code: Select all

$ git stash
Saved working directory and index state WIP on wip-develop: e64b069c Plugin wip-develop version 7.2.006

Now again the checkout, pull and python3:

Code: Select all

$ git checkout stable7
Switched to branch 'stable7'
Your branch is behind 'origin/stable7' by 561 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
$ git pull
$ sudo python3 -m pip install -r requirements.txt --upgrade

gives an error message: externally-managed-environment
The message suggests to use --break-system-packages, which I did:

Code: Select all

$ sudo python3 -m pip install -r requirements.txt --break-system-packages

OK now,
the crypto package was not updated, so I didn't have to downgrade it again.

Restart domoticz and the plugin comes to life at version 7.1.020
So far, so good.
I had a problem with the start of one of the temperature sensors, but this is not related to this upgrade.
I might post another message for this problem.