Python libraries and venv

[pipiche]

It looks like more and more we could get to a point where it becomes difficult to install a specific version of a python package.
The Zigbee for Domoticz plugin is using quiet a number of specific python package and to prevent having the error message when installing a python module with

Code: Select all

 pip install 

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Here is a link to a Wiki page which tend to document how to use python virtual environment with Domoticz.

[lost]

Hello,

Thanks for the tip!
Looks there was some modifications done on plugin side recently (at least to print PYTHONPATH in use), do they explain setting only a base venv path is now enough?

I ask this question because in former tries I did a few months ago I had to modify startup script (/etc/init.d/domoticz.sh) like this (setting the same names as your Z4D wiki additions):

Code: Select all

DZ_PYTHON_VENV_NAME=Domoticz_Python_Environment
DZ_USER_NAME=pi # CHANGE OF NEEDED!!!
if [ -d /home/$DZ_USER_NAME/domoticz/plugins/$DZ_PYTHON_VENV_NAME ]; then
   DZ_VENV_SITE=$(ls -d /home/$DZ_USER_NAME/domoticz/$DZ_PYTHON_VENV_NAME/lib/python*/site-packages | tail -n1)
   export PYTHONPATH=${DZ_VENV_SITE}
fi

This aimed to setup PYTHONPATH as something like "/home/pi/domoticz/Domoticz_Python_Environment/lib/python3.11/site-packages" for instance, if python 3.11 was system most up-to-date installed version (Bookworm default).

If PYTHONPATH was not including full path up to "site-packages", looks the plugin was not able to use the dependencies installed in the venv in my setup tries and then could not be started.

One difference may be I ran the "pip3 install -r requirements.txt" from inside created and activated venv. Your recommendation to use pip with '-d' option may make a difference there?

Only 2 packages that were remaining out-of-venv tree were z4d_certified_devices ones in my attempts, but looks they are downloaded at plugin startup: Did not find a solution to enforce them to venv path at the time.

[gizmocuz]

When using Domoticz via Docker Compose there is already a virtual environment created and you can use the normal pip(3) commands.
But there is a file you can use 'customstart.sh' that is executed when you first run the docker container.
You can use this file to install any python/system modules you need.

Another approach could be to modify the Domoticz install/startup script to create/use a virtual environment underneath the Domoticz installation folder as partly outlined here above.

As it is becoming more and more complex for a new/average user to install python modules (and forcing system-wide installation is not recommend by python), the easiest approach is the Docker Compose way.
It also takes hardly any system resources.
Keeping everything in Docker containers also does not pollute a operating system and you can easily transfer/backup the containers.