Python plugins: Using venv

Python and python framework

Moderator: leecollings

Post Reply
lost
Posts: 643
Joined: Thursday 10 November 2016 9:30
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Python plugins: Using venv

Post by lost »

Hello,

Something I find a bit weird with python plugins is saving all your domoticz setup for full reinstall is no more as simple as making an archive of the whole domoticz tree...

You'll have to rebuild all plugin stuff because all dependencies they install will be in some system (sub)directories like:
/usr/lib/python3/dist-packages

=> Need to do the whole plugin install procedure again & restoring their specific config files afterwards... Not so easy to remember after a few months/years when an issue occurs.

A (light, don't talk me about Docker!) way to solve this issue may be using python venv (natively supported since python3), aka virtual environments.

Unfortunately, Domoticz don't support this for now but it's at least possible to cheat (with the limit that only one venv may be setup, shared by all plugins but this should not be a big issue IMHO)!

Once a venv is setup, best way to use it should be calling the python interpreter (copy or link to system one):

To enforce this, I tried modifying /etc/init.d.domoticz.sh script to add venv bin path (venv python interpreter location): No success. A plugin installed from a venv (see later Z4D example) will not start because required sys.path does not look correct/modified.

Another solution could be to alias native python to the venv ful path location: Did not worked either.

Only change that proved to work was adding (with venv named domoticzPythonVenv at creation done in ~/domoticz/plugins directory):

Code: Select all

DZ_PYTHON_VENV_NAME=domoticzPythonVenv
DZ_USER_NAME=pi
if [ -d /home/$DZ_USER_NAME/domoticz/plugins/$DZ_PYTHON_VENV_NAME ]; then
   #export PYTHONPATH="/home/$DZ_USER_NAME/domoticz/plugins/$DZ_PYTHON_VENV_NAME/lib/python3.11/site-packages"
   DZ_VENV_SITE=$(ls -d /home/$DZ_USER_NAME/domoticz/plugins/$DZ_PYTHON_VENV_NAME/lib/python*/site-packages | tail -n1)
   export PYTHONPATH=${DZ_VENV_SITE}
fi
To have PYTHONPATH set to something like /home/pi/domoticz/plugins/domoticzPythonVenv/lib/python3.11/site-packages

Maybe a way to enforce which python interpreter to call in domoticz config (or even for any found plugins, as this would remove the limit of 1 shared venv) may be doable to avoid this kind of hack?

Anyway, as an example, Z4D plugin install in a venv should then be (instead of wiki procedure):

Code: Select all

Save previous Z4D cfg, if needed (after stopping domoticz service):

cd ~/domoticz/plugins
tar -czvf Z4D_CfgSav.tar.gz Domoticz-Zigbee/Conf/PluginConf-*.json Domoticz-Zigbee/Data/ Domoticz-Zigbee/Reports/

Install, using python3 native venv:

cd ~/domoticz/plugins
python3 -m venv domoticzPythonVenv/ --upgrade-deps

( if error/previous non venv install : sudo \rm -rf domoticzPythonVenv/ /usr/local/lib/python3.11/dist-packages/z4d_*  & retry )

source ./domoticzPythonVenv/bin/activate

git clone https://github.com/zigbeefordomoticz/Domoticz-Zigbee.git
cd Domoticz-Zigbee/
pip3 install -r requirements.txt

git config --add submodule.recurse true
git submodule update --init --recursive

chmod +x plugin.py

( If previous Z4D config save : cd ~/domoticz/plugins && tar -xvf Z4D_CfgSav.tar.gz )

deactivate (=> exit venv)

In /etc/init.d/domoticz.sh, add (just after the few commented lines at the beginning of the script file):

DZ_PYTHON_VENV_NAME=domoticzPythonVenv
DZ_USER_NAME=pi
export PYTHONPATH=$(ls -d /home/$DZ_USER_NAME/domoticz/plugins/$DZ_PYTHON_VENV_NAME/lib/python*/site-packages | tail -n1)

sudo service domoticz (re)start
Just change DZ_USER_NAME if not using usual pi user & DZ_PYTHON_VENV_NAME if name used at venv creation is changed.

Take care, if installing other plugins, to proceed after venv activate if you want their specific dependencies setup in the same way.

Did not tried for now the plugin Web UI upgrade procedure with this setup. I'm not sure upgraded python libraries will then be updated in the venv as they would for sure be with some Domoticz updates to handle this use case. For Z4D, I think I'll stick to the command line git based update procedure.

I'm not really a python expert, so if there is a better way to proceed feel free to answer.
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests