Using Docker and userscripts (Lua, DzVents, Python) Topic is solved

[waltervl]

I just want to bring up a discussion about using user scripts (Lua, python, DzVents) running outside the internal eventsystem and in a Domoticz Docker. This install gets more attention but it also should be easy to maintain by the user. Let us see if we come to a possible solution to configure Domoticz and the Docker image so it will work easily. I hope experience Docker users can shine a light on this.

Use case: When using the Domoticz Docker install users have difficuly to setup and later update their own users scripts (lua, Python, DzVents). See viewtopic.php?f=4&t=35454

Currently advised is to create a folder on the host (eg ./config) and map that to /opt/domoticz/userdata in the docker.
When launching the docker container for the first time, a plugin folder is created in the userdata folder You need to place your python plugins in this folder.
When updating the Docker image the plugins are not erased by the new image. Everything OK!
Also the database is created in the userdata folder so no issues when updating.

But what with the users scripts?
You cannot create a scripts folder in userdata because there is already a scripts folder that contains all kind of script that are needed to run Domoticz (and DzVents etc). So everytime a new image is imported the scripts folder will be overwritten by the Docker image.

Questions:
1. Is this the correct conclusion? Will userscripts be overwritten in an Domoticz Docker update?
2. Is there a Docker configuration way that some kind of userdata scripts folder can be merged with the scripts folder in the Docker image?
3. Are the lua, python and DzVents scripts from the eventsystem really stored in the database and no issues when updating?

Possible solution I see when the questions above give no satisfying solution and that would have impact on the Domoticz sources/configuration:
Is it possible to create a userscripts folder in userdata (like the plugins folder) and have Domoticz also look at that folder for scripts next to the original scripts and the scripts from the eventsystem. So at startup of Domoticz deamon look at 2 folders scripts and userscripts ?

[waaren]

Questions:
3. Are the lua, python and DzVents scripts from the eventsystem really stored in the database and no issues when updating?

dzVents scripts created with the internal event editor are stored in the database and so part of the backup. Wherever the database is, these script are.
dzVents also allow scripts to be placed in the <domoticz dir>/scripts/dzVents/scripts folder. These scripts are not stored in the database and a backup of these scripts is a user responsibility.
Because dzVents is Lua any script can also use separate utility or data modules and they can live at any place on the user system. It is completely up to the creator of the script where to put these.

Hope this clarifies the dzVents part.

[waltervl]

dzVents scripts created with the internal event editor are stored in the database and so part of the backup. Wherever the database is, these script are.

Are the Blockly, Lua and Python scripts created with the internal event system also stored in the database or is this mechanism only for dzVents scripts?

[YOYO]

Hi Waaren,

i got a question about the quote below:

dzVents scripts created with the internal event editor are stored in the database and so part of the backup. Wherever the database is, these script are.
dzVents also allow scripts to be placed in the <domoticz dir>/scripts/dzVents/scripts folder. These scripts are not stored in the database and a backup of these scripts is a user responsibility.

If the above would be true how come i could create a dzVents script through the internal editor and then backup the script separately from the location mentioned?
Because i managed to do that in the Linuxserver docker image.
The problem as walter mentioned is that we cant get the generated scripts back into domoticz using the official docker image.

Cheers,


Edit:

Small correction in the backup i only see the .LUA file.
i'm not sure it would behave the same although i would suspect i should be able to see them.

[waaren]

If the above would be true how come i could create a dzVents script through the internal editor and then backup the script separately from the location mentioned?

Good question !

Because the Lua interpreter responsible for the execution of the dzVents runtime modules and the for dzVents userscripts, the userscripts stored in the database are written to the domotcz server filesystem by domoticz before they can be executed. This happens on every save of a scripta and every night at 00:00 and every (re) start of domoticz.


Hope this clarifies.

[waaren]

dzVents scripts created with the internal event editor are stored in the database and so part of the backup. Wherever the database is, these script are.

Are the Blockly, Lua and Python scripts created with the internal event system also stored in the database or is this mechanism only for dzVents scripts?

They are all stored in the database.

Code: Select all

cd <domoticz dir>
sudo sqlite3 domoticz.db "select interpreter, count(*) as 'number of scripts' from eventMaster group by interpreter"

Interpreter|number of scripts
Blockly|11
Lua|13
Python|1
dzVents|91

btw. Mechanism to execute the Lua and Blockly scripts is different then the way it works for dzVents.
Don't know much about Python scripts and how they are handled.

[YOYO]

Good question !

Because the Lua interpreter responsible for the execution of the dzVents runtime modules and the for dzVents userscripts, the userscripts stored in the database are written to the domotcz server filesystem by domoticz before they can be executed. This happens on every save of a scripta and every night at 00:00.

Hope this clarifies.

Hi Waaren,

Yes it does a litle bit.
I think i have to see it in the way that dzVents needs a place to transform from the dzvents format to the correct LUA format.
The result of that transformation is placed in the

Code: Select all

<domoticz dir>/scripts/dzVents/generated_scripts

before its been sended to the database.

Experimenting further with scripts on the official docker image i notice that blocky scripts are generated ok but on the dzvents part it fails with:

Code: Select all

 2021-03-18 21:19:18.108 Status: dzVents: Write file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua
2021-03-18 21:19:18.108 Error: EventSystem: problem writing file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua 

i suspect this is because of a issue with read/writing rights in the container.
I tried to set the correct rights through putty with:

Code: Select all

docker exec -it DomoTest chmod -R 777  /opt/domoticz/userdata/

But even that doesn't allow me to write the script.
Could a permission issue be the reason of not seeing my scripts?


Cheers,

[waaren]

I think i have to see it in the way that dzVents needs a place to transform from the dzvents format to the correct LUA format.
The result of that transformation is placed in the

Code: Select all

<domoticz dir>/scripts/dzVents/generated_scripts

before its been sended to the database.

No it is the other way around. The result of the edit session is stored in the database as one long string (type TEXT in sqlite)
When the script is set to On in the event editor. The TEXT field of the database record is written as file to <domoticz dir>/scripts/dzVents/generated_scripts where it will be handled by the dzVents run time module.

I have no knowledge at all on docker so cannot help with that,

[EddyG]

I use dzVents only in the 'external' version. I checked the 'internal' version which are saved in <domo dir>/scripts/dzVents/generated_scripts
And both work in my system I did not check blocky, python and lua, but I suppose they work the same way.
The only problem I got was with JSON.lua vs. json.lua and other files.
Solved that, as mentioned in the other thread, and everything is working fine.
I can advise to use Portainer and Stacks (docker-compose) to get Domoticz running.
Portainer can control Docker Swarm and Kubernetes too.
It the easiest way to change settings and experiment with settings on the container.
My final configuration on the Domoticz container on Synology is:

Code: Select all

version: '3.3'

services:
  domoticz:
    image: domoticz/domoticz
    container_name: domoticz
    restart: unless-stopped
    # Pass devices to container
    devices:
    - "/dev/serial/by-id/usb-XXXX:/dev/ttyACM0"
    ports:
      - "9080:8080"
    volumes:
      - /volume1/docker/domoticz/config:/opt/domoticz/userdata
    environment:
      - TZ=Europe/Amsterdam
      - LOG_PATH=/opt/domoticz/userdata/domoticz.log
    labels:
      - com.centurylinklabs.watchtower.enable=false

The last label is just to get manual control over watchtower on the automatic updates on Domoticz.

[EddyG]

Experimenting further with scripts on the official docker image i notice that blocky scripts are generated ok but on the dzvents part it fails with:

Code: Select all

 2021-03-18 21:19:18.108 Status: dzVents: Write file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua
2021-03-18 21:19:18.108 Error: EventSystem: problem writing file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua 

i suspect this is because of a issue with read/writing rights in the container.

I think it might be a rights issue outside the container.

[waltervl]

I use dzVents only in the 'external' version. I checked the 'internal' version which are saved in <domo dir>/scripts/dzVents/generated_scripts

So when the Docker Domoticz image is updated your external dzVents scripts saved in /volume1/docker/domoticz/config:/opt/domoticz/userdata/ scripts/dzVents/scripts will not be deleted?

[EddyG]

Yes, that is correct and the whole Idea behind docker 'external'

[waltervl]

Yes, that is correct and the whole Idea behind docker 'external'

I am lost now....
What do you mean with docker 'external'? I do not see you making a reference to a docker 'external' but to dzVents external (= not in the eventsystem).

Do configure your Domoticz Docker container different than stated in the Domoticz Docker instruction?

[EddyG]

I am sorry. I mean 'external' filesysteem from docker view point.

[waltervl]

Questions:
1. Is this the correct conclusion? Will userscripts be overwritten in an Domoticz Docker update?

So the answer to my original question is: NO.

Conclusion: there is no issue.

[YOYO]

I think it might be a rights issue outside the container.

Well i looked it up in OMV but the rights there are read/write/execute for everyone.
Double checked it with putty so the rights outside the container are fine.

Also when adding a new script there is usually a example or part of the code.
They are not showing up either.

[EddyG]

Got into docker with

Code: Select all

docker exec -it domoticz /bin/bash

do a touch.

Code: Select all

touch /opt/domoticz/userdata/scripts/dzVents/scripts/text.txt

There should be no error, and exit the container.
Find where that 'text.txt' file went.

[waltervl]

Experimenting further with scripts on the official docker image i notice that blocky scripts are generated ok but on the dzvents part it fails with:

Code: Select all

 2021-03-18 21:19:18.108 Status: dzVents: Write file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua
2021-03-18 21:19:18.108 Error: EventSystem: problem writing file: /opt/domoticz/userdata/scripts/dzVents/generated_scripts/Script #2.lua 

i suspect this is because of a issue with read/writing rights in the container.

I think it might be a rights issue outside the container.

Playing around with Docker on a Ubuntu 20.04 LTS system.
I have the same error. I can touch a file with root in that folder from within the container but dzvents eventsystem is giving this error for me too. So this looks like a rights issue within the container.
Also that touched file is NOT showing up in my external file system. So after an update of the container the file is lost. For these generated scripts no issue but for the scripts in scripts/lua it is. Also those files will not show up in my external file system.
My Domoticz.db is showing up and stays after an update.
I also notice there are differences with the Synology docker implementation as I have Python installed and can use the plugin system without changes.

But I will play further....

[pvklink]

Perhaps you have to add you rpi -user to the docker group?
sudo usermod -aG docker <rpi user>

[Gravityz]

i am also doing some tests regarding the mapping of external directories to the domoticz container.
i wanted to try ro get rid of the write error

if i create the directory path on synology i notice that the write error is gone and the dzvents file is written to the correcvt synology directory.
apparently the error is caused because the container can not create directory paths
It however creates a lot of other file not found errors which is really strange
i am testing with a clean domoticz image without any devices so maybe this is wahat is causing the errors

use winscp to login the synology en create this directory path

Code: Select all

/volume1/docker/domoticz/scripts/dzVents/generated_scripts

on the docker image do the following mapping

Code: Select all

/docker/domoticz   to /opt/domoticz/userdata

i however get a lot of other errors which looks like the container does not know anymore were it's files are


so without creating the path, 1 write error, file not written

when the path is created no write error, file appears in docker/domoticz/scripts/dzVents/generated_scripts

lot's of other errors which look like an old problem. i am using the latest docker build 13149

Code: Select all

2021-04-09 18:28:00.389 Error: EventSystem: in /opt/domoticz/dzVents/runtime/dzVents.lua: /opt/domoticz/dzVents/runtime/Utils.lua:1: module 'JSON' not found:
2021-04-09 18:28:00.389 no field package.preload['JSON']
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/dzVents/runtime/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/dzVents/runtime/device-adapters/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/dzVents/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/scripts/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/../lua/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/scripts/modules/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/generated_scripts/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/data/JSON.lua'
2021-04-09 18:28:00.389 no file '/opt/domoticz/userdata/scripts/dzVents/modules/JSON.lua'
2021-04-09 18:28:00.389 no file '/usr/local/share/lua/5.3/JSON.lua'
2021-04-09 18:28:00.389 no file '/usr/local/share/lua/5.3/JSON/init.lua'
2021-04-09 18:28:00.389 no file '/usr/local/lib/lua/5.3/JSON.lua'
2021-04-09 18:28:00.389 no file '/usr/local/lib/lua/5.3/JSON/init.lua'
2021-04-09 18:28:00.389 no file '/usr/share/lua/5.3/JSON.lua'
2021-04-09 18:28:00.389 no file '/usr/share/lua/5.3/JSON/init.lua'
2021-04-09 18:28:00.389 no file './JSON.lua'
2021-04-09 18:28:00.389 no file './JSON/init.lua'
2021-04-09 18:28:00.389 no file '/usr/local/lib/lua/5.3/JSON.so'
2021-04-09 18:28:00.389 no file '/usr/lib/x86_64-linux-gnu/lua/5.3/JSON.so'
2021-04-09 18:28:00.389 no file '/usr/lib/lua/5.3/JSON.so'
2021-04-09 18:28:00.389 no file '/usr/local/lib/lua/5.3/loadall.so'
2021-04-09 18:28:00.389 no file './JSON.so'