Python event scripts

[moroen]

After the introduction of the python plugin system, compiling with support for both the plugins and the usage of python event scripts have become impossible, due to requiring different python versions.

I have managed to create a fork which compiles the event system using python 3, and allows compilation of domoticz with both plugins and event support, but it crashes the event-system quite horribly when domoticz is run as a service, and somewhat less horribly when run from the command line. I suspect a kind of memory hole or something to do with file handling, but been unable to trace it down within my current time constraints...

I've worked around this by updating my python scrips to run from "On Action" and "Off action", so the acute need for python event scripts is no more, but I'm wondering if there are any interest in using python events AND plugins, before committing more time to try updating the event system to use python 3, or if this is work in progress by someone else.

M

[Dnpwwo]

@moroen,

It certainly makes sense to move the event scripting to the same version of Python and add it to the main code base so that it is available to a wider audience.

@gizmocuz and I discussed this a while ago but I have been waiting for the Python framework to stabilize before taking this on so its great that someone else wants to get involved

From memory, the existing functionality relies on Boost Python rather than direct calls to Python libraries. What approach did you take?

Can I suggest that you switch to the Python-Plugins branch in git and put your changes there, that way I can easily have a look and help resolve your issues.

[moroen]

Thanks for the reply!

Actually I've tried both approaches...

First I took the existing code, updating the code using python 3 style module initializing etc, using boost compiled with python3. When that failed when domoticz was run as a service, I discarded most of the old code, and tried using "pure" python libraries. I got as far as creating the log-function, and a script that just logs the name of the changed device. That also crashes the event-system as well after a while, and I kind of gave up...

I guess porting the old code requires the least work, since namespaces are already in place, coding using python3-libraries will probably require a lot more work, but I kind of like the idea of creating a module using just python3.

I'm not sure how to interpret the following, but I have discovered that domoticz takes longer to crash when the domotics directory are stored on a USB-drive, rather than on the raspberry pi SD-card, so I guess it might have something to do with file access.

I'll make sure it still compiles and works (for a while) using my latest changes, and then make pull request into the python-plugins branch!

M

[Dnpwwo]

@moroen,

I thought about this a bit more:

• You will need to use the 'pure' python calls version. Gizmocuz did not want to force everyone to install Python so the plugin framework links to Python at runtime and disables itself if Python is not found. I believe the boost python implementation links directly to Python and Domoticz will not start if it is not found. Have a look at DelayedLink.h to see the currently mapped Python functions, more can be added if required.
• To avoid fighting with the Python Global Inter-Lock (GIL) the current framework implementation is single threaded and you will need to work in with this. Given the GIL is not being used there will need to be a synchronization mechanism between the to systems like a mutex to avoid runtime errors

[moroen]

Ok, I'll continue trying with "pure python", and take a closer look at how it's implemented in the plugin-system. Using mutex sounds a bit over my head at the moment, my C++ skills are not that great, but I'll look into it!

M

[Dnpwwo]

@moroen,

I can do the mutex bit, its pretty straight forward.

From memory most of the event scripting code is around setting up Python and building various dictionaries. If you have a look in Plugins.cpp at CPlugin::HandleStart() and CPlugin::HandleInitialise() there might be a fair bit of code you can copy.

[moroen]

@Dnpwwo, thanks!

Just to give a small status update, I've managed to create a basic DomoticzEvents module, just containing the "changed_device_name"-object and logging, testing for stability now. I've made a lot of assumptions that probably warrants further discussion, but at least the ball is rolling...

M

[moroen]

A update:

I've opened a pull-request for the first version of the python3 event-script module. It lacks some of the dictionaries from the old module (temperature, rain etc), this will be added when I know the viability of the current approach. It's currently not possible to compile with support for event-scripts without also compiling with support for python plugins. Two small scripts are included to test and show the API.

Hopefully this will be a starting point for further development.

M

[Dnpwwo]

@moroen,

Thanks, 17 files is easier to deal with !

I have suggested to gizmocuz that we consolidate the USE_PYTHON_PLUGINS and ENABLE_PYTHON switches because they only existed to handle the different Python versions. Now no longer the case.

Maybe you could do that if you have time?

Otherwise I'm happy to merge it to the Plugin branch and have a look at making them work together.

[Dnpwwo]

@moroen,

Can you change your merge request to be into Domoticz:Python-Plugins rather than Domoticz:master? That would make it easier to make sure that it all works together before release to all beta users.

[moroen]

@Dnpwwo,

A new pull-request into Python-Plugins has been created.

I'll take a look at merging the USE_PYTHON_PLUGINS and ENABLE_PYTHON switches, I would guess calling it ENABLE_PYTHON is ok?

M

[Dnpwwo]

@moroen,

I'm happy with either ENABLE_PYTHON or PYTHON_ENABLED whatever matches the other build switches best.

Its should be enabled by default given that users don't need to have Python installed to to use Domoticz (only need it to build from source).

The test on whether to show python on the Script page needs to be updated from PYTHON_ENABLED to Py_Initialized() to reflect if Python 3.4+ was found on the user's system.

[moroen]

@Dnpwwo,

I'll take a closer look on the other switches and try to consolidate them!

M

[Dnpwwo]

@moroen,

I merged your pull request into the Python-Plugins path and had a look at it on the weekend.

It looks pretty good overall but I had a number of questions/comments:

• It doesn't compile for me on Windows, it passes the checks on github because ENABLE_PYTHON is disabled by default. When I enabled it I had errors. I commented out a few bits but it didn't link either. Do you have a VS 2017 setup? Most of the devs seem to use that first then confirm functionality on Linux.
• There seemed to be some references back to boost python. I believe that will be removed completely, wasn't sure what used it apart from the event system
• I didn't see use of a Python interpreter in your code and you will need to use one. Python 3 uses interpreters to create isolated Python environments within the program it is embedded in. I use this to keep plugins separated from each other and the event system needs to be separated from the plugins in the same way. Have a look at Plugins::Initialise, its pretty easy to create one rather than initialising the whole Python engine.
• From looking at your sample Python library so seem to be suggesting a move away from the commandArray structure.
  You should talk to @gizmocuz about that. Currently Blockly and Lua work the same way (as does the previous Python implementation) so he may have a view on the user impact of a new approach

If you can get a working version on Windows with the ENABLE_PYTHON and USE_PYTHON_PLUGINS switches merged I can have another look.

Might be worth you getting on domoticz.slack.com chat group so that its easier to talk.

[moroen]

@Dnpwwo,

Thank you for the feedback!

I currently don't have access to VS2017, as I do all my work and developement on macOS, and using debian linux to test (and run my production domoticz environment). I do believe that the failure to link on Windows is due to the fact that EventsPythonDevice.cpp and EventsPythonModule.cpp are not added to the project. I'll try to get my hands on a Windows installation and VS2017, and see if I can manage to compile with python on Windows.

References to boost::python should all be commented out, as far as I can tell, and I'm able to compile on macOS and Debian without libboost_python* installed, but I'll take another look, and clean the references out completely!

The need to use a separate interpreter escaped me, and now that you mention it, naturally makes totally sense, I'll implement it before issuing a new pull request.

I'm not sure I follow you on the last point, using the domoticz helper-module for 2.7 you didn't have to work with the commandArray, using the command-function you could send commands directly:

Code: Select all

def command(name, action, file):
	if testing:
		commands.append((name, action))
	else:
		event_system.command(name, action, file)

I tried to mimic this in the new module, all my old scripts now run after updating the syntax to python3 and change all calls for domotic.command(name, action, file) to DomoticzEvents.Command(name, action). I do see the point of keeping the syntax as consistent between the different scripting languages as possible, so I have no reservations against requiring building and passing a commandArray instead!

The switches has been merged in my latest build, I didn' t really find a consistent usage of build-flags, so I went for ENABLE_PYTHON. Now to get hold of a version of Windows in a virtual machine...

M

[Dnpwwo]

@moroen,

I pushed a commit yesterday that has a VS file with the files added, I also changed quite a lot of my code so please pull the branch again.

I wasn't very familiar with the previous implementation details so if you are copying that then I'm sure its all good.