Python Plugin Beta testers - please read this

[zak45]

in windows this create also a "cache" directory called __pycache__ with inside a precompiled script.

[zaraki673]

As you can see from the edit I made of my post, problem is solved. The import fails if you have other files in the plugin directory. This has been confirmed by @zaraki673 as well. In my case python has created a cache directory, and that caused my problems. I wonder if this problem is platform dependent. I'm running Rasbian. You develop on Windows or?

My dev is on rpi3, i have also the cache directory, but it's don't fail to start ... (even on restart service or rpi)

I have some more test to test ASAP, but i think plugin crash on domoticz restart if there are more than one file in the plugins directory, but if you start domoticz, and then put other files in plugin directory, it's run fine ... (has i said i have to test more to be sure)

[G3rard]

@G3rard,

I have seen occasional crashes when reloading plugins, those that can be reproduced reliably I have resolved.

If you can crash the Sonos plugin regularly can you send me steps you use, I have an Ubuntu VM I can try it on (assuming you are running 14.04)

I am running Domoticz on Ubuntu server 14.04 indeed.
I use this version of the Sonos plugin https://github.com/tester22/Domoticz-So ... /plugin.py.

The plugin crashes when changing when changing the Update interval or changing the Debug parameter in the Hardware page (haven't tried any other setting such as IP address).

[ubee]

My dev is on rpi3, i have also the cache directory, but it's don't fail to start ... (even on restart service or rpi)

I have some more test to test ASAP, but i think plugin crash on domoticz restart if there are more than one file in the plugins directory, but if you start domoticz, and then put other files in plugin directory, it's run fine ... (has i said i have to test more to be sure)

In most cases I can restart Domoticz without any problem even if I have other files in the plugin directories (like __pycache__). E.g, I have made several updates of my own plugin without any problem. It might be so the problem appears if you create a completely new plugin, i.e. add a new directory in the plugin folder.

[ubee]

I not that skilled in python, and it is the first time I'm using Python3. I try to use a library, jsonrpclib, in my plugin. This is Python2.7 library, and the library is not found when I try to import it. Then I copied the jsonrpclib files from /usr/local/lib/python2.7/dist-packages to /usr/local/lib/python3.4/dist-packages. By that I came one step further, but fails on other libraries.

I guess this is due to path error. Even if this is more a python related question than a plugin question, I hope anyone can help out with this.

Edit: It´s funny, I found the solution this time as well.... pip3 instead of pip was the trick. However, now I have a syntax error in one library module. python2 vs python3 compatibility issue I guess. I'm not a fan of python3 to be honest. I hope there are good reasons why this framework is built for python3.....

[Dnpwwo]

@ubee,

I don't think you can use Python 2 libraries with Python 3.

The Kodi example plugin imports the Python 3 'json' library and uses it in the onMessage callback.

[ubee]

@ubee,

I don't think you can use Python 2 libraries with Python 3.

You're right. I installed jsonrpclib-pelix instead, and that library worked fine. However, I had to modify a few line of codes in one module as a workaround for a TypeError: 'NoneType' problem. I found that solution in another forum.

Now everything works fine in my login environment, but fails when the plugin is called by domoticz running as a service. Obviously a path problem again. Import of jsonrpclib fails in that environment. It´s really tricky to understand python paths....

[ubee]

I have problems with different python paths when I execute my plugin from domoticz compared to if I test my plugin from an interactive python session.

Looking in the log file I see the following python path is used when the plugin is imported. And the import fails since one library cannot be found using this path:

• Python Path used was '/home/pi/domoticz/plugins/UniPIx/:/usr/lib/python3.4/:/usr/lib/python3.4/plat-arm-linux-gnueabihf:/usr/lib/python3.4/lib-dynload'

In my interactive session printing sys.path I get this path:

• '''/usr/lib/python3.4', '/usr/lib/python3.4/plat-arm-linux-gnueabihf', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.4/dist-packages']

My libraries are in /usr/local/lib/python3.4/dist-packages, so this explains why it doesn't work running the plugin from domoticz.

Again, my python experience could be a lot better, but how the heck do I make sure I have the right path executing the plugin from domoticz?

[zak45]

Have been able to have Kodi plugin working in windows, for that :

- kodi need to be up first
- modified plugin.py file :
added this line: Status = 0
modified this other : def onConnect():

probably not the right way, but when done like that, all Kodi devices running well, even remote.

Have also take a look to domoticz.py under scripts folder. A few errors appear as I run py 3.x , code looks like for py 2.x.
To have it run without error:
- all print() need to be changed e.g. print ("\tsetting %s On" % name) -- enclosed with ()
- and some space/tab error corrected.

so.. my last investigation !!!

[ubee]

I solved my path problem by including

• import sys
  sys.path.append('/usr/local/lib/python3.4/dist-packages')

just before the import statement where I import the support library (jsonrpclib) I need. I used pip3 to install the python library, but it seems like something goes wrong and $PYTHONPATH is not updated correctly.

I've come to the point I can control the relays on the UniPi module with my plugin. All relays show up in Domoticz, and I can turn them on. However, I have still problems to update the status of the switches (relays) in Domoticz with the UpdateDevice method. While debugging this issue I noticed the debug printout stopped in the log file and I decided to reboot the system.

And by that I'm back to the point no plugins are loaded any longer!!!!! I have gone through all the tricks I've learned the last few days like looking for any additional files in the plugin directories. I have also deleted the domoticz database to start with an empty and fresh copy. I have also reinstalled Domoticz from git. No result. Neither the Kodi plugin nor my UniPi plugin is loaded. And no fault indication whatsoever in the log file. It is completely clean. The last lines are as follows:

• 2017-02-01 09:29:28.726 PluginSystem: Entering work loop.
  2017-02-01 09:29:31.341 Hardware Monitor: Fetching data (System sensors)

This is weird. The plugins loaded perfectly before, and I have not changed one single line in the modules. I removed the Kodi plugin just as a test, but that didn't make any difference.

The only thing I have changed is the addition of a few more directories to the python path. I added '/usr/local/lib/python3.4/dist-packages'. For some reason '/usr/lib/python3/dist-packages' and '/usr/lib/python3.4/dist-packages' have been added as well, but not by me.

Is there any reason why a few additional library directories in the python path can prevent the plugin modules to be loaded?

[SweetPants]

Sure '/usr/lib/python3.4/dist-packages' exists? Not on my system

[ubee]

Sure '/usr/lib/python3.4/dist-packages' exists? Not on my system

Yes, both /usr/lib/python3.4/dist-packages and /usr/lib/python3/dist-packages are there. It all depends on what packages you have installed. However, I have never figured out what goes into those and what goes into /usr/local/lib.

[SweetPants]

Guys, just playing with the Kodi plugin, and all loads well. However is I try to turn on my Player (WOL), i get the following error:

2017-02-01 14:45:09.764 Error: (Kodi Palyers) 'onCommand' failed 'error':'byte format requires -128 <= number <= 127'.
2017-02-01 14:45:09.764 Error: (Kodi Palyers) ----> Line 315 in /home/pi/domoticz/plugins/Kodi/plugin.py, function onCommand
2017-02-01 14:45:09.764 Error: (Kodi Palyers) ----> Line 482 in /home/pi/domoticz/plugins/Kodi/plugin.py, function TurnOn

Something with singed/unsigned char/byte?

Never mind, code was based on python2. Just replaced the WOL part with:

Code: Select all

def TurnOn():
    if (Parameters["Mode1"] == ""):
        Domoticz.Error("Kodi can not be turned on, No MAC address configured.")
    else:
        import struct, socket
        # Took some parts of:https://github.com/bentasker/Wake-On-Lan-Python/blob/master/wol.py for python3 support 
        mac_address = Parameters["Mode1"].replace(':','')
        # Pad the synchronization stream.
        data = ''.join(['FFFFFFFFFFFF', mac_address * 20])
        send_data = b''
        # Split up the hex values and pack.
        for i in range(0, len(data), 2):
            send_data = b''.join([send_data, struct.pack('B', int(data[i: i + 2], 16))])
            
        # Broadcast it to the LAN.
        sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
        sock.sendto(send_data, ('<broadcast>', 7))
    return

and now it works

[avgays]

If you really need numbers that big I would suggest setting the heartbeat to 17 or 18 and doing what you want to do on every 4 call.

Dnpwwo, thank you!

Is it possible in any way to set DeviceID while creating device?

It’s look like that now it is automatically generated. Can be useful for storing device address like it is made for Blinds

[ubee]

Just to make an attempt to find a solution to my import plugin problem, I have prepared a completely fresh Raspbian system and installed Domoticz from scratch with 'sudo curl -L install.domoticz.com | bash'. After that, I have upgraded to latest beta by using ./updatebeta. So I'm running 3.6572.
Domoticz works fine.

Now I have installed one single plugin in /home/pi/domoticz/plugin/UniPIx:

• pi@raspberrypi:~/domoticz $ ls -AlR plugins/
  plugins/:
  total 4
  drwxr-xr-x 2 pi pi 4096 Feb 2 12:22 UniPIx
  
  plugins/UniPIx:
  total 4
  -rwxr-xr-x 1 pi pi 3836 Feb 2 12:36 plugin.py

The plugin is really simple. Just populated with a few line of codes. Does not refer to any external libraries, except for Domoticz. Code comes here:

Code: Select all

# UniPI plugin
#
# Author: Ubee
#
"""
<plugin key="UniPIx" name="Plugin for UniPI" author="ubee" version="1.0.0" wikilink="http://www.domoticz.com/wiki/plugins/plugin.html" externallink="https://www.google.com/">
    <params>
		<param field="Mode1" label="Debug" width="75px">
			<options>
				<option label="True" value="Debug"/>
				<option label="False" value="Normal"  default="true" />
			</options>
		</param>
    </params>
</plugin>
"""
#import sys
#sys.path.append('/usr/local/lib/python3.4/dist-packages')

import Domoticz

#from jsonrpclib import Server


UNIPI_URL="localhost:8082"



def onStart():
    Domoticz.Log("onStart called")
    if Parameters["Mode1"] == "Debug":
        Domoticz.Debugging(1)
    if (len(Devices) == 0):
        Domoticz.Device(Name="Relay 1", Unit=1, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 2", Unit=2, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 3", Unit=3, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 4", Unit=4, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 5", Unit=5, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 6", Unit=6, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 7", Unit=7, TypeName="Switch").Create()
        Domoticz.Device(Name="Relay 8", Unit=8, TypeName="Switch").Create()
        Domoticz.Log("Relay Devices created.")
#    else:
#		if (1 in Devices): SMPstate = Devices[1].nValue
#		if (2 in Devices): SMPconso = Devices[2].nValue
    DumpConfigToLog()
    Domoticz.Heartbeat(60)
    return True

	

def onStop():
    Domoticz.Log("onStop called")
    return True
 

def onConnect(Status, Description):
    Domoticz.Log("onConnect called")
    return True


def onMessage(Data, Status, Extra):
    Domoticz.Log("onMessage called")
    return True


def onCommand(Unit, Command, Level, Hue):
    Domoticz.Log("onCommand called for Unit " + str(Unit) + ": Parameter '" + str(Command) + "', Level: " + str(Level))

 #   s=Server("http://"+UNIPI_URL+"/rpc")

 #   Command = Command.strip()
 #   action, sep, params = Command.partition(' ')
 #   action = action.capitalize()
	
    if (action == 'on'):
  #      s.relay_set(unit,1)
        UpdateDevice(unit,1,'On')

    elif (action == 'Off'):	
  #      s.relay_set(unit,0)
        UpdateDevice(unit,0,'Off')	
    return True

def onNotification(Data):
    Domoticz.Log("onNotification: " + str(Data))
    return True

def onDisconnect():
    Domoticz.Log("onDisconnect called")
    return True

def onHeartbeat():
    Domoticz.Log("onHeartbeat called")
    return True

 # Generic helper functions
def DumpConfigToLog():
    for x in Parameters:
        if Parameters[x] != "":
            Domoticz.Debug( "'" + x + "':'" + str(Parameters[x]) + "'")
    Domoticz.Debug("Device count: " + str(len(Devices)))
    for x in Devices:
        Domoticz.Debug("Device:           " + str(x) + " - " + str(Devices[x]))
        Domoticz.Debug("Device ID:       '" + str(Devices[x].ID) + "'")
        Domoticz.Debug("Device Name:     '" + Devices[x].Name + "'")
        Domoticz.Debug("Device nValue:    " + str(Devices[x].nValue))
        Domoticz.Debug("Device sValue:   '" + Devices[x].sValue + "'")
        Domoticz.Debug("Device LastLevel: " + str(Devices[x].LastLevel))
    return
	
def UpdateDevice(Unit, nValue, sValue):
# Make sure that the Domoticz device still exists (they can be deleted) before updating it 
    if (Unit in Devices):
        if (Devices[Unit].nValue != nValue) or (Devices[Unit].sValue != sValue):
            Devices[Unit].Update(nValue, str(sValue))
            Domoticz.Log("Update "+str(nValue)+":'"+str(sValue)+"' ("+Devices[Unit].Name+")")
    return

The import does not work! Log file is as follows:

• 2017-02-02 12:37:01.534 Domoticz V3.6572 (c)2012-2017 GizMoCuz
  2017-02-02 12:37:01.535 Build Hash: 662b174, Date: 2017-02-01 13:16:02
  2017-02-02 12:37:01.535 System: Raspberry Pi
  2017-02-02 12:37:01.536 Startup Path: /home/pi/domoticz/
  2017-02-02 12:37:02.543 PluginSystem: Started, Python version '3.4.2'.
  2017-02-02 12:37:02.570 Active notification Subsystems: http (1/12)
  2017-02-02 12:37:02.673 WebServer(HTTP) started on address: :: with port 8085
  2017-02-02 12:37:02.765 WebServer(SSL) started on address: :: with port 443
  2017-02-02 12:37:02.769 Proxymanager started.
  2017-02-02 12:37:02.779 Starting shared server on: :::6144
  2017-02-02 12:37:02.779 TCPServer: shared server started...
  2017-02-02 12:37:02.780 RxQueue: queue worker started...
  2017-02-02 12:37:04.827 EventSystem: reset all events...
  2017-02-02 12:37:04.831 EventSystem: reset all device statuses...
  2017-02-02 12:37:04.834 EventSystem: Started
  2017-02-02 12:37:04.900 PluginSystem: Entering work loop.
  2017-02-02 12:39:12.347 Incoming connection from: 192.168.1.132

I can find "Plugin for UniPI" at the end of the dropdown list on the Hardware tab. That indicates the file is found by the plugin system, but it does not load.

I start to believe there is a problem with the latest beta, 3.6572. I remember I made an update just before I ran into this problem.
@Dnpwwo, can you check if you can load this module on 3.6572? I can't load any other plugins either like your Kodi or the Awox plugin.

?????

Edit: Tried with Version: 3.6584 as well. Same negative result....

[avgays]

I can find "Plugin for UniPI" at the end of the dropdown list on the Hardware tab. That indicates the file is found by the plugin system, but it does not load.

@ubee,
I got in the same situation several times, without changing version of Domotics. Moreover – already added plugin continues work normally, but couldn’t be updated.
In my case if I opened Domoticz from Chrome – plugin was in the end of list, if I open it from IE – plugin can be added and updated. After several days it start normally work under Chrome too. Clearing cache and hard refresh of page in Chrome doesn’t help…

[ubee]

@avgays,


I was about going nuts. Thanks from saving me that experience!

I tried with the Edge browser, and the plugin loads as a charm....

Edit: I have to give you an update on this. Actually I hade two errors that made it extremely difficult to understand what was going on. First the browser issue that @avgays gave me a hint about. That problem made it impossible to add a HW device.

The second problem was my path problem I described in an earlier post. I tried to use a Python3 version of jsonrpclib by installing jsonrpclib-pelix. I couldn't make that work without modifying sys.path in the beginning of the plugin.py. Without modifying sys.path, plugin.py didn't load since references to jsonrpclib was unknown. After modifying sys.path, the module loads, but do not execute! Nothing happens at all! No error messages and no log messages in the log file. Nothing! I noticed something was wrong since restart of the domoticz service took a lot longer than before. About 15 seconds or something.

So stay away for jsonrpclib is my advice! Python3 is a challenge and you need to spend quite some time to be able to reuse Python2 code with REST or json support. And json doesn't work at all..... What's the problem with Python2?

[Dnpwwo]

@ubee,

You should probably create a separate topic for your plugin so that your questions don't get lost in this general one.

From looking at your code the real question is not why you can't load the jsonrpc library but why you would want to. One of the most important parts of the Python framework is that it will handle connectivity for you. This make plugins easier to write but more importantly keeps them responsive if you run more than one (I run 5 currently) because the communications is done asynchronously rather than within a call back.

if you have a look at https://github.com/domoticz/domoticz/bl ... SP-W215.py there is a sample of using the framework to handle the HTTP connection for you. Its as easy as:

Code: Select all

    Domoticz.Transport(Transport="TCP/IP", Address=Parameters["Address"], Port=Parameters["Port"])
    Domoticz.Protocol("HTTP")
    Domoticz.Connect()

assuming you allow the IP Address and Port to be set in the Hardware page. When messages are received you can just do something like this in 'onMessage':

Code: Select all

    strData = Data.decode("utf-8", "ignore")
    if (Status == 200):
        Response = json.loads(strData)

where you have an 'import json' somewhere in your plugin.py

[ubee]

@Dnpwwo,

You are right. I've understood you have built-in support for manage a REST interface in the framework, but I couldn't figure out how to use it. That´s why I tried to use the server support in jsonrpclib. After running into the path problem, I modified my code to pure HTTP request by using urlopen and urlencode, and now it works. I can probably change this quite easy to use the built support as you propose.

The line of code that manipulate the relays on UniPi is like this:

response = urlopen("http://"+ UNIPI_URL+"/rest/relay/"+str(Unit),bytes(urlencode({'value': '1'}),'utf-8')).read()

Turns on a relay. Changing value to 0, turns relay off.

And you are also right I should move out to a separate thread. I think it is about time to have a separate sub area for plugin development under Software. I see there are quite many threads under General all addressing plugin development.

[G3rard]

@G3rard,

I have seen occasional crashes when reloading plugins, those that can be reproduced reliably I have resolved.

If you can crash the Sonos plugin regularly can you send me steps you use, I have an Ubuntu VM I can try it on (assuming you are running 14.04)

I am running Domoticz on Ubuntu server 14.04 indeed.
I use this version of the Sonos plugin https://github.com/tester22/Domoticz-So ... /plugin.py.

The plugin crashes when changing when changing the Update interval or changing the Debug parameter in the Hardware page (haven't tried any other setting such as IP address).

@Dnpwwo,
I saw this post of @thomasvillagers describing the same issues.
He describes that Domoticz only freezes when a connection is established.

I can confirm that. I am using Domoticz v3.6630 and when the there is a connection with my Sonos, just changing a parameter in the hardware menu of the Sonos and pressing Update, will make Domoticz crash.
When the Sonos is not playing (so no info shown in the devices), then I can change and press Update without any problems.

Hopefully this gives you some more information to find the issue.