Page 1 of 3

Python Plugin Framework limitation to 256 Units

Posted: Saturday 06 February 2021 13:27
by pipiche
From the https://www.domoticz.com/wiki/Developin ... in#Devices page, it is mentioned that the Max number of Unit is limited to 256.
Which means that 1 plugin cannot address more than 256 widgets/devices.

Is that limit still valid ?

This becoming a heavy constraint , as if we look to a standard Xiaomi Temp/Humi/Baro this could already created 6 widgets/devices in Domoticz and so use 6 units

We saw on the market Zigbee controleur capable to go above 255 connected objects, and we will be reaching very soon in front of the 256 Units limit.

Is the liit due to the Python Framework ? or is that a limit driven by Domoticz itself ?

Re: Python Plugin Framework limitation to 256 Units

Posted: Friday 12 February 2021 1:08
by Dnpwwo
@pipiche,

It is still the case.

Internally Domoticz uses a byte for the Unit in some code so higher numbered units would not work properly.

The actual fix (IMHO) is that we should take the opportunity to create a v2 plugin API that has a proper hierarchy that maps Hardware -> Device -> Unit rather than just a dictionary of Units. There are a number of other things that could be improved as well.

I've been talking to @gizmocuz about this, yet to settle on an approach.

Re: Python Plugin Framework limitation to 256 Units

Posted: Friday 12 February 2021 10:37
by pipiche
thanks for taking the time to respond. I'll stay tuned ;-)

Big thanks to the work you do

Re: Python Plugin Framework limitation to 256 Units

Posted: Thursday 11 March 2021 22:53
by psubiaco
Hi.
I also ran into the 256 units limitation problem with Creasol DomBus modules (RS485 bus).
For now the solution was to disable all useless devices (inputs/outputs/sensors) attached to the bus, but increasing the 256 limit should be a very helpful!!
Thanks for your effort.
Paolo

Re: Python Plugin Framework limitation to 256 Units

Posted: Saturday 13 March 2021 5:59
by Dnpwwo
I am working on a solution for this but rather than slapping a Band-Aid on it I am using this as an opportunity to uplift the framework.

A number of changes have already been made:
  • Plugins now have separate event queues so if one crashes Domoticz the stack trace will identify the culprit
  • If plugin debugging is enabled (debug mask '2') then exceptions will write the values of local and Domoticz variables to the log
  • Connection objects now have an optional timeout (ms) that will fire an onTimeout event
  • Connection objects can now have an event target specified during Connect or Listen operations. All events for the connection will be sent to the specified object
The next change will be to create some new objects that will allow plugin authors to have much more control and that provides a better mapping to more modern hardware types. Specifically:
  • A new extended Device object not keyed by Unit (so not constrained to 256)
  • Ability to override the class that Domoticz uses to store Devices to allow better encapsulation
  • Direct callbacks to Device objects on addition, modification or deletion
So something like this:

Code: Select all

class KodiDevice(Domoticz.Device):
    def __init__(self, DevicelID):
        super().__init__(DevicelID)

	#class variables 

    def onCreate(self):
        Domoticz.Debug("Device onCreate called")

    def onUpdate(self, Previous):
        Domoticz.Debug("Device onUpdate called")

    def onDelete(self):
        Domoticz.Debug("Device onDelete called")

Domoticz.RegisterClass(Device=KodiDevice)
would make Domoticz create the Devices dictionary as a dictionary of KodiDevices and notify them individually of changes.

I have a question for this thread though: What is the best way to expose DeviceStatus details to plugins?

I see two possibilities:
  • In the extended Device class, simply expose relevant DeviceStatus details in a simple dictionary with DeviceID/Unit as the key
  • Create a new Device class keyed by DeviceID that has minimal data and holds a dictionary of Units (keyed by Unit) that have the relevant fields from DeviceStatus
I don't have strong feelings either way but prefer the second because I feel it more accurately reflects modern devices where one physical device has multiple values associated with it. The last single Z-Wave device I bought had 10 different things it reported for example.

From a Python perspective the difference between the two would look like:

Code: Select all

x = Devices[DeviceID+UnitID].Value
vs

Code: Select all

x = Devices[DeviceID].Units[UnitID].Value
For hardware plugins that have separate connections to to different physical devices the second option allows for encapsulated code like:

Code: Select all

class KodiDevice(Domoticz.Device):
    def __init__(self, DevicelID):
        super().__init__(DevicelID)

        self.kodiConn = Domoticz.Connection(Name=self.Name, Transport="TCP/IP", Protocol="JSON", Address=Parameters['Address'], Port=9090)
        self.kodiConn.Connect(Target=self)

    def onUpdate(self, Previous):
        Domoticz.Debug("Device onUpdate called")

    def onConnect(self, Connection, Status, Description):
        Domoticz.Debug("Device onConnect called")

    def onMessage(self, Connection, Response):
        ...
        this.Units[0] = "Playing"
        ...
        
Domoticz.RegisterClass(Device=KodiDevice)
Thoughts?

Re: Python Plugin Framework limitation to 256 Units

Posted: Monday 15 March 2021 9:05
by pipiche
hello and thanks for all of the work you are doing.

If I understood the idea will be that we will have a new way to allocate Domoticz Devices ? Mainly my question is driven by compatibility .

I do also feel that the 2nd option much nicer and cleaner.

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 16 March 2021 13:24
by Dnpwwo
@pipiche,

I plan on making this backwards compatible.

If you register a class to override the Domoticz default you will get the new Device/Unit structure. If you don't you will get the default which is the current structure.

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 16 March 2021 16:55
by moroen
@ Dnpwwo,
Dnpwwo wrote: Saturday 13 March 2021 5:59 The next change will be to create some new objects that will allow plugin authors to have much more control and that provides a better mapping to more modern hardware types. Specifically:
  • A new extended Device object not keyed by Unit (so not constrained to 256)
  • Ability to override the class that Domoticz uses to store Devices to allow better encapsulation
  • Direct callbacks to Device objects on addition, modification or deletion
I'm mimicking this behavour by using a custom dictinary of custom device-objects, keyed by the domoticz unitid. Coming from a programing background that heavily favours OOP, this change would be excellent! Should you need any help with testing, I'll be more than happy to rewrite the Tradfri plugin with this approach.
Dnpwwo wrote: Saturday 13 March 2021 5:59 I have a question for this thread though: What is the best way to expose DeviceStatus details to plugins?

I see two possibilities:
  • In the extended Device class, simply expose relevant DeviceStatus details in a simple dictionary with DeviceID/Unit as the key
  • Create a new Device class keyed by DeviceID that has minimal data and holds a dictionary of Units (keyed by Unit) that have the relevant fields from DeviceStatus
I don't have strong feelings either way but prefer the second because I feel it more accurately reflects modern devices where one physical device has multiple values associated with it. The last single Z-Wave device I bought had 10 different things it reported for example.

Thoughts?
In the same vein as above, I'm all for as much encapsulation as possible, so I would say that second approach is favourable. IMHO it also allows for much cleaner and readable code.

It can't be expressed enough how much your work on the plugin framework is appreciated!

Regards,
M

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 16 March 2021 17:52
by heggink
Hear hear on the plugin framework. It's turning out to be one of the most important pieces (together with dzvents scripting) and a game changer for domoticz vs other systems. If only someone would build a zwave-node-js mqtt plugin and life would be SWEET!!!

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 16 March 2021 18:03
by pipiche
@dnpwwo are you also envisioning changing the way to update devices ? I understood that the Python Framework is updating directly the DB, while it should use some other Domoticz Layer.

I think one major benefit would be the Master/Slave capability of Domoticz

Re: Python Plugin Framework limitation to 256 Units

Posted: Saturday 20 March 2021 9:31
by Dnpwwo
@heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.

@pipiche,

I was planning on making minimal changes so wasn't planning on playing with the update but maybe I can look when the dust settles.

My current thinking is to create a new Domoticz import called something like

Code: Select all

import DomoticzEx
that uses the different Device and Unit structure so that it's easy to use the new functionality and easy to maintain backwards compatibility.

Re: Python Plugin Framework limitation to 256 Units

Posted: Saturday 20 March 2021 10:40
by Masure
Very good news about this plugin system upgrade, thank you for this Dnpwwo !

I know this is not about the topic subject, but could take a look at an issue I encountered with the plugin. This is not an real issue but something lacking : viewtopic.php?f=65&t=35121

This is about accessing other devices status from a plugin.

As you're making a plugin system overhaul, maybe it would be the good time to consider my problem.

Regards

Re: Python Plugin Framework limitation to 256 Units

Posted: Saturday 20 March 2021 16:20
by pipiche
Dnpwwo wrote: Saturday 20 March 2021 9:31
@pipiche,

I was planning on making minimal changes so wasn't planning on playing with the update but maybe I can look when the dust settles.

My current thinking is to create a new Domoticz import called something like

Code: Select all

import DomoticzEx
that uses the different Device and Unit structure so that it's easy to use the new functionality and easy to maintain backwards compatibility.
I like the idea. A try/except will easily give us the feature or not !

Re: Python Plugin Framework limitation to 256 Units

Posted: Sunday 21 March 2021 10:36
by heggink
Dnpwwo wrote: Saturday 20 March 2021 9:31 @heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.
@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 ;-))? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 23 March 2021 9:07
by pipiche
Looks like there is already a PR ongoing
https://github.com/domoticz/domoticz/pull/4743

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 23 March 2021 20:58
by moroen
heggink wrote: Sunday 21 March 2021 10:36
Dnpwwo wrote: Saturday 20 March 2021 9:31 @heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.
@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 ;-))? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...
Perhaps we should split this into it's own topic, but do you have any thoughts on how a plugin with zwave-node-js should work? I've taken a quick look at Zwavejs2Mqtt, based on zwave-node-js, and creating a plugin using MQTT with zwavejs2mqtt as a backend looks manageable, as much of the "hard work" has been done...

Regards,
M

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 23 March 2021 21:24
by jkl1337
Hi,

While some API cleanup would be worthwhile, the point of the pull request was to demonstrate something else: Complete decoupling of the plug-in API from the Device+Unit+Type+Subtype hierarchy. Using the ID (not DeviceID) as an opaque identifier allows individual plugins to use whatever hierarchy schema makes sense on a case by case basis. It’s also a POC of backward compat, but it sounds like there is consensus on that. It’s proof in code that using ID is clean and feasible for plug-in authors.

Re: Python Plugin Framework limitation to 256 Units

Posted: Tuesday 23 March 2021 21:37
by heggink
moroen wrote:
heggink wrote: Sunday 21 March 2021 10:36
Dnpwwo wrote: Saturday 20 March 2021 9:31 @heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.
@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 ;-))? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...
Perhaps we should split this into it's own topic, but do you have any thoughts on how a plugin with zwave-node-js should work? I've taken a quick look at Zwavejs2Mqtt, based on zwave-node-js, and creating a plugin using MQTT with zwavejs2mqtt as a backend looks manageable, as much of the "hard work" has been done...

Regards,
M
Agree its own topic would be better. That would allow us to discuss multiple approaches.
done: viewtopic.php?f=65&t=35881

Sent from my SM-G980F using Tapatalk

Re: Python Plugin Framework limitation to 256 Units

Posted: Saturday 27 March 2021 7:35
by StasDemydiuk
@dnpwoo Thank you so much for your effort in implementing this python plugin system in Domoticz

Other feature that might be useful for plugin developers is the ability to expose some API from the plugin which can be called using Domoticz JSON API. This is very useful if you are going to implement a custom UI page for the plugin. Something like

Code: Select all

/json.htm?type=plugin_command&id=<plugin_id>&command=<command_name>&myparam=myvalue&<...other command params>
and in plugin

Code: Select all

def onPluginCommand(self, command, params):
        # command in this case equals <command_name> value from query string
        # params is a dictionary of additional query params i.e. {"myparam": "myvalue"}
        Domoticz.Debug("Device onPluginCommandcalled")
        
        return some_response_to_send
And an additional feature to trigger status message from plugin which then will be exposed to UI through Domoticz web socket connection

Code: Select all

...
Domoticz.SendUpdate(payload)
...
# will send MQTT message {event: "plugin_update", id: <plugin_hardware_id>, payload: <payload>}

Re: Python Plugin Framework limitation to 256 Units

Posted: Monday 05 April 2021 11:29
by pipiche
@Dnpwwo anything I can help you in testing ?