Python Plugin Framework limitation to 256 Units Topic is solved

[pipiche]

From the https://www.domoticz.com/wiki/Developin ... in#Devices page, it is mentioned that the Max number of Unit is limited to 256.
Which means that 1 plugin cannot address more than 256 widgets/devices.

Is that limit still valid ?

This becoming a heavy constraint , as if we look to a standard Xiaomi Temp/Humi/Baro this could already created 6 widgets/devices in Domoticz and so use 6 units

We saw on the market Zigbee controleur capable to go above 255 connected objects, and we will be reaching very soon in front of the 256 Units limit.

Is the liit due to the Python Framework ? or is that a limit driven by Domoticz itself ?

[Dnpwwo]

@pipiche,

It is still the case.

Internally Domoticz uses a byte for the Unit in some code so higher numbered units would not work properly.

The actual fix (IMHO) is that we should take the opportunity to create a v2 plugin API that has a proper hierarchy that maps Hardware -> Device -> Unit rather than just a dictionary of Units. There are a number of other things that could be improved as well.

I've been talking to @gizmocuz about this, yet to settle on an approach.

[pipiche]

thanks for taking the time to respond. I'll stay tuned

Big thanks to the work you do

[psubiaco]

Hi.
I also ran into the 256 units limitation problem with Creasol DomBus modules (RS485 bus).
For now the solution was to disable all useless devices (inputs/outputs/sensors) attached to the bus, but increasing the 256 limit should be a very helpful!!
Thanks for your effort.
Paolo

[Dnpwwo]

I am working on a solution for this but rather than slapping a Band-Aid on it I am using this as an opportunity to uplift the framework.

A number of changes have already been made:

• Plugins now have separate event queues so if one crashes Domoticz the stack trace will identify the culprit
• If plugin debugging is enabled (debug mask '2') then exceptions will write the values of local and Domoticz variables to the log
• Connection objects now have an optional timeout (ms) that will fire an onTimeout event
• Connection objects can now have an event target specified during Connect or Listen operations. All events for the connection will be sent to the specified object

The next change will be to create some new objects that will allow plugin authors to have much more control and that provides a better mapping to more modern hardware types. Specifically:

• A new extended Device object not keyed by Unit (so not constrained to 256)
• Ability to override the class that Domoticz uses to store Devices to allow better encapsulation
• Direct callbacks to Device objects on addition, modification or deletion

So something like this:

Code: Select all

class KodiDevice(Domoticz.Device):
    def __init__(self, DevicelID):
        super().__init__(DevicelID)

	#class variables 

    def onCreate(self):
        Domoticz.Debug("Device onCreate called")

    def onUpdate(self, Previous):
        Domoticz.Debug("Device onUpdate called")

    def onDelete(self):
        Domoticz.Debug("Device onDelete called")

Domoticz.RegisterClass(Device=KodiDevice)

would make Domoticz create the Devices dictionary as a dictionary of KodiDevices and notify them individually of changes.

I have a question for this thread though: What is the best way to expose DeviceStatus details to plugins?

I see two possibilities:

• In the extended Device class, simply expose relevant DeviceStatus details in a simple dictionary with DeviceID/Unit as the key
• Create a new Device class keyed by DeviceID that has minimal data and holds a dictionary of Units (keyed by Unit) that have the relevant fields from DeviceStatus

I don't have strong feelings either way but prefer the second because I feel it more accurately reflects modern devices where one physical device has multiple values associated with it. The last single Z-Wave device I bought had 10 different things it reported for example.

From a Python perspective the difference between the two would look like:

Code: Select all

x = Devices[DeviceID+UnitID].Value

vs

Code: Select all

x = Devices[DeviceID].Units[UnitID].Value

For hardware plugins that have separate connections to to different physical devices the second option allows for encapsulated code like:

Code: Select all

class KodiDevice(Domoticz.Device):
    def __init__(self, DevicelID):
        super().__init__(DevicelID)

        self.kodiConn = Domoticz.Connection(Name=self.Name, Transport="TCP/IP", Protocol="JSON", Address=Parameters['Address'], Port=9090)
        self.kodiConn.Connect(Target=self)

    def onUpdate(self, Previous):
        Domoticz.Debug("Device onUpdate called")

    def onConnect(self, Connection, Status, Description):
        Domoticz.Debug("Device onConnect called")

    def onMessage(self, Connection, Response):
        ...
        this.Units[0] = "Playing"
        ...
        
Domoticz.RegisterClass(Device=KodiDevice)

Thoughts?

[pipiche]

hello and thanks for all of the work you are doing.

If I understood the idea will be that we will have a new way to allocate Domoticz Devices ? Mainly my question is driven by compatibility .

I do also feel that the 2nd option much nicer and cleaner.

[Dnpwwo]

@pipiche,

I plan on making this backwards compatible.

If you register a class to override the Domoticz default you will get the new Device/Unit structure. If you don't you will get the default which is the current structure.

[moroen]

@ Dnpwwo,

The next change will be to create some new objects that will allow plugin authors to have much more control and that provides a better mapping to more modern hardware types. Specifically:

• A new extended Device object not keyed by Unit (so not constrained to 256)
• Ability to override the class that Domoticz uses to store Devices to allow better encapsulation
• Direct callbacks to Device objects on addition, modification or deletion

I'm mimicking this behavour by using a custom dictinary of custom device-objects, keyed by the domoticz unitid. Coming from a programing background that heavily favours OOP, this change would be excellent! Should you need any help with testing, I'll be more than happy to rewrite the Tradfri plugin with this approach.

I have a question for this thread though: What is the best way to expose DeviceStatus details to plugins?

I see two possibilities:

• In the extended Device class, simply expose relevant DeviceStatus details in a simple dictionary with DeviceID/Unit as the key
• Create a new Device class keyed by DeviceID that has minimal data and holds a dictionary of Units (keyed by Unit) that have the relevant fields from DeviceStatus

I don't have strong feelings either way but prefer the second because I feel it more accurately reflects modern devices where one physical device has multiple values associated with it. The last single Z-Wave device I bought had 10 different things it reported for example.

Thoughts?

In the same vein as above, I'm all for as much encapsulation as possible, so I would say that second approach is favourable. IMHO it also allows for much cleaner and readable code.

It can't be expressed enough how much your work on the plugin framework is appreciated!

Regards,
M

[heggink]

Hear hear on the plugin framework. It's turning out to be one of the most important pieces (together with dzvents scripting) and a game changer for domoticz vs other systems. If only someone would build a zwave-node-js mqtt plugin and life would be SWEET!!!

[pipiche]

@dnpwwo are you also envisioning changing the way to update devices ? I understood that the Python Framework is updating directly the DB, while it should use some other Domoticz Layer.

I think one major benefit would be the Master/Slave capability of Domoticz

[Dnpwwo]

@heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.

@pipiche,

I was planning on making minimal changes so wasn't planning on playing with the update but maybe I can look when the dust settles.

My current thinking is to create a new Domoticz import called something like

Code: Select all

import DomoticzEx

that uses the different Device and Unit structure so that it's easy to use the new functionality and easy to maintain backwards compatibility.

[Masure]

Very good news about this plugin system upgrade, thank you for this Dnpwwo !

I know this is not about the topic subject, but could take a look at an issue I encountered with the plugin. This is not an real issue but something lacking : viewtopic.php?f=65&t=35121

This is about accessing other devices status from a plugin.

As you're making a plugin system overhaul, maybe it would be the good time to consider my problem.

Regards

[pipiche]

@pipiche,

I was planning on making minimal changes so wasn't planning on playing with the update but maybe I can look when the dust settles.

My current thinking is to create a new Domoticz import called something like

Code: Select all

import DomoticzEx

that uses the different Device and Unit structure so that it's easy to use the new functionality and easy to maintain backwards compatibility.

I like the idea. A try/except will easily give us the feature or not !

[heggink]

@heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.

@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 )? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...

[pipiche]

Looks like there is already a PR ongoing
https://github.com/domoticz/domoticz/pull/4743

[moroen]

@heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.

@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 )? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...

Perhaps we should split this into it's own topic, but do you have any thoughts on how a plugin with zwave-node-js should work? I've taken a quick look at Zwavejs2Mqtt, based on zwave-node-js, and creating a plugin using MQTT with zwavejs2mqtt as a backend looks manageable, as much of the "hard work" has been done...

Regards,
M

[jkl1337]

Hi,

While some API cleanup would be worthwhile, the point of the pull request was to demonstrate something else: Complete decoupling of the plug-in API from the Device+Unit+Type+Subtype hierarchy. Using the ID (not DeviceID) as an opaque identifier allows individual plugins to use whatever hierarchy schema makes sense on a case by case basis. It’s also a POC of backward compat, but it sounds like there is consensus on that. It’s proof in code that using ID is clean and feasible for plug-in authors.

[heggink]

@heggink,

I do have a ZWave Python plugin but it is pretty rough. It works with the ZWave Gen 5 stick from Aeotec, not sure about other sticks (not the S2 though).

It should work with the upgraded framework, I'll make it available if it does.

@dnpwwo: VERY interested. Any chance I can gave a look (I have the gen 5 )? Also, what framework is it based on? zwave-node-js by any chance? If so then that would be brilliant since it seems to be the next generation ozw successor...

Perhaps we should split this into it's own topic, but do you have any thoughts on how a plugin with zwave-node-js should work? I've taken a quick look at Zwavejs2Mqtt, based on zwave-node-js, and creating a plugin using MQTT with zwavejs2mqtt as a backend looks manageable, as much of the "hard work" has been done...

Regards,
M

Agree its own topic would be better. That would allow us to discuss multiple approaches.
done: viewtopic.php?f=65&t=35881

Sent from my SM-G980F using Tapatalk

[StasDemydiuk]

@dnpwoo Thank you so much for your effort in implementing this python plugin system in Domoticz

Other feature that might be useful for plugin developers is the ability to expose some API from the plugin which can be called using Domoticz JSON API. This is very useful if you are going to implement a custom UI page for the plugin. Something like

Code: Select all

/json.htm?type=plugin_command&id=<plugin_id>&command=<command_name>&myparam=myvalue&<...other command params>

and in plugin

Code: Select all

def onPluginCommand(self, command, params):
        # command in this case equals <command_name> value from query string
        # params is a dictionary of additional query params i.e. {"myparam": "myvalue"}
        Domoticz.Debug("Device onPluginCommandcalled")
        
        return some_response_to_send

And an additional feature to trigger status message from plugin which then will be exposed to UI through Domoticz web socket connection

Code: Select all

...
Domoticz.SendUpdate(payload)
...
# will send MQTT message {event: "plugin_update", id: <plugin_hardware_id>, payload: <payload>}

[pipiche]

@Dnpwwo anything I can help you in testing ?