Python Plugin Framework limitation to 256 Units Topic is solved

[Dnpwwo]

I stopped work on this because of https://github.com/domoticz/domoticz/pull/4743 and there is no need for two different solutions.

Did it ever get completed?

[heggink]

Don't think it has

Sent from my SM-G980F using Tapatalk

[pipiche]

I think he gave up


Envoyé de mon iPhone en utilisant Tapatalk

[Dnpwwo]

I have a working mostly working version, I'm creating a sample plugin to test it out.

[pipiche]

Great news . If you want me to test it in a larger scale and checking backwards compatibility, I’m your man

[Dnpwwo]

I've pushed a base version just now, it should be backwards compatible but I will wait for the beta community to confirm.

To activate the most simple feature of the extended framework you just need to 'import DomoticzEx' instead of 'import Domoticz'. That should create a dictionary of Devices where each Device contains a dictionary of Units. If you do that for an existing plugin then each Device dictionary will probably contain one Unit. I would be interested to know if that at least works.

Using the new import may cause some issues for existing plugins because a number of callbacks have an additional first parameter (DeviceID), if that causes an issue just go back to the old import.

Once I'm confident that I haven't broken backwards compatibility I will post examples on how to use the new functionality:

• Creating Units within Devices to remove the 256 Unit restriction
• Getting the framework to use your own custom classes in place of the built in ones
• Localising callbacks to Units or Devices to improve code segregation

[pipiche]

Latest Domoticz build with the ZiGate plugin works well.

However, a small remark which is quiet "annoying" when trying on a large python plugin.
as it will required to update all DomoticzEx , the Domoticz.Log() ; Domoticz.Error(), Domoticz.Status()

Taking into account that DomoticzEx might not be available for Build prior 13324, I'm wondering if I'll be able to do:

Code: Select all

try:
    import DomoticzEx as Domoticz
except:
    import Domoticz

[pipiche]

Never got this issue before.

Jun 14 22:05:43 rasp domoticz[16661]: Fatal Python error: Segmentation fault
Jun 14 22:05:43 rasp domoticz[16661]: Thread 0xb6f6d8a0 (most recent call first):
Jun 14 22:05:43 rasp domoticz[16661]: <no Python frame>
Jun 14 22:05:43 rasp domoticz[16661]: 2021-06-14 22:05:43.425 Error: Domoticz(pid:16661, tid:16678('Plugin_ASIO')) received fatal signal 11 (Segmentation fault)
Jun 14 22:05:43 rasp domoticz[16661]: 2021-06-14 22:05:43.425 Error: siginfo address=0x4115, address=0xaf0fe624
Jun 14 22:05:45 rasp domoticz[16661]: 2021-06-14 22:05:45.693 ZiGate-V2: (ZiGate-V2) Zigate command round trip 0x8011 Max: 6473 ms with an of average: 6473 ms
Jun 14 22:05:52 rasp domoticz[16661]: 2021-06-14 22:05:52.193 ZiGate-V2: (ZiGate-V2) Zigate command round trip 0x8011 Max: 6482 ms with an of average: 6477 ms

Jun 14 22:06:05 rasp domoticz[16661]: 2021-06-14 22:06:05.775 Error: ZiGate-V2: (ZiGate-V2) writerThread Timeout
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.070 Error: Thread 14 (Thread 0xaf0ff3a0 (LWP 16678)):
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.070 Error: #0 0xb68c7e90 in __GI___wait4 (pid=16695, stat_loc=0xaf0fdf18, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.070 Error: #1 0x0077e440 in dumpstack_gdb(bool) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #2 0x0077e918 in dumpstack(void*, void*) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #3 0x0077ebd0 in signal_handler(int, siginfo_t*, void*) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #4 <signal handler called>
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #5 raise (sig=11) at ../sysdeps/unix/sysv/linux/raise.c:50
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #6 <signal handler called>
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #7 0x00d1fca8 in Plugins::py3__Py_INCREF(_object*) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #8 0x00d2165c in Plugins::CPluginTransportTCP::handleAsyncAccept(boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::execution::any_executor<boost::asio::execution::context_as_t<boost::asio::execution_context&>, boost::asio::execution::detail::blocking::never_t<0>, boost::asio::execution::prefer_only<boost::asio::execution::detail::blocking::possibly_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::tracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::untracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::fork_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::continuation_t<0> > > >*, boost::system::error_code const&) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.071 Error: #9 0x00d27448 in auto Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::operator()<boost::system::error_code const&>(boost::system::error_code const&) const ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.072 Error: #10 0x00d326bc in boost::asio::detail::binder1<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::system::error_code>::operator()() ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.072 Error: #11 0x00d31800 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder1<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::system::error_code> >(boost::asio::detail::binder1<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::system::error_code>&, ...) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.072 Error: #12 0x00d30818 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::system::error_code>, {lambda(auto:1&&)#1}>(boost::asio::detail::binder1<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::system::error_code>&, {lambda(auto:1&&)#1}&) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.072 Error: #13 0x00d2f2e4 in void boost::asio::detail::handler_work<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::asio::execution::any_executor<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::context_as_t<boost::asio::execution_context&>, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0>, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::prefer_only<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::possibly_t<0> >, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0><Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::context_as_t<boost::asio::execution_context&>::outstanding_work::tracked_t<0> >, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0><Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::prefer_only<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::possibly_t<0> >::untracked_t<0> >, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0><Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::context_as_t<boost::asio::execution_context&>::relationship::fork_t<0> >, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0><Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::blocking::never_t<0><Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::prefer_only<Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}::detail::possibly_t<0> >::untracked_t<0> >::continuation_t<0> > >, void>::complete<boost::asio::detail::binder1<{lambda(auto:1&&)#1}, boost::system::error_code> >(boost::asio::detail::binder1<{lambda(auto:1&&)#1}, boost::system::error_code>&, {lambda(auto:1&&)#1}&) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.072 Error: #14 0x00d2ca9c in boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::execution::any_executor<boost::asio::execution::context_as_t<boost::asio::execution_context&>, boost::asio::execution::detail::blocking::never_t<0>, boost::asio::execution::prefer_only<boost::asio::execution::detail::blocking::possibly_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::tracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::untracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::fork_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::continuation_t<0> > > >, boost::asio::ip::tcp, Plugins::CPluginTransportTCP::handleListen()::{lambda(auto:1&&)#1}, boost::asio::execution::any_executor<boost::asio::execution::context_as_t<boost::asio::execution_context&>, boost::asio::execution::detail::blocking::never_t<0>, boost::asio::execution::prefer_only<boost::asio::execution::detail::blocking::possibly_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::tracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::outstanding_work::untracked_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::fork_t<0> >, boost::asio::execution::prefer_only<boost::asio::execution::detail::relationship::continuation_t<0> > > >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned int) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #15 0x009957cc in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned int) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #16 0x00998634 in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned int) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #17 0x009957cc in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned int) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #18 0x00999648 in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #19 0x009990c4 in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #20 0x009b27f0 in boost::asio::io_context::run() ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #21 0x00d0716c in Plugins::BoostWorkers() ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #22 0x00d0dedc in boost::detail::thread_data<void (*)()>::run() ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.073 Error: #23 0x00e8f800 in thread_proxy ()
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.074 Error: #24 0xb6a7d348 in start_thread (arg=0xaf0ff3a0) at pthread_create.c:477
Jun 14 22:06:06 rasp domoticz[16661]: 2021-06-14 22:06:06.074 Error: #25 0xb6900508 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/libc.so.6
Jun 14 22:06:06 rasp systemd[1]: Started Process Core Dump (PID 16715/UID 0).
Jun 14 22:06:16 rasp systemd-coredump[16716]: Process 16661 (domoticz) of user 1001 dumped core.

Stack trace of thread 16678:
#0 0x00000000b6a8b274 raise (libpthread.so.0 + 0x15274)
#1 0x000000000077ebe4 _Z14signal_handleriP9siginfo_tPv (domoticz + 0x776be4)

And if I fall back from 13324 to 13311 it works

[Dnpwwo]

@pipiche,

My bad, I've pushed a fix.

You certainly can import the extended Domoticz module 'as' another name, that's standard Python. The API is a little different though so the fallback approach may not have the value you are looking for.

If you want to do a minimal upgrade for an existing plugin to the new framework points change will be:

• 2 tier DeviceStatus mapping: Devices[<DeviceID>].Units[<UnitID>]
• Update() function. In the existing framework the Update() function takes a lot of params, in the extended framework it does not. Attributes are updated directly and simply pushed to the database using Update(). e.g:

  Code: Select all

  if not device in Devices:
  	if (device == "SYST"):
  		Options = {"LevelActions": " ", "LevelNames": "Off", "LevelOffHidden": "false", "SelectorStyle": "0"}
  		Domoticz.Unit(Name="Mode", DeviceID=device, Unit=1, TypeName="Selector Switch", Switchtype=18, Options=Options).Create()
  	if (device == "HGOM"):
  		Domoticz.Unit(Name="Heating", DeviceID=device, Unit=1, TypeName="Switch", Image=15).Create()
  		Devices["SYST"].Units[1].Image = 15
  		Devices["SYST"].Units[1].Options["LevelActions"] = Devices["SYST"].Units[1].Options["LevelActions"] + "|"
  		Devices["SYST"].Units[1].Options["LevelNames"] = Devices["SYST"].Units[1].Options["LevelNames"] + "|Heating"
  	Devices["SYST"].Units[1].Update()   # Commit changes to database
  

[pipiche]

Thanks it does fix the problem on build 13327, will continue my testing.

[waltervl]

Some other members complaining about crashing Domoticz due to plugins on latest beta: https://github.com/domoticz/domoticz/issues/4861

[pipiche]

just move on 13329 and the ZiGate python plugin works well.

could be related to other communication mode

[waltervl]

Should be fixed in build 13330 (not yet released)

[pipiche]

Should be fixed in build 13330 (not yet released)

Do confirm that 13330 no regression

[pipiche]

@pipiche,

You certainly can import the extended Domoticz module 'as' another name, that's standard Python. The API is a little different though so the fallback approach may not have the value you are looking for.

If you want to do a minimal upgrade for an existing plugin to the new framework points change will be:

• 2 tier DeviceStatus mapping: Devices[<DeviceID>].Units[<UnitID>]
• Update() function. In the existing framework the Update() function takes a lot of params, in the extended framework it does not. Attributes are updated directly and simply pushed to the database using Update(). e.g:

  Code: Select all

  if not device in Devices:
  	if (device == "SYST"):
  		Options = {"LevelActions": " ", "LevelNames": "Off", "LevelOffHidden": "false", "SelectorStyle": "0"}
  		Domoticz.Unit(Name="Mode", DeviceID=device, Unit=1, TypeName="Selector Switch", Switchtype=18, Options=Options).Create()
  	if (device == "HGOM"):
  		Domoticz.Unit(Name="Heating", DeviceID=device, Unit=1, TypeName="Switch", Image=15).Create()
  		Devices["SYST"].Units[1].Image = 15
  		Devices["SYST"].Units[1].Options["LevelActions"] = Devices["SYST"].Units[1].Options["LevelActions"] + "|"
  		Devices["SYST"].Units[1].Options["LevelNames"] = Devices["SYST"].Units[1].Options["LevelNames"] + "|Heating"
  	Devices["SYST"].Units[1].Update()   # Commit changes to database
  

If I well understood , it will more than challenging to have a single source code able to work on Domoticz prior your update and before ? Cannot we have in the new version the 2 APIs available ? where the old API just have the same constraint as before and the new API get the extended one ?
In such DomoticzEx will be just an extension of Domoticz and not a replacement ?

[pipiche]

hello,

I'm trying to play with the DomoticzEx and to start with I have a number of Devices created with the legacy framework that I try to load with the new.
The first thing I'm doing in onStart() is to call this function.

Code: Select all

def DumpDevicesEx():
    for x in Devices:
        Domoticz.Log("Devices[%s]" %x)
        for y in Devices[x].Units:
            Domoticz.Log("Devices[%s][%s].Name= %s" %(x,y,Devices[x].Units[y].Name))
            Domoticz.Log("Devices[%s][%s].nValue= %s" %(x,y,Devices[x].Units[y].nValue))
            Domoticz.Log("Devices[%s][%s].sValue= %s" %(x,y,Devices[x].Units[y].sValue))

Do I miss something ?

The outcome looks bizarre:

Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.606 ZiGate-V2: (ZiGate-V2) Devices[60a423fffe529d60]
Jun 29 19:24:11 rasp domoticz[1054]: [84B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.606 ZiGate-V2: (ZiGate-V2) Devices[60a423fffe529d60][1].nValue= 0
Jun 29 19:24:11 rasp domoticz[1054]: [86B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.606 ZiGate-V2: (ZiGate-V2) Devices[90fd9ffffe86c7a1]
Jun 29 19:24:11 rasp domoticz[1054]: [84B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.607 ZiGate-V2: (ZiGate-V2) Devices[90fd9ffffe86c7a1][2].nValue= 1
Jun 29 19:24:11 rasp domoticz[1054]: [86B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.607 ZiGate-V2: (ZiGate-V2) Devices[Zigate-02- 9]
Jun 29 19:24:11 rasp domoticz[1054]: [80B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.607 ZiGate-V2: (ZiGate-V2) Devices[Zigate-02- 9][3].nValue= 1
Jun 29 19:24:11 rasp domoticz[1054]: [82B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.607 ZiGate-V2: (ZiGate-V2) Devices[Zigate-03- 9]
Jun 29 19:24:11 rasp domoticz[1054]: [80B blob data]
Jun 29 19:24:11 rasp domoticz[1054]: 2021-06-29 19:24:11.608 ZiGate-V2: (ZiGate-V2) Devices[Zigate-03- 9][8].nValue= 0
Jun 29 19:24:11 rasp domoticz[1054]: [82B blob data]

and here after is the output of a sqlite query

Code: Select all

sqlite> select * from DeviceStatus where HardwareID = 9;

327|9|Zigate-03- 9|8|ZiGate-V2 - Zigate Notifications 9|0|243|19|0|0|12|255|0|Zigate Channel: 11|2021-06-22 22:46:02|328|0.0|1.0|0.0|1.0|||0|0|0|||
328|9|60a423fffe529d60|1|ZiGate-V2 - LXEK-1_ColorControlRGBWW-60a423fffe529d60-01|0|241|4|7|0|3|255|0|Off|2021-06-22 22:46:23|329|0.0|1.0|0.0|1.0|||0|0|0|||
329|9|90fd9ffffe86c7a1|2|ZiGate-V2 - TRADFRI bulb E27 WS clear 950lm_ColorControlWW-90fd9ffffe86c7a1-01|0|241|8|7|0|1|255|1|100|2021-06-22 22:46:36|330|0.0|1.0|0.0|1.0|||0|0|0|||
330|9|Zigate-02- 9|3|ZiGate-V2 - Zigate Status 9|0|243|22|0|0|12|255|1|Ready|2021-06-22 22:46:17|331|0.0|1.0|0.0|1.0|||0|0|0|||

[Dnpwwo]

Your code looks okay except for the fact that 'y' is the Unit number so not a string.

Can't explain the 'blob data' message though.

I'll keep looking...

[pipiche]

But then the sValue and nValue should have the same issue, no ?

[Dnpwwo]

Pipiche,

I found what this is and will push a fix. If you str(Devices[x].Units[y]) the data is correct.

It's a chance to make things simpler but I will need to test it.

[pipiche]

indeed, I do confirm that the str(Device[x].Units[y]) works