Bugs in Python framework and/or Domoticz

[gerardvs]

Hi All,

The Python Plugins are to my opinion one of the best additions to Domoticz lately. Thanks for that!

However, in my opinion there are two issues with the Python PLugin system.
1. deviceID mismatch
After the plugin is started it has created a device resulting in a deviceID and IDX. In my case this is a "Custom Sensor". When I try to update the sensor via the json api e.q. /json.htm?type=command&param=udevice&idx=1&nvalue=0&svalue=103 then the device is not updated but a new device is created with a new IDX and a deviceID with an extra leading 0.

After investigating the code (domoticz/hardware/plugins/PythonObjects.cpp) at or around line 630 there is

Code: Select all

sprintf(szID, "%X%02X%02X%02X", 0, 0, (self->HwdID & 0xFF00) >> 8, self->HwdID & 0xFF);

When I change the code to:

Code: Select all

sprintf(szID, "%02X%02X%02X%02X", 0, 0, (self->HwdID & 0xFF00) >> 8, self->HwdID & 0xFF);

The above problem is gone and the by the plugin created device is accessible via json api.
I already created an issue (#1287) so hopefully someone will look into this and correct this.

2. Notification
Notifications are not working well on the device created by the plugin. When I update the device with the json api notifications do work fine. However when the plugin updates the device nothing happens.

So it seems that the notification system is not monitoring deviceupdates by the plugin.

Hopefully some can look into these issues.

Thanks,
--Gerard

**EDIT** renamed subject after rejection of dev issue

[gerardvs]

Sorry to bump this question again. I made, to my opinion, a perfect clear issue regarding a bug in Domoticz but the developper is not taking action without a forum discussion.

So where to go in resolving these nasty bugs?

--Gerard

[Dnpwwo]

@gerardvs,

What you say might be true but it is not that simple. Domoticz has a rich heritage that means that things are not always as straightforward as they seem.

Even though the Python framework doesn't really use the deviceID field I create a valid one in the most common format used by similar C++ plugins. A simple 'find in files' for the shorter form used for the device ID as used by Python yields the following:

Code: Select all

  D:\Domoticz\domoticz\hardware\BleBox.cpp(596):	sprintf(szIdx, "%X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\Daikin.cpp(167):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\DenkoviSmartdenLan.cpp(132):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\Dummy.cpp(167):					sprintf(ID, "%X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\ETH8020.cpp(136):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\FritzboxTCP.cpp(226):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\Kodi.cpp(156):	sprintf(m_szDevID, "%X%02X%02X%02X", 0, 0, (m_DevID & 0xFF00) >> 8, m_DevID & 0xFF);
  D:\Domoticz\domoticz\hardware\Kodi.cpp(1085):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\Kodi.cpp(1108):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\Kodi.cpp(1122):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\LogitechMediaServer.cpp(488):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\LogitechMediaServer.cpp(581):			sprintf(pnode.szDevID, "%X%02X%02X%02X", 0, 0, (pnode.DevID & 0xFF00) >> 8, pnode.DevID & 0xFF);
  D:\Domoticz\domoticz\hardware\MySensorsBase.cpp(1089):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\MySensorsBase.cpp(1148):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\MySensorsBase.cpp(1221):		sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\Nest.cpp(144):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\Nest.cpp(285):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, Idx, 0);
  D:\Domoticz\domoticz\hardware\OTGWBase.cpp(73):	sprintf(szIdx,"%X%02X%02X%02X",0,0,0,Idx);
  D:\Domoticz\domoticz\hardware\PanasonicTV.cpp(165):	sprintf(m_szDevID, "%X%02X%02X%02X", 0, 0, (m_DevID & 0xFF00) >> 8, m_DevID & 0xFF);
  D:\Domoticz\domoticz\hardware\PanasonicTV.cpp(852):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\PanasonicTV.cpp(875):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\PanasonicTV.cpp(889):	sprintf(szID, "%X%02X%02X%02X", 0, 0, (ID & 0xFF00) >> 8, ID & 0xFF);
  D:\Domoticz\domoticz\hardware\PhilipsHue\PhilipsHue.cpp(451):		sprintf(szID, "%X%02X%02X%02X", 0, 0, 0, NodeID);
  D:\Domoticz\domoticz\hardware\Pinger.cpp(227):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\Pinger.cpp(249):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\Pinger.cpp(268):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\plugins\PythonObjects.cpp(630):					sprintf(szID, "%X%02X%02X%02X", 0, 0, (self->HwdID & 0xFF00) >> 8, self->HwdID & 0xFF);
  D:\Domoticz\domoticz\hardware\Sterbox.cpp(141):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\ToonThermostat.cpp(251):	sprintf(szIdx, "%X%02X%02X%02X", 0, 0, 0, Idx);
  D:\Domoticz\domoticz\hardware\WOL.cpp(181):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\WOL.cpp(204):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\WOL.cpp(221):	sprintf(szID,"%X%02X%02X%02X", 0, 0, (ID&0xFF00)>>8, ID&0xFF);
  D:\Domoticz\domoticz\hardware\ZWaveBase.cpp(339):	sprintf(szID,"%X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\DomoticzHardware.cpp(540):	sprintf(szIdx, "%X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\DomoticzHardware.cpp(562):	sprintf(szIdx, "%X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\main\mainworker.cpp(1702):		sprintf(szTmp,"%X%02X%02X%02X", pResponse->LIGHTING2.id1, pResponse->LIGHTING2.id2, pResponse->LIGHTING2.id3, pResponse->LIGHTING2.id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(1729):		sprintf(szTmp, "%X%02X%02X%02X", pResponse->RADIATOR1.id1, pResponse->RADIATOR1.id2, pResponse->RADIATOR1.id3, pResponse->RADIATOR1.id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(1778):		sprintf(szTmp, "%X%02X%02X%02X", pMeter->id1, pMeter->id2, pMeter->id3, pMeter->id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(4420):	sprintf(szTmp,"%X%02X%02X%02X", pResponse->LIGHTING2.id1, pResponse->LIGHTING2.id2, pResponse->LIGHTING2.id3, pResponse->LIGHTING2.id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(8164):	sprintf(szTmp, "%X%02X%02X%02X", pResponse->RADIATOR1.id1, pResponse->RADIATOR1.id2, pResponse->RADIATOR1.id3, pResponse->RADIATOR1.id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(9397):	sprintf(szTmp,"%X%02X%02X%02X", pMeter->id1, pMeter->id2, pMeter->id3, pMeter->id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(9439):	sprintf(szTmp,"%X%02X%02X%02X", pMeter->id1, pMeter->id2, pMeter->id3, pMeter->id4);
  D:\Domoticz\domoticz\main\mainworker.cpp(9481):	sprintf(szTmp,"%X%02X%02X%02X", pMeter->id1, pMeter->id2, pMeter->id3, pMeter->id4);
  Matching lines: 43    Matching files: 21    Total files searched: 600

43 matches while the longer form matches just 14

Code: Select all

  D:\Domoticz\domoticz\hardware\BleBox.cpp(256):		sprintf(szIdx, "%02X%02X%02X%02X", atoi(strarray[0].data()), atoi(strarray[1].data()), atoi(strarray[2].data()), atoi(strarray[3].data()));
  D:\Domoticz\domoticz\hardware\BleBox.cpp(260):		sprintf(szIdx, "%02x%02x%02x%02x", atoi(strarray[0].data()), atoi(strarray[1].data()), atoi(strarray[2].data()), atoi(strarray[3].data()));
  D:\Domoticz\domoticz\hardware\BleBox.cpp(526):		sprintf(level, "%02x%02x%02x%02x", red, green, blue, 255);
  D:\Domoticz\domoticz\hardware\Dummy.cpp(350):					sprintf(ID, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(444):	sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(488):	sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(1131):			sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(1136):            sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(1148):			sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetTCP.cpp(1153):			sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\hardware\OpenWebNetUSB.cpp(250):	sprintf(szIdx, "%02X%02X%02X%02X", ID1, ID2, ID3, ID4);
  D:\Domoticz\domoticz\main\mainworker.cpp(1723):		sprintf(szTmp, "%02X%02X%02X%02X", pResponse->HOMECONFORT.id1, pResponse->HOMECONFORT.id2, pResponse->HOMECONFORT.id3, pResponse->HOMECONFORT.housecode);
  D:\Domoticz\domoticz\main\mainworker.cpp(5403):	sprintf(szTmp, "%02X%02X%02X%02X", pResponse->HOMECONFORT.id1, pResponse->HOMECONFORT.id2, pResponse->HOMECONFORT.id3, pResponse->HOMECONFORT.housecode);
  D:\Domoticz\domoticz\main\mainworker.cpp(5843):	sprintf(szTmp, "%02X%02X%02X%02X", pResponse->BLINDS1.id1, pResponse->BLINDS1.id2, pResponse->BLINDS1.id3, pResponse->BLINDS1.id4);
  Matching lines: 14    Matching files: 5    Total files searched: 600

some files have a mixture of the two.

On that basis I would suggest that the fix is not obvious. There is also an argument to be made that devices that are controlled by C++ or Pyhton plugins should not have their values updated by the web interface.

For me a better question would be: Why does the JSON interface care about the device ID format when you supply a device index which is all it needs to do the update? Could be more of that heritage stuff I suspect.

I will have a look at the notifications stuff when I get a chance, its on the list.

[gerardvs]

Hi @Dnpwwo,

Good discussion and thanks for taking the time.

With the, somewhat limited, knowledge of cpp I also have concluded that looking at the code there is no uniformity in the different types of devices. One could question why there are so many of them and why not use generic devices, inherit all the useful methods and add functionality when needed. In this case we could say: "Quantity outlives Quality" (sorry, but I couldn't resist ).

There is also an argument to be made that devices that are controlled by C++ or Python plugins should not have their values updated by the web interface.

Good question, if I may answer I would say, yes certainly.

Notifications are on many devices a problem. So I think your thoughts are absolutely right about the heritage stuff being an issue within the Domoticz core.

I would really like to help on anything that could help resolving these issues. I'm too rusty on cpp but am willing to test the crap out of every system I have so please let me know when and what to do to help.

Thanks again,
--Gerard

[Dnpwwo]

@gerardvs,

Pull request #1308 will give you some of what you want when it is merged. You can set the DeviceID to whatever you want during device creation./

[gerardvs]

Thanks, this could really help!
From @Gismocuz I also received useful information about implementing notifications so hopefully we can make things better.
Let's say, to be continued....

[gerardvs]

@Dnpwwo
After some testing with the #1308 code I can not get the DeviceID to work well.
Your description of usage is:

Set the DeviceID to be used with the device. Only required to override the default which is and eight digit number dervice from the HardwareID and the Unit number in the format "000H000U"

To the best of my knowledge the DeviceID has no relation with the HwdID and Unit. I see you use both of them to construct the DeviceID. Is there a reason to do so or is it just a way to generate a DeviceID string?

The funny thing is when I use a DeviceID of "8C8304" a device with an ID of 0004 is created but not added. After adding the device from the Domoticz GUI manually values are updated normally resulting in me to be more confused than I already was.

Do you have any thoughts on this?

--Gerard

[Dnpwwo]

@gerardvs,

Pull request #1308 has not be accepted into the beta channel yet, its still pending so unless you are building from source against the "Python-Plugins" branch you will not be able to set the DeviceID (see https://github.com/domoticz/domoticz/pull/1308). The documentation tends to be slightly ahead of the code merge otherwise I forget to go back and add it.

Its also really hard for me to comment on people's problems if they don't post the Python code and debug logs because I'm just guessing. I can help more if you share the detail with me.

[gerardvs]

Hi @Dnpwwo,

The environment I tested with was a Domoticz main branch V3.6883 and compiled it just to be sure it was free of errors, which it was. Then I took PythonObjects.cpp from your pull request and compiled again, which went also without problems.

My (python) code is nothing more than a Mode1 parameter which is brutally added as the DeviceID like below.

Code: Select all

devID=Parameters["Mode1"]
Domoticz.Device(Name="Rain2Come", Unit=1, TypeName="Humidity",DeviceID=devID).Create()

So the questions are more on your part of the code than on mine

The code, without the above mentioned DeviceID change, I use at the moment can be found on https://github.com/seventer/raintocome/ ... ist-device
In the coming days I will add an emulation mode so the code can also be used outside The Netherlands or create some activity for testing purposes only.

Bottomline, my confusion about the relation between Idx,Id,Hwid,Unit etc. is still there. I hope someone can clarify this how this should work.

--Gerard

[avgays]

Hi @Dnpwwo,
Domoticz V3.6905 accepted pull request #1308
I created device with

Code: Select all

Domoticz.Device(Name="Test",  Unit=1, Type=244, Subtype=73, Switchtype=7, Used=1, DeviceID="aabbccddee").Create()

Device with an ID(IDX) of "29" and DeviceID (ID) of "aabbccddee" is created and listed in "Devices" page, but marked as "Unused".
Looks like "Used" parameter is ignored...

And is it any way to get value of "DeviceID" back to python script?

[gerardvs]

Yeah, funny things happen.
The DeviceID can only be set on certain devices. So yours is working, so is a Custom sensor. Weather(Humidity) doesn't work, they get a scrambled DeviceID. I think this is a Domoticz core behaviour and not a bug in the Python framework.

Used 0 or 1 doesn't matter.
However if you change the code in PythonObjects.cpp around line 556 from

Code: Select all

if (Used == 0) self->Used = Used;

into

Code: Select all

if ((Used != -1) && Used) self->Used = Used;

it does work as expected.

--Gerard

[Dnpwwo]

@gerardvs and @avgays,

I flipped the Used default to better align with how Domoticz works with other protocols and messed up the flag handling, pull request #1326 will fix that.

Also added Device.DeviceID attribute so that plugins can look it up. This:

Code: Select all

    for x in Devices:
        Domoticz.Debug("Device:           " + str(x) + " - " + str(Devices[x]))
        Domoticz.Debug("External ID:     '" + str(Devices[x].DeviceID) + "'")

logs:

Code: Select all

2017-03-11 09:51:44.193 (Lounge Amplifier) 'HardwareID':'11'
 ...
2017-03-11 09:51:44.195 (Lounge Amplifier) Device: 1 - ID: 24, Name: 'Amplifier - Power', nValue: 0, sValue: 'Off'
2017-03-11 09:51:44.195 (Lounge Amplifier) External ID: '12:34:56:78:90:12:34:56'
2017-03-11 09:51:44.196 (Lounge Amplifier) Device: 2 - ID: 25, Name: 'Amplifier - Main Zone', nValue: 0, sValue: '0'
2017-03-11 09:51:44.196 (Lounge Amplifier) External ID: '000B0002'

where Device 1 has a DeviceID specified on creation and the other was generated for hardware id 11.

AFAIK the DeviceID code works and devices are added to the database correctly. The WebServer code does appear to mangle some device IDs for unknown reasons for these types:

Code: Select all

							(dType == pTypeTEMP) ||
							(dType == pTypeTEMP_BARO) ||
							(dType == pTypeTEMP_HUM) ||
							(dType == pTypeTEMP_HUM_BARO) ||
							(dType == pTypeBARO) ||
							(dType == pTypeHUM) ||
							(dType == pTypeWIND) ||
							(dType == pTypeRAIN) ||
							(dType == pTypeUV) ||
							(dType == pTypeCURRENT) ||
							(dType == pTypeCURRENTENERGY) ||
							(dType == pTypeENERGY) ||
							(dType == pTypeRFXMeter) ||
							(dType == pTypeAirQuality) ||
							(dType == pTypeRFXSensor)

and I've added handling to stop it (only for plugins obviously). I'm worried we are going to end up chasing these special cases all the way through the code base

[avgays]

@Dnpwwo, thank you.

With Domoticz 6905 and switch Type 244 everything work ok.

BTW, seems it's a typo in wiki example "E.g: Domoticz.Log(Device[2].Name)" should be "E.g: Domoticz.Log(Devices[2].Name)"