[REQUEST] Ikea Trådfri Gateway (smart lights hub controller) plugin?

[moroen]

So I don't have to restart domoticz, just disable the observe changes and wait a while.

Does this help?

Kind regards,
Bart

I've not been able to recreate your problem with using both HASS and domoticz, but I do suspect that this might be related to timeouts leaving behind open UDP-connections, since you noticed that the plugin seemed to respond slowly before locking out HASS.

Please try updating to the latest version of py3coap, and see if that fixes this!

Regards,
M

Hi Moroen,

- Before the tests, I upgraded already to IKEA Tradfri Plugin - version 0.8.1. This is the latest at this moment, not?
- is it expected behaviour that you can NOT use "observe changes" with other connections to the IKEA bridge? Than the plugin becomes quit useless for me. I want to be as much as independent from any software. Playing around now with node-red, home-assistant, domoticz, openhab, etc... If a state is changed somewhere, I want to have it reflected in all my tools. It's really a bummer if the plugin locks out other connections
- If you have not been able to recreate my problems: does it mean that you have a working setup with both hass and domoticz? Or you just haven't got the opportunity to test this combination?

I alsof forgot to mention: I have 4 instances of home-assistant running on different servers. The "IKEA integrations" on al these servers talk smoothly to the IKEA bridge, so at least it is possible to have multiple connections to the bridge...

Kind regards,
Bart

That's the latest version of the plugin, but you need to upgrade to the latest version of py3coap as well. And no, it's not expected behavior that using "observe changes" should lock out other connections, apps or programs. I haven't tried the combination of Domoticz and HASS specifically at this time, but have seen no trouble running multiple instances of domoticz at the same time or observed any problems using the IKEA Tradfri app while running multiple instances of domoticz talking to the same gateway. There was, as noted in by wvdweerd, an error in py3coap that created multiple connections on timeouts, and it's possible that this somehow have blocked other connections, this should be fixed in the latest version of py3coap.

I do know, however, that HASS uses COAP observations to update the state when the state is changed, either by HASS it self, by another program or a Tradfri remote. I also know, that getting stable observations for a prolonged time has proven difficult to achieve, which is the reason that this plugin uses polling, which is less efficient, but at least produces consistent updated information, albeit somewhat delayed.

So I really don't know why using both HASS and Domoticz with this plugin breaks HASS. In your expericence running multiple instances of HASS is working smoothly, and in my experience running multiple instances of Domoticz is working smoothly as well.

I might get around to set up HASS to investigate this further, but my main priority is to make the plugin work as smoothly and stable on Domoticz as I can...

Regards,
M

[bartplessers]

Hi Moroen,

No offence. Your hard work is really really really appreciated!!!
Is there an easy way how I can check open connections to/from IKEA bridge? Can I help with debugging?

kind regards,
B

[moroen]

There's (yet) another release available on github, version 0.9.0, along with py3coap 0.7.3. Updating both the plugin and py3coap is recommended. Users of coapcmd only needs to update the plugin.

Major changes:
- Custom sensor for monitoring the battery levels on battery-powered devices (thanks to wpclipart.com and Logread for the icons)

Bugfixes:
- (plugin) Fixed onStart error when a device has been removed from the gateway
- (py3coap) Fixed bug in underlying gocoap library where connections to the gateway wasn't closed properly on timeouts

A note about battery levels:
I'm still not confident when it comes to the accuracy of the battery levels as they are reported from the gateway. Battery level for the curtains/blinds appears accurate, battery levels for remotes not so much... I really have no idea on how or when the gateway updates the battery levels, but the plugin shows the levels as they are read from the gateway, and there's nothing that can be done on the plugin side to improve this...

Code: Select all

$ cd domoticz/plugins/IKEA-Tradfri
$ git pull
$ sudo -H pip3 install -U py3coap

Regards,
M

[moroen]

Hi Moroen,

No offence. Your hard work is really really really appreciated!!!
Is there an easy way how I can check open connections to/from IKEA bridge? Can I help with debugging?

kind regards,
B

No offence taken!

On linux you can use lsof to watch for connections from a process or ss to check ports and connections. Have you tried the latest version of py3coap, and if so, do you still have the same problems?

Regards,
M

[moroen]

Hi Moroen,

No offence. Your hard work is really really really appreciated!!!
Is there an easy way how I can check open connections to/from IKEA bridge? Can I help with debugging?

kind regards,
B

I've installed the latest version of HASS, and so far I can confirm your observations. It appears that when the plugin gets the list of devices, HASS gets a timeout it's unable to recover from until it resets the observations (I think this reset happens once every two minutes). Simple connections from the plugin to the gateway, like switching a light state, appears not to cause this kind of timeouts in HASS...

Currently the underlying gocoap uses one-off connections, it opens a connection, performs the query, and then closes the connection again. When getting the list of devices, this means quite a lot of subsequent requests to the gateway in as short amount of time, it might be possible that this somehow throws of the connection HASS uses... I'll experiment with gocoap and try to use a persistent connection instead, and see what happens...

Regards,
M

[bartplessers]

Hello Moroen,

I upgraded to the latest and greatest

Code: Select all

# stop domoticz
sudo systemctl status domoticz.service
sudo systemctl stop domoticz.service

# set security
sudo chown domoticz:domoticz -R /home/domoticz

# switch to user
sudo su -s /bin/bash domoticz

# change directory
cd ~

# install
cd /home/domoticz/domoticz/plugins/IKEA-Tradfri
git pull
sudo -H pip3 install -U py3coap


# start domoticz
sudo systemctl start domoticz.service
sudo systemctl status domoticz.service

However, my observations:
01. hass working, domoticz plugin disabled
02. enable domoticz plugin, observe changes = false
03. hass becomes unresponsive for lights integrated with IKEA plugin
04. wait 5min
05. hass becomes responsive
06. change domoticz plugin, observe changes = true, timeout = 300sec
07. hass becomes unresponsive for lights integrated with IKEA plugin
08. wait 5min
09. hass becomes responsive again
10. wait 5min
11. domoticz reflects new state of light (set by hass)
12. hass becomes unresponsive!!!
13. wait 5min
14. hass stays unresponsive
15. domoticz can control light
16. change domoticz plugin, observe changes = false
17. restart hass
18. hass working, domoticz working
19. wait 5min
20. hass becomes unresponsive!!! (even with observe changes = false)
21. restart hass
22. hass sill unresponsive!!!
23. disable domoticz plugin
24. restart hass
25. hass working, domoticz plugin disabled

Conclusion:
- there still is a conflic of interest between hass and domotic connections to the IKEA bridge

Kind regards,
B

[bartplessers]

I'll experiment with gocoap and try to use a persistent connection instead, and see what happens...

Regards,
M

Okay, thanx a lot!!!

[jeroenkl]

There's (yet) another release available on github, version 0.9.0, along with py3coap 0.7.3. Updating both the plugin and py3coap is recommended. Users of coapcmd only needs to update the plugin.

Major changes:
- Custom sensor for monitoring the battery levels on battery-powered devices (thanks to wpclipart.com and Logread for the icons)

Bugfixes:
- (plugin) Fixed onStart error when a device has been removed from the gateway
- (py3coap) Fixed bug in underlying gocoap library where connections to the gateway wasn't closed properly on timeouts

A note about battery levels:
I'm still not confident when it comes to the accuracy of the battery levels as they are reported from the gateway. Battery level for the curtains/blinds appears accurate, battery levels for remotes not so much... I really have no idea on how or when the gateway updates the battery levels, but the plugin shows the levels as they are read from the gateway, and there's nothing that can be done on the plugin side to improve this...

Code: Select all

$ cd domoticz/plugins/IKEA-Tradfri
$ git pull
$ sudo -H pip3 install -U py3coap

Regards,
M

Works fine! thxs!

[dressie]

Hi,

I did the update on my Synology and since this morning I get this:
2020-02-03 11:36:36.099 Error: IKEA hardware (28) thread seems to have ended unexpectedly
2020-02-03 11:43:22.731 Error: IKEA hardware (28) thread seems to have ended unexpectedly
2020-02-03 11:56:28.324 Error: IKEA hardware (28) thread seems to have ended unexpectedly

More users seeing this with the 0.9.0 version?

P.S. I really appriciate your work here @moroen !

EDIT: reboot made the problem go away. Will keep an eye on it.

[moroen]

Hi,

I did the update on my Synology and since this morning I get this:
2020-02-03 11:36:36.099 Error: IKEA hardware (28) thread seems to have ended unexpectedly
2020-02-03 11:43:22.731 Error: IKEA hardware (28) thread seems to have ended unexpectedly
2020-02-03 11:56:28.324 Error: IKEA hardware (28) thread seems to have ended unexpectedly

More users seeing this with the 0.9.0 version?

P.S. I really appriciate your work here @moroen !

EDIT: reboot made the problem go away. Will keep an eye on it.

As the onHeartBeat function performs more and more actions, it's not unexpected that this error creeps up, especially on slower devices. You're using coapcmd as well, which is a bit slower than py3coap, so it's even more likely to appear. In it self, besides filling up the log with messages, this error is quite harmless, but that doesn't mean that I'm not going to try fixing it...

I've been thinking, and have come to the conclusion that it's probably not necessary to have separate polling for battery devices. There is nothing suggesting that reading the value from the gateway somehow drains the battery of the remote in any way, so I'll just change this to be updated along with the regular polling. This will shave some time from the onHeartBeat function, and hopefully get it to stay below the timeout threshold, which is what's generating the error you see in the first place...

Regards,
M

[moroen]

I'll experiment with gocoap and try to use a persistent connection instead, and see what happens...

Regards,
M

Okay, thanx a lot!!!

Just a small update: Initial tests suggests that using a persistent connection to the gateway, enables the peaceful coexistence of both HASS and Domoticz! I'm not quite ready to release this yet, a bit more development and testing needs to be done, but at the moment it looks promising! As a side note, until I started this, I didn't release quite how often the IKEA-Gateway actually times out, so I need to spend some time trying to make gocoap/pycoap a bit more robust and try to handle timeouts on "library-level" before raising timeouts to the plugin...

Regards,
M

[JimmyH1969]

Hi moroen,

I'm still following this thread and updated to 0.9.0 this mornig. It al seems to work fine on Windows. I haven't seen any errors for a while now.
Tnx again!

[bartplessers]

Just a small update: Initial tests suggests that using a persistent connection to the gateway, enables the peaceful coexistence of both HASS and Domoticz!

That's great news!

[jeroenkl]

I'll experiment with gocoap and try to use a persistent connection instead, and see what happens...

Regards,
M

Okay, thanx a lot!!!

Just a small update: Initial tests suggests that using a persistent connection to the gateway, enables the peaceful coexistence of both HASS and Domoticz! I'm not quite ready to release this yet, a bit more development and testing needs to be done, but at the moment it looks promising! As a side note, until I started this, I didn't release quite how often the IKEA-Gateway actually times out, so I need to spend some time trying to make gocoap/pycoap a bit more robust and try to handle timeouts on "library-level" before raising timeouts to the plugin...

Regards,
M

Hi Moroen,

Are you running on Buster or Stretch? I have issues running it on Buster and not on Stretch.

2020-02-04 06:40:31.320 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:40:45.347 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:40:59.387 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:42:09.556 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:42:23.608 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:43:33.729 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:43:47.769 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:45:12.498 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:45:26.541 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:46:36.798 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:46:50.851 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:47:04.903 Error: Ikea hardware (2) thread seems to have ended unexpectedly

Update: just found zigbee2mqtt plugin running together with Ikea plugin on Buster are having issues. When stopping 1 of them it runs fine....

[dressie]

Well....the reboot did work, not for long. It's so bad now that the Ikea app won't connect from time to time and looses it's settings.
Is there a way to go back to the 8 version? The 0.9.0 really killed it.

[moroen]

Well....the reboot did work, not for long. It's so bad now that the Ikea app won't connect from time to time and looses it's settings.
Is there a way to go back to the 8 version? The 0.9.0 really killed it.

I'm working on a fix, hopefully it should be available later today! Until then, this should revert you back to 0.8.1:

Code: Select all

$ cd domoticz/plugins/IKEA-Tradfri
$ git reset --hard b17e467de7966a8853c0dda03550fd7e1b047446

Edit: Increasing the time between polling for changes also will reduce the problem with the IKEA app loosing it's connection to the gateway...

Regards,
M

[moroen]

Okay, thanx a lot!!!

Just a small update: Initial tests suggests that using a persistent connection to the gateway, enables the peaceful coexistence of both HASS and Domoticz! I'm not quite ready to release this yet, a bit more development and testing needs to be done, but at the moment it looks promising! As a side note, until I started this, I didn't release quite how often the IKEA-Gateway actually times out, so I need to spend some time trying to make gocoap/pycoap a bit more robust and try to handle timeouts on "library-level" before raising timeouts to the plugin...

Regards,
M

Hi Moroen,

Are you running on Buster or Stretch? I have issues running it on Buster and not on Stretch.

2020-02-04 06:40:31.320 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:40:45.347 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:40:59.387 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:42:09.556 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:42:23.608 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:43:33.729 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:43:47.769 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:45:12.498 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:45:26.541 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:46:36.798 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:46:50.851 Error: Ikea hardware (2) thread seems to have ended unexpectedly
2020-02-04 06:47:04.903 Error: Ikea hardware (2) thread seems to have ended unexpectedly

Update: just found zigbee2mqtt plugin running together with Ikea plugin on Buster are having issues. When stopping 1 of them it runs fine....

This is a known problem, the longer a plugin onHeartBeat takes to run, the more freqeuent these errors are. If you have more than one plugin running, the chances of a timeout increases. I'm working on reducing the impact from IKEA-Tradfri, hopefully I'll be able to release a fix later today. Redusing the polling intervall also will help, normally I would recommend at least 300 seconds between each poll...

Regards,
M

[moroen]

Well....the reboot did work, not for long. It's so bad now that the Ikea app won't connect from time to time and looses it's settings.
Is there a way to go back to the 8 version? The 0.9.0 really killed it.

There's a new version available (0.9.1) that hopefully should address this and the (frequent) "thread seems to have ended unexpectedly"-error. There is no need to update py3coap.

Major changes:
- Batteries are polled along with other devices, using the same polling interval
- Monitoring batteries or not are configurable from the IKEA Hardware page

As before, it's recommended that the polling interval isn't to short, frequent polling used to freeze the gateway, now it appears to invalidate IDENT/PSK-pairs for other connections (including the IKEA App) and possibly still freeze the gateway. An interval of about 60 seconds should work, but a longer interval is recommended!

Regards,
M

[moroen]

Just a small update: Initial tests suggests that using a persistent connection to the gateway, enables the peaceful coexistence of both HASS and Domoticz!

That's great news!

There's an updated version of py3coap (0.8.0) which at least on my testing setup enables working with both HASS and domoticz at the same time. As a added bonus it also offers a slightly faster connection to the gateway (listing devices using plugin.py reduced from 0.6 s to 0.3s on my setup), should be a bit gentler on the gateway and offers somewhat better handling of timeouts. It's available on testpypi, and after a bit more testing, it will be available on the official index.

Would you be as kind as to try it out, and confirm that it works alongside HASS? Install with:

Code: Select all

$ sudo -H pip3 install -U -i https://test.pypi.org/simple/ py3coap

Regards,
M

[bartplessers]

Would you be as kind as to try it out, and confirm that it works alongside HASS? Install with:

Code: Select all

$ sudo -H pip3 install -U -i https://test.pypi.org/simple/ py3coap

Regards,
M

Code: Select all

# ======================================================================
# Domoticz IKEA Tradfri plugin update
# ======================================================================
# https://www.domoticz.com/forum/viewtopic.php?p=235735#p235735


# stop domoticz
sudo systemctl status domoticz.service
sudo systemctl stop domoticz.service

# set security
sudo chown domoticz:domoticz -R /home/domoticz

# switch to user
sudo su -s /bin/bash domoticz

# change directory
cd ~

# install
cd /home/domoticz/domoticz/plugins/IKEA-Tradfri
git pull
sudo -H pip3 install -U -i https://test.pypi.org/simple/ py3coap


# start domoticz
sudo systemctl start domoticz.service
sudo systemctl status domoticz.service

Up and running
First observations:
coexistence between domoticz and hass:
- (observe change: false) --> OK
- (observe change: true) --> OK

just running now for 5 minutes, will let it run as long as it works

nice job!

Bart