Fixing Z-Wave for once and for all!

[forumdomo]

In other words the z-wave part of domoticz is shit. Any other raspi compatible OS with a stable z-wave implementation?

[waltervl]

An OS with with zwave implementation? Never heard of. Besides for some people the current OpenZwave implementation works perfectly, for some not.
If you read a little bit further on this forum there are 2 alternatives being programmed to connect ZwaveJS2MQTT with Domoticz (Python plugin and native MQTT).

[kiddigital]

In other words the z-wave part of domoticz is shit.

No it is not! Z-Wave is a (maybe even over) complex protocol and Domoticz uses one of the best Z-Wave implementation available (Open Z-wave 1.6) which has been incorporated into Domoticz.

Any other raspi compatible OS with a stable z-wave implementation?

Good luck finding one. Let us know if you found it.

If you look around carefully and read up on the many fora that have topics on Z-Wave (including this Domoticz forum), you will find that some users have reached a very stable
Z-wave network state but only after tuning a lot. This ‘road’ to get to an optimal network is not for every one and especially compared to ‘simpler’ protocols where people can reach a sufficiently stable state with a lot less effort.

Btw, feel free to spend some time and improve the ‘shit’

[solarboy]

I spend many hours per week resolving various Zwave problems, although admittedly I have a house with very thick stone walls and a mixed ZW & ZW+ network. The amount of wall channelling I have done to move nodes to better positions is incredible, especially when I am supposedly working with a "backwards compatible wireless network".

Originally I was under the impression that more routing nodes would solve the problems so I added these at strategic locations but the problems of dead nodes etc remained and then I read that networks with more than 50 nodes start to create more issues which I certainly noticed with all the extra messaging sometimes through non ZW+ nodes causing slowdowns (so not so backwards compatible after all). And then there is the unreliable pairing and unpairing rituals requiring I shut down my house while I open up difficult distribution boxes while tapping tiny buttons 3 times, over and over as my Aeotec stick continues flashing at me and my wife glares over my shoulder. I won't mention the need to rebuild the 10 or 20 scripts that reference any of the nodes devices. I must add that I am still on OZW 1.4, the attempt at using 1.6 rendered my home a prison of mindless scrabbling in the dark.

I kept a diary of the issues my system has had over the last 27 days and the problems have almost always been Zwave related, either hardware or software...

21/08/21 Fibaro Door sensor Zwave pool temp sensor failed, can't be re-included, is dead. Replaced with Qubino module temp sensor involving digging, channelling and extra wall boxes.
23/08/21 Power cut caused Qubino Thermostat Zwave module to change its set point to 23C so the AC stopped working on a really hot day.
24/08/21 Bedroom Fibaro Zwave motion sensor stuck "on". Woke up node to repair.
28/08/21 Node 104 (bathroom light and switch qubino) spontaneously removed from openzwave. Was easily re-included and scripts changed. Moved node (channelled to location closer to hub)
29/08/21 Had to reboot as zwave node 7, Fibaro shutter 2 (loungelarge2) was sending constant timeouts.
15/09/21 Node 7 started spamming Zwave with thousands of messages, rebooting Domoticz and power cycling the module didn't help, had to un-include/re-include and then rebuild all the scripts.

This has been going on all the time since 2018 when I started with my "smart home". I have taken care to evolve the mesh to be mainly ZW+ at it's core but often seemingly unrelated changes/faults can have implications on seemingly unrelated nodes. I am not sure I want to throw good money after bad and invest in a Zniffer...

I am not wanting to rant to find someone to blame although perhaps it appears that way, but the zwave issues have been very very real for me.

I have recently started using ZIgbee2mqtt with Slaesh's latest Zigbee stick with a 3dB antenna and the difference is night and day. I am able to build a very solid network with far less nodes that reacts much faster under the same conditions. I have some mixed lighting with both zwave and zigbee, sometimes the zwave lamps turn on a minute or more later although the opposite never happens. "Pairing" is a delight and the whole plugin just "works" as I would expect it to.

In no way am I unappreciative of the developers, especially Domoticz itself (respect gizmocuz) which is absolutely fabulous and "tight" piece of software and most issues for me have been mainly cosmetic and related to the browser cache failings. I don't want to be a moaner or a whiner and I do everything I can to deal with my own problems but there are real failings with zwave, not just with Domoticz but also with other platforms. I imagine the complaints about Zwave also cause a lot of stress to the genius's that are behind all this for which I empathise.

I briefly tried Openhab and HASS and found them much less user friendly as a beginner, but I have to say the Domoticz community can be very sensitive about any criticism which is bound to come when the errors affect everyone's day to day family life/sleep/relations, asides from the fact I have spent a vast (for me) amount of money on this, quite a lot simply to try and improve the Zwave mesh so that it is functional. I see a lot of people shot down for this which I think its counterproductive.

All that being said, I will absolutely be sticking with Domoticz for sure and am really looking forward to some possible relief with the Mqtt Discovery developments in the pipeline (and also thanks to the guys efforts with zwavejs2mqtt) and look forward to my home eventually settling down and not frying my nerves so much.

Apologies for any offences caused. Huge thank you to everyone sharing their expertise.

[rrozema]

21/08/21 Fibaro Door sensor Zwave pool temp sensor failed, can't be re-included, is dead. Replaced with Qubino module temp sensor involving digging, channelling and extra wall boxes.

Just to pick one of your issues. I recently had a similar experience with one of my devices and found a way around it, without having to exclude and re-include the device, hence avoiding using another of the maximum of 250 available z-wave device id's, nor having to adjust any scripts. And ofcourse no digging, wall boxes etc...

To let you in on why I think this is a good work-around for the problem I need to give some background information first:

Most z-wave related problems are caused by Domoticz (no longer) having the correct device information in memory for your node. There are 4 ways (A.F.A.I.K) Domoticz updates this in-memory device information:

• when an (unsollicited) value-update is received for a device Domoticz doesn't know about yet/any more,
• when initializing the openzwave library,
• when including a device,
• when you request a node's NIF via the Refresh Node Information button

When a device is (re-)added because a value update is received for an 'unknown' device (situation 1), only the device information for that single value gets updated in Domoticz, plus it may not be complete and even not fully correct, as the node's information is not retrieved from the z-wave database (= the file somewhere inside the Config folder for your node). So any overrides or specific details will be missed and any other node capabilities won't be known either. Type information for the single device to store this value into is "guessed" from the value received. This is not a very good situation and should be avoided as much as possible. (More on this with situation 4).

When initializing the openzwave library (situation 2), the node information for all nodes is read from the openzwave cache (the xml file in your Config folder). Reading the xml file is -A.F.A.I.K- pretty solid code and it seems to work well, although I do have my doubts on a few very little things, among which is retrieving the node's options. BUT, the information in this cache is only as good as it was at the moment the information was put in there. And that's where the problem lies: writing the cache file is not 'transactional': the cache file gets updated with life status information without any protection for when/if the current operation fails. For example, if you're refreshing a node's information, first the device's information is removed from memory and a call is made to the node to send it's NIF (i.e. have the node send it's meta information on it's capabilities). If the node doesn't receive the request or the node can't respond (correctly), the meta information for the node will still be erased from Domoticz' memory. And if in this state you shutdown your domoticz or Domoticz crashes, the current device state in memory will automatically be written into the cache file (i.e. with NO device information for this node). The next time you boot Domoticz, the node will no longer be known to Domoticz and the device information for it will not be restored in memory: The node is still included with your controller, but Domoticz doesn't know about it any more. This is only one scenario how the cache can become corrupted, other things can happen too that can corrupt the information in memory, resulting in other corruptions in the cache file, and thus mis-reading the device's meta information upon the next re-start of Domoticz.

When including a New node (situation 3), Domoticz -among other things- requests the node to send it's NIF and upon receipt the meta information is built both in memory and in the cache (xml) file. Sometimes, especially in bussier networks, the response doesn't arrive or doesn't arrive completely. The node may be included with your controller, but Domoticz doesn't know about it. Typically you see timeout messages reported by the new node ID in Domoticz' log file, without the node being listed in Domoticz.

When refreshing a node (situation 4), Domoticz erases the node's meta information from memory and sends out a request for a NIF to the node, waiting for the response(s) to rebuild the in-memory and cached meta information. Same problems as when including a new node can occur here. Refreshing node information is actually our way to fix most of the problems caused by situation 1, 2, and 3 not completing correctly, but it can be a somewhat tedious process because the refresh can be somewhat unreliable itself.

So that is how Domoticz gets to know the meta information on your node(s), and now we get to one other component of the specific problem with this node: it is marked 'dead'. A 'dead' node is one that the controller or other nodes reportedly haven't been able to communicate with for quite some time already. Since there is only limited bandwidth to send and receive z-wave messages, nodes that don't respond can slow down the entire network, making devices that do work seem slow or even non-operational. For this reason z-wave has a provision to isolate such non-responding nodes so that the other nodes can continue to operate. This isolation is the 'dead' status: OpenZwave will no longer actually send out any messages to a dead node. Instead it will automatically give such message a failed status, saving a lot of time in the process. BUT: not sending out any messages to the node, also means we can't send it any messages to solve the problem why it is not responding...

Now that we know all that, what I did was this: I selected the "dead" node, then clicked the 'is node broken' button. It showed me: 'Node is OK' and I pressed 'OK'. But more importantly: the status of the node is now changed from 'dead' (the red icon) into 'OK' (the green checkmark). So after this I again selected the node and now I clicked the 'Refresh Node info' to reqeuest the node's NIF. Now I waited for the node to report it's NIF (Node Information Frame) (have the log shown in another window and wait for the 'Add_Value' messages) and the device was correctly registered in Domoticz and in the cache file again.

Receiving the responses to the NIF request can be somewhat unreliable: sometimes not all messages sent by the node are received, sometimes no messages are received at all and sometimes it just takes a very long time for the messages to arrive (timeout messages sent by the node during the process are not uncommon on nodes with multiple capabilities). And also, sometimes the request for the NIF doesn't arrive at the node, so it will not send any messages at all. This is why you need to keep a log window open whenever you have OpenZwave request a node's NIF to see if the add_value messages have come in. Do be patient, as it may take a couple of minutes before the responses come in if your mesh routing is very bad. However, if they don't come in or not all come in, you may have to retry the NIF request (by doing a Refresh Node information) after 5 or so minutes. If after a lot of retries you still don't get the in memory meta information to come in correctly you may still have to (temporarily) move the node closer to the node or the controller closer to the node. And if it still doesn't work then, this is the point to give up and re-include the node. If the controller still knows about the node, it may assign it the same z-wave node-id. So don't reset the node, just re-include the node and start the inclusing process like you would on a newly bought node. Again you must be patient (the NIF request process takes a long time here too) and you may have to repeat a few times, as starting the inclusion process is often the same procedure as starting the exlusion procedure on many nodes, i.e. while the controller is waiting for an inclusion, your node may be performing an exclusion, you just don't have a way of knowing which one it is doing. So just try starting the inclusion in the node a couple of times, but wait at least 5 minutes after each try and keep a close eye on Domoticz' log file to see if any add_value messages show up. If they do, your mission is accomplished!

Godspeed!

Y.M.M.V. (Your Milage May Vary), it hasn't worked for me in all situations too, but at least it's another tool in our toolbelt to get non-working z-wave nodes back into operation.

[solarboy]

Hey thanks for your reply rrozema, that's a very comprehensive description of some solid methods to revive troublesome nodes. I do often use the "Refresh Node" function in combination with making sure the device is awake but I hadn't yet used the "Is dead node" function to revive a dead node, but I have used it to remove actually failed hardware. Often I also revert to using the Zensys tools to remove particularly stubborn nodes although as I am on Linux this involves shutting down Domoticz and putting the stick in my laptop.
The more tools the better !

[lost]

some users have reached a very stable Z-wave network state but only after tuning a lot.

My z-wave network is very stable, but must admit I gave up with "tuning a lot". I fact, the less I modify/tune the setup the best is works!
I also skipped 2020 versions, first including OZW 1.6 as 1.4 proved very stable in v4.10717. This probably avoided former issues others had.

With 2021.1 (OZW 1.6), I appreciate a faster startup (from 10/15mn in 1.4 to 2/3mn in 1.6 to get "all awake nodes queried" message)... when this works flawlessly: 1/3 startups shows issues (dead nodes or even signal 11/SIGSEGV in OZW driver during bring-up) and a domoticz service restart mostly solve the issues (just waking up devices,sometimes, but can also trigger a SIGSEGV).

After correct bring-up, that's stable & I sometimes have 2 month uptime between restarts after some raspbian updates.

Also thanks rrozema for the "is node dead" explanations/tip. This may have saved a few exclusion/re-inclusions in the past: Maybe this information could be added in the wiki!

Hope OZW will find a new maintainer or some other library allowing native z-wave support in Domoticz will show-up: Looks silabs have some plans to fill current gap in open source projects by the end of this year. Worst issue is no support for gen7 controllers. At the time, gen5 are still sold but maybe in a few months this'll not be true anymore and a z-wave controller dying will become a big issue for Domoticz users (same for Jeedom).

[Plantje]

Thanks for the replies! There is definitely something to it! Especially what rrozema points out.

It's almost a year ago that I started this thread and I think it is about time to draw some conclusions.

Currently my Z-Wave setup is pretty much ok. I am running version 2020.2.13150. So that is with OZW 1.6. Due to the fact that the Windows machine running Domoticz at times freezes I have now implemented a nightly restart of the entire machine. I had few occasions where in the morning all my Z-Wave devices were gone, but all in all it is "ok'ish"

Still, I have decided to move away from Domoticz and switch to Athom Homey. Why? Because it simply works! And that is the main driver for my home automation. The amount of time I have spent re-adding Z-Wave nodes, installing new versions of plugins, installing new Domoticz versions, figuring out how to connect to Google Home, adding my Tesla again, debugging time outs of devices, connecting to Toon thermostat, enabling HTTPS etc etc.

I used to have DVB Link/TV Mosaic to watch and record live TV from Ziggo. Started doing this with Windows Media Center, then with Kodi. When DVB Logic stopped with their product (it is currently available as open source for people that are interested) I briefly tried TV Headend. It worked.... "ok'ish". At some point I bit the bullet and switched to the Ziggo Next with Ziggo Go to view recordings in other locations. What a BREEZE! This is soooooo relaxed! IT JUST WORKS! No more blocks in the screen, no more tickers (the "nieuwskrant" in the bottom of the screen) that don't run fluently....Really great!

I used to have my own library of CD's and had them all ripped to my hard drive. Nicely leveled to a specific dB level, with MP3 tag added all tags. Every machine that I had needed to have 128GB of additional storage so I could put my entire music collection on it. Was really nice. But what a shit load of work! Now I have Spotify.....and guess what.... IT JUST WORKS! I have my playlists and all of the music I want everywhere!

Same goes for movies. Not as extensive as my CD library, but what a breeze having Netflix!

I have been using Windows Mobile for years. Although it was a great OS I was always looking for ways to make something work that on Android and iOS JUST WORKS!

What I read in a previous post really applies: criticism is not really appreciated. At times when people give some feedback that is not all that positive the responses are all in direction of "Then you should do it better yourself!" The fact that someone is not happy with something doesn't mean he or she knows how to do it better.
In the Dutch Domoticz forum it is even worse... As soon as you DON'T want Zigbee2MQTT but use a Zigbee stick and Deconz you're doing it wrong. If you use the Tuya plugin you are basically regarded an idiot. How can you let Chinese people in your home?!? If you use a Neo Coolcam smart plug on your washing machine and have a question on it you don't get an answer.... only "Should I call the fire department?"

It was a nice run. First steps in home automation at an acceptable price! I think the people from Machinon perhaps have something good going. As I understood it, they will be delivering all in one out of the box Domoticz solutions. I use their theme in Domoticz and found it to be really nice: https://www.machinon.com/
I am not sure when I exactly will switch as the Homey Pro is still pretty expensive and I do need some time to initially set things up...but then.... I can hardly wait!

[hestia]

If you read a little bit further on this forum there are 2 alternatives being programmed to connect ZwaveJS2MQTT with Domoticz (Python plugin and native MQTT).

Hi waltervl
I've read a lot in this forum and I'm still a bit confused about the possibilities with z-wave. I understand an alternative to open z-wave is ZwaveJS2MQTT, and that there are 2 ways to do it: Python plugin and native MQTT. Could you summarize the status of both please?

[heggink]

The plugin supports all my devices but will not be maintained because if the native mqtt development.
The mqtt development studio supports all my devices but I have not thoroughly tested everything. It's been running for a couple of days on my test system. If it continues to function correctly, I will switch over.
The mqtt wiki explains current device support.

Sent from my SM-G980F using Tapatalk

[hestia]

Thanks heggink for such a quick answer
I didn't know about the mqtt wiki being updated with this.
"Domoticz devices will be created once data is received"
So if I have already installed zwavejs2mqtt and the last version of dz, it should work. I'll try it

[heggink]

Correct! Interested to see your feedback. Keep us posted.

Sent from my SM-G980F using Tapatalk

[hestia]

The plugin supports all my devices but will not be maintained because if the native mqtt development.
The mqtt development studio supports all my devices but I have not thoroughly tested everything. It's been running for a couple of days on my test system. If it continues to function correctly, I will switch over.
The mqtt wiki explains current device support.

Sent from my SM-G980F using Tapatalk

Did you already switch over?

[heggink]

Yes. 2 weeks ago. And for both zwave as well as zigbee. Works really well. No issues so far.

Sent from my SM-G980F using Tapatalk

[Ragdag]

Currently have a setup with:

• 1 Aeotec zwave stick
• 27 Fibaro Dimmers
• 3 Fibaro Smoke Sensors
• 5 Fibaro Wall Plugs
• 6 Fibaro Roller Shutter
• 2 Forest Shuttle Zwave
  
  It works pretty stable most of the time, do occasionally have a corrupt ozcw.xml which leads to all nodes being shows as unknown, after deleting the file and restart it resolves most of the time.
  The only thing I need to enable is polling on all the dimmers because otherwise after an hour Domoticz will lose the Energy Usage.
  
  Very interested in the MTTQ integration, is there somewhere with a good write-up of the installation yet?

[mgugu]

The procedure is well descibed at https://www.domoticz.com/wiki/MQTT#Auto ... .28Beta.29
in summary:
- Allow Homeassistant integration in the zwavejs2mqtt gateway
- Set a prefix (ie "zwaveauto") in the zwavejs2mqtt gateway
- restart zwave2jsmqtt gateway
- create a new mqtt hardware in domoticz and delete default topic in/topic out
- In the field autodiscovery prefix put the prefix you have set in zwavejs2mqtt (zwaveauto)
- allow new devices in domoticz
- restart domoticz (not mandatory)
That's it, it should work.
My experience on zwave:
I am running about 50 nodes for many years with ozw. No major problem when I switched from ozw 1.4 to 1.6. Main problem concerned a popp rain sensor which is is battery powered and update cycle set by default to 24h.
After re-inclusion I mofified the update cycle to 2h. Everyting work very silent for many month now.
After reading many posts from @Heggink I decided to test the domoticz zwave2mqtt domoticz plugin, so I created a zwave2mqtt gateway on a different machine and with a different zwave network.
Everything was OK and I use that solution for new nodes (ozw still running for other nodes). I have now 4 nodes in that network.
I recently wanted to test the mqtt autodiscovery feature: It works like a charm. Some minor problem concerning a blind which was not discovered in the same way (blind percentage vs blind inverted percentage) but easyly corrected.

[heggink]

The blinds are indeed the only outstanding 'thing'. It appears that there was an old inconsistecy in domoticz handling blinds that is now more consistent after the autodiscovery implementation but it has consequences in a couple of areas (e.g. having to switch to an inverted device). That discussion is ongoing on GitHub and probably needs to conclude before a new stable is ready but, afaict, it's the only outstanding item. The last HA autodiscovery change was over a week ago and my system has been rock solid since.

Sent from my SM-G980F using Tapatalk

[hestia]

Yes. 2 weeks ago. And for both zwave as well as zigbee. Works really well. No issues so far.

Sent from my SM-G980F using Tapatalk

Thanks, I'll move soon

[Ragdag]

The procedure is well descibed at https://www.domoticz.com/wiki/MQTT#Auto ... .28Beta.29
in summary:
- Allow Homeassistant integration in the zwavejs2mqtt gateway
- Set a prefix (ie "zwaveauto") in the zwavejs2mqtt gateway
- restart zwave2jsmqtt gateway
- create a new mqtt hardware in domoticz and delete default topic in/topic out
- In the field autodiscovery prefix put the prefix you have set in zwavejs2mqtt (zwaveauto)
- allow new devices in domoticz
- restart domoticz (not mandatory)
That's it, it should work.
My experience on zwave:
I am running about 50 nodes for many years with ozw. No major problem when I switched from ozw 1.4 to 1.6. Main problem concerned a popp rain sensor which is is battery powered and update cycle set by default to 24h.
After re-inclusion I mofified the update cycle to 2h. Everyting work very silent for many month now.
After reading many posts from @Heggink I decided to test the domoticz zwave2mqtt domoticz plugin, so I created a zwave2mqtt gateway on a different machine and with a different zwave network.
Everything was OK and I use that solution for new nodes (ozw still running for other nodes). I have now 4 nodes in that network.
I recently wanted to test the mqtt autodiscovery feature: It works like a charm. Some minor problem concerning a blind which was not discovered in the same way (blind percentage vs blind inverted percentage) but easyly corrected.

Thanks for this, it helped me get started.
Currently, have ZWavejs2mqtt running as a docker image on a spare rpi I had lying around and have been able to add my aeotec zwave stick and add 1 Fibaro Wall Plug.
Now I'm trying to connect Domoticz to it but can't get the plugin to connect.


Running with the default docker-compose template https://github.com/zwave-js/zwavejs2mqt ... ompose.yml

In ZWavejs2mqtt I have the following

[hestia]

port and address are those for the mqtt broker, not the ws server
1883 and localhost for me