Z-Stick: All devices suddenly timeout

[arjanm83]

I'm using Domoticz 4.97 and using only the Z-Wave functionality with a Aeotec Z-Stick Gen 5.
This has been running for a year now and worked fine.
I haven't changed anything lately, no new devices have been added for weeks.

Suddenly out of the blue, since this evening, I'm receiving timeout notifications for each and every z-wave device in my house, see the log below.
I've tried restarting, powering off and a fresh boot.

I can't control any light anymore. Updated values from my sensors are still received (power, temperature, etc).
What happened? And how can I fix this?


2018-12-09 23:42:49.902 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 4 (0x04)
2018-12-09 23:42:50.903 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 5 (0x05)
2018-12-09 23:42:51.905 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 6 (0x06)
2018-12-09 23:42:52.906 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 7 (0x07)
2018-12-09 23:42:53.907 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 9 (0x09)
2018-12-09 23:42:54.908 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 11 (0x0b)
2018-12-09 23:42:55.910 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 13 (0x0d)
2018-12-09 23:42:56.913 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 14 (0x0e)
2018-12-09 23:42:57.914 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 16 (0x10)
2018-12-09 23:42:58.915 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 17 (0x11)
2018-12-09 23:42:59.916 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 18 (0x12)
2018-12-09 23:43:00.918 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 21 (0x15)
2018-12-09 23:43:01.921 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 22 (0x16)
2018-12-09 23:43:02.923 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 23 (0x17)
2018-12-09 23:43:03.924 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 24 (0x18)
2018-12-09 23:43:04.925 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 26 (0x1a)
2018-12-09 23:43:15.122 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 35 (0x23)
2018-12-09 23:43:16.123 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 36 (0x24)
2018-12-09 23:43:17.124 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 37 (0x25)
2018-12-09 23:43:18.126 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 38 (0x26)
2018-12-09 23:43:19.700 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 39 (0x27)
2018-12-09 23:43:20.730 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 40 (0x28)
2018-12-09 23:43:21.731 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 41 (0x29)
2018-12-09 23:43:22.733 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 42 (0x2a)
2018-12-09 23:43:23.734 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 43 (0x2b)
2018-12-09 23:43:24.736 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 44 (0x2c)
2018-12-09 23:43:25.737 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 47 (0x2f)
2018-12-09 23:43:26.739 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 48 (0x30)
2018-12-09 23:43:27.740 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 51 (0x33)
2018-12-09 23:43:28.742 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 52 (0x34)
2018-12-09 23:43:29.744 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 58 (0x3a)
2018-12-09 23:43:30.748 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 59 (0x3b)
2018-12-09 23:43:31.749 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 60 (0x3c)
2018-12-09 23:43:32.751 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 62 (0x3e)
2018-12-09 23:43:33.752 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 63 (0x3f)
2018-12-09 23:43:34.754 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 64 (0x40)
2018-12-09 23:43:35.757 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 65 (0x41)
2018-12-09 23:43:36.759 Status: OpenZWave: Received timeout notification from HomeID: 3747932707, NodeID: 66 (0x42

[Nautilus]

Not quite as bad, but I'm seeing something quite similar. A lot of timeout notifications for z-wave devices (mainly magnetic switches and PIR's) but still they seem to work normally (show if motion is detected and if a door is open). I also have Aeotec z-stick gen 5 and running Domoticz v. 4.10241.

[arjanm83]

Another reboot showed the following errors:

2018-12-09 23:33:00.002 EventSystem thread seems to have ended unexpectedly
2018-12-09 23:33:00.002 Error: Scheduler thread seems to have ended unexpectedly
2018-12-09 23:33:00.002 Error: Z-Stick hardware (2) thread seems to have ended unexpectedly

And another reboot, and waiting for 30 minutes to have all devices queried, all devices did work again.
I have no idea what happened, but it's back to normal now. No more timeout notifications.

If anyone has an idea what could have caused this situation, please feel free to let me know.

[jefft]

I get the same, every few months or so. Power down (not reboot), pull the z-stick, then put it back and start the system again. Generally, it's fine after that.
At worst - has happened a couple of times - something messes up the z-wave network data in the stick and I can't see any of my devices. Aeotec doesn't know the cause of that, suspects a bad device; that's happened twice now in the last few years and I've had to restore a backup of the z-stick's EEPROM to recover it.

[lost]

I get the same, every few months or so. Power down (not reboot), pull the z-stick, then put it back and start the system again. Generally, it's fine after that.
At worst - has happened a couple of times - something messes up the z-wave network data in the stick and I can't see any of my devices. Aeotec doesn't know the cause of that, suspects a bad device; that's happened twice now in the last few years and I've had to restore a backup of the z-stick's EEPROM to recover it.

OZW should be able to manage this & send a controller soft reset if no messages made it's way during some (configurable) time. IMO, this is some defense to recover a stuck controller.

Don't remember where this is setup, but in the past this was a very short delay, maybe 2mn only... and could lead to issues!

I remember I ran into such issues at the very beginning: I started my setup for heat level control driven by schedules.

Problem was I sometimes had hours between shedules, with a z-wave network not exchanging a single packet meantime.
=> This was triggering this defense delay, periodically soft-resetting the controller!

At the time, with only a few devices attached, OZW needed about 1mn to query all nodes. Then, after 2mn without communication, controller reset was triggered again and so on. Resulting in my z-wave network being stuck ~50% of time!

Extended this delay to max time between schedules, maybe 6 hours at the time: Issues gone!

Later, I added some devices that "talk" muck more: PIRs, temperature sensors, power counter... so I was able to reduce this delay to 10mn.

=> Take care of this, especially if you have mostly devices that are controller driven only (i.e. not sending data/measurements on their own, or setup to send them very scarcely to preserve battery): You may run in z-wave controller reset loops making z-wave mostly unusable (until full node querying, before next reset).

Log analysis should show the controller reset trigger if this is your issue.