Easiest way to migrate Openzwave to ZWave-JS-UI

[Dave47]

But that is your domoticz container, that is under a different name (I hope).

Different name?
I have another thread:
viewtopic.php?t=40022

Can you help me please. I'm struggeling with this.

[MikeyMan]

Did anyone ever manage to migrate and do some form of 'replace device' afterwards?

[MikeyMan]

Had some time to spare, and some issues with a zwave node, so gave it another try... Cant get replace to work at all, since i can't have both plugins enabled at the same time. Also, still some devices unsupported.

Thirdly, my smoke alarm got added by MQTT Auto discovery. However, it's not updated rightly.
MQTT shows a change at 10:30, when i pushed the test button:

2023-04-04 10_41_31-MQTT Explorer.png (93.43 KiB) Viewed 6576 times

However, it isn't updating in domoticz:

2023-04-04 10_41_58-Thuis en nog 8 andere pagina's - Profiel 1 - Microsoft​ Edge.png (66.92 KiB) Viewed 6576 times

Any thoughts? Domoticz log is quite empty

[MikeyMan]

With some persistance, managed to get all devices in.

However; it seems that the move loses quite some functionality. In the OpenZwave setup; the devices seem to have been grouped in the hardware. When domoticz crashed, the buttons still worked. And i also had the possibility to use a 8-button switch to dim the lights.

however in the new setup, it seems fully reliant on the groups/scenes. And doesn't allow for the push to dim functionality (Keep on pushed ==> Brighter, keep off pushed ==> dimming)

Am i missing something?

[MikeyMan]

Managed to get the link set up as well through the groups in ZWaveJS.

Only thing i'm now noticing is that Domoticz seems to send double MQTT messages.

[thomasbaetge]

Did you set the 'prevent loop' setting in the MQTT Gateway settings in Domoticz to 'true'?
If it is on 'false' that may cause the issue.

[MikeyMan]

Did you set the 'prevent loop' setting in the MQTT Gateway settings in Domoticz to 'true'?
If it is on 'false' that may cause the issue.

Yup it's set to True.
The issue is independent of any actions i'm doing in Node Red/MQTT.

It just always sends two messages.

This is one "switch on" command:

2023-04-13 14_24_24-Node-RED _ 10.0.0.4 en nog 8 andere pagina's - Profiel 1 - Microsoft​ Edge.png (33.4 KiB) Viewed 6516 times

[thomasbaetge]

That is strange.
I am using a lot of MQTT and I never had domoticz sending duplicate messages.
Do you have 2 Gateway configured in Domoticz?
(really just guessing here)

[MikeyMan]

That is strange.
I am using a lot of MQTT and I never had domoticz sending duplicate messages.
Do you have 2 Gateway configured in Domoticz?
(really just guessing here)

Yes, i have multiple, for multiple devices. But that doesn't seem to be the issue. When i turn off the default it's 0 messages.
When i turn it on i have two.

The only other one sending to domoticz/out is the shelly mqtt plugin. Turning that off or on doesnt make a difference.

[MikeyMan]

That is strange.
I am using a lot of MQTT and I never had domoticz sending duplicate messages.
Do you have 2 Gateway configured in Domoticz?
(really just guessing here)

Yes, i have multiple, for multiple devices. But that doesn't seem to be the issue. When i turn off the default it's 0 messages.
When i turn it on i have two.

The only other one sending to domoticz/out is the shelly mqtt plugin. Turning that off or on doesnt make a difference.

Do you also use the combination with Zwave-JS?

[thomasbaetge]

Not yet.
I have everything prepared and installed for it, but I will need some time for migration and replacing old devices with new.
I have no room for errors, because I have crucial infrastructure like heating control and doorlock running on z-wave.

[MikeyMan]

Not yet.
I have everything prepared and installed for it, but I will need some time for migration and replacing old devices with new.
I have no room for errors, because I have crucial infrastructure like heating control and doorlock running on z-wave.

With openZwave more often than not i didnt have any MQTT out messages

[thomasbaetge]

That's why I let others test

I will migrate 'last minute'...for now, OZW runs just fine here.

[sincze]

I started the migration today.

Noticed the following.

1. My (A) meters from OpenZwave *CM113, Electrisave) are now created in autodiscovery as: Type (General) Current.
Hence it seems I am unable to replace the old with the new.
Can this be fixed easily ??

2. Second thing I noticed is while replacing a lot of devices the webserver becomes unusable.

Code: Select all

Error: WebServer:8080 thread seems to have ended unexpectedly (last update 62.000000 seconds ago)

3. Retain (ON)
While leaving retain left to ON the migrated new devices are automatically re-created (in domoticz) so I temporarily disabled this in Zwave-JS-UI.
Need to figure out how to remove the retained messages from MQTT and then re-enable this again.

4. Replace Switch.
While replacing a switch on the switch tab after pressing ok it jumps back to the DASHBOARD in stead of staying on the SWITCH tab.

5. NEO Sensor negative value.
I unfortunately have some NEO Powerplugs known for their negative values.
https://github.com/zwave-js/node-zwave-js/issues/2692

Code: Select all

This device is buggy and sends large negative Meter Reports from time to time.

Domoticz used to IGNORE these messages,https://github.com/domoticz/domoticz/issues/4277,
But it seems with the MQTT solution these negative numbers are back in town.

Suggestions are welcome.

[d3smo]

Thanks! That guide looks very promising! After everything (hopefully) works, I'll write a full guide here.

Today I successfully finished my upgrade. Pi is now running Domoticz 2022.2 with ZWave-JS-UI (via SNAP) and Bullseye OS.
Since there is no complete guide for this upgrade process, I decided to make my own.
The basics steps I made were:
1. Backup Z-Wave data
2. Install Mosquitto
3. Install Snap
4. Install Z-Wave JS UI
5. Disable OpenZwave gateway
6. Configure Z-Wave JS UI
7. Install MQTT Auto Discover Client gateway in Domoticz
8. Replace old Z-Wave devices
9. Delete old Z-wave controller
10. Upgrade Domoticz to 2022.2
11. Upgrade Pi from Buster to Bullseye

According viewtopic.php?p=296307#p296307 the upgrade to bullseye should be done after the Domoticz upgrade to 2022.2, but in step 11 I ran into some minor conflicts which made me have to redo two of the installation steps of Mosquitto (2.5) and Snap (3.3b). Maybe it is better to upgrade your Pi to Bullseye as step 1 instead of 11.

In detail, there were many, many more steps:

Code: Select all

1. Backup Z-Wave data from Domoticz
		Most imporant data is:
		1. Security Network Key
		2. List of NodeID's and node names 
		3. The USB address of your Z-Wave usb stick.

2. Install Mosquitto according https://randomnerdtutorials.com/how-to-install-mosquitto-broker-on-raspberry-pi/ (en https://www.domoticz.com/wiki/MQTT)
	
		1. Update and upgrade system of Pi:
		sudo apt update && sudo apt upgrade

		2. Install Mosquitto:
		sudo apt install -y mosquitto mosquitto-clients
		
		3. Make Mosquitto auto start when the Raspberry Pi boots
		sudo systemctl enable mosquitto.service
		
		4. Start-up and version check:
		mosquitto -v  (Gives version 1.5.7, version 2.xx will be installed with Bullseye)
	
		5. Enable remote access without authentication:
		sudo nano /etc/mosquitto/mosquitto.conf
		Add these lines at the end:
		listener 1883
		allow_anonymous true
		Use CTRL-X to exit and save.
		
		6. Restart Mosquitto:
		sudo systemctl restart mosquitto
		
		7. Test status of Mosquitto:
		sudo systemctl status mosquitto
		
		8. Check version:
		mosquitto -h | grep version    (again version 1.5.7)
		
3.  Install Snap according https://snapcraft.io/docs/installing-snap-on-raspbian en https://community.home-assistant.io/t/install-and-setup-zwave-js-ui-with-snap-on-a-raspberry-pi/486484
		1. Update system:
		sudo apt update
		
		2. Install snap
		sudo apt install snapd
		
		3. Work around a bug which seems to exist only on Raspberry Pi OS 32-bit system: https://stackoverflow.com/questions/42443273/raspberry-pi-libarmmem-so-cannot-open-shared-object-file-error/50958615#50958615 en https://github.com/zwave-js/zwave-js-ui/discussions/1216
		3a
		sudo nano /etc/ld.so.preload
		Comment out this line:
		(Change /usr/lib/arm-linux-gnueabihf/libarmmem.so to #/usr/lib/arm-linux-gnueabihf/libarmmem.so)
		3b
		sudo visudo
		add :/snap/bin to the end of the secure_path, BUT before the closing quote ". Note the :-separator!
				
		4. Reboot Pi
		sudo reboot
		
		5. After this, install the core snap in order to get the latest snapd.
		sudo snap install --channel=stable core
		
		6. Enable hotplug support and restart:
		sudo snap set system experimental.hotplug=true
		sudo systemctl restart snapd
		
		7. To test your system, install the hello-world snap and make sure it runs correctly:
		sudo snap install hello-world
		hello-world
		
4.  Install Z-Wave JS UI according https://www.domoticz.com/wiki/Zwave-JS-UI en https://zwave-js.github.io/zwave-js-ui/#/getting-started/other-methods and https://community.home-assistant.io/t/install-and-setup-zwave-js-ui-with-snap-on-a-raspberry-pi/486484
		1. Install zwave-js-ui
		sudo snap install --channel=stable zwave-js-ui
		
		2. Give the package access to use USB devices and observe hardware:
		sudo snap connect zwave-js-ui:raw-usb
		
		3. Needed for the program to list available devices in the UI:
		sudo snap connect zwave-js-ui:hardware-observe
		
		4. Open help to see some info about commands:
		zwave-js-ui.help	
		
		5. To access the UI from outside your Pi:
		sudo snap set zwave-js-ui server.host=0.0.0.0
		
		6. Then we need to connect the serial-port to the plug zwave-js-ui. First list available serial slots:
		sudo snap interface serial-port
		
		7. Then use this command to connect the correct serial slot to zwave-js-ui (use correct core: from list you got in previous step):
		sudo snap connect zwave-js-ui:serial-port core:aeotecz-stickgen5zw0
		
		8.Turn service on:
		sudo zwave-js-ui.enable

5.	Disable (not delete!) OpenZwave gateway (USB stick) in Domoticz (menu Setup - hardware)
		Idea is that the old devices are not deleted.
		Sadly this did not work. All my old devices disappeared. Maybe I should have upgraded Domoticz from 2022.1 to 2022.2 first. See also https://www.domoticz.com/forum/viewtopic.php?p=297183&sid=8601aab4dc1990f1d0199e8caced281b#p297183
	
6.	Configure Z-Wave JS UI (from https://www.domoticz.com/wiki/Zwave-JS-UI)

		1. Browse to http://192.168.178.41:8091 
		
		2. Menu settings → MQTT: 
		Retain on
		
		3. Menu settings → Gateway:
		Use nodes names on
		Ignore location on
		Include Node info on
		
		4. Menu settings → Home Assistant:
		MQTT Discovery on
		Retained discovery on (not mentioned in text, but that's according picture)
		
		5. Menu settings → Z-Wave
		Get the key from Domoticz (mine was just "0102030405060708090A0B0C0D0E0F10") and enter as S2 Unauthenticated. 
		Enter the right Serial Port address. In my case: /dev/ttyUSB-ZwaveStick
		I also had to enter something at S0 legacy to get rid of an error message regarding the security status of a Qubino ZMNHDD. I just pressed the button to generate a random key. Result was 643294277C1E50FAEA0C497F116F9897.
		
		6. Menu settings 
		Save settings (Press "save" at the bottom)

		7. Menu Control Panel:
		Give all nodes a useful name based on their Node ID and the info you saved in step 1

7. 	Install MQTT Auto Discover Client gateway in Domoticz (see https://www.domoticz.com/wiki/MQTT#Add_hardware_.22MQTT_Auto_Discovery_Client_Gateway.22)
		
		1. Domoticz → Menu Setup → Hardware 
		Name = MQTT Auto Discovery
		Type = MQTT Auto Discover Client Gateway with LAN interface
		Data Timeout: Disabled
		Remote Address: localhost
		Username: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account name
		Password: Leave empty when mqtt broker has no user accounts setup else give mqtt broker account password
		Auto Discovery Prefix: homeassistant
		
		2. Make sure it is enabled and press "add".

		3. New devices are created in Domoticz by MQTT AutoDiscover
		(After renaming the devices in Z-Wave JS UI I should have gotten nice names, but sadly that didn't work...)
		
		4. Domoticz → Menu Setup → Devices
		Manually rename all new devices to something usefull

8.	Replace old Z-Wave devices with new ones
		If you want to keep the history logging of the old devices then you need to replace the old device with the new one if identical devices are created. Use the replace function in Edit device, see wiki https://www.domoticz.com/wiki/Managing_Devices#Replace_device Devices that were replaced will get the old name and old IDX, no need to change scripts.
	
9. 	Delete old Z-wave controller in Domoticz.
		1. Domoticz → Menu Setup → Hardware
		Select old Z-wave controller (Aeotec USB stick) and press Delete.
		
		2. Reboot pi
		sudo reboot 

10. Upgrade Domoticz to 2022.2 according https://www.domoticz.com/forum/viewtopic.php?t=39291
		1. Backup version 2022.1
		sudo service domoticz stop
		sudo cp -R /home/pi/domoticz /home/pi/domoticz_2022.1  (may take while...)
		sudo service domoticz start

		2. Make sure you're on the Stable Release Channel
		Menu Setup → Settings → Software Updates → Release Channel → Stable
		
		3. Update to 2022.2 (called #14606 in dialogue box...)
		Menu Setup → Check for updates → OK
		
11. Upgrade Pi to Bullseye according https://pimylifeup.com/upgrade-raspberry-pi-os-bullseye/ and https://linuxhint.com/upgrade-raspberry-pi-os-buster-to-bullseye/
		1. Backup your Pi.
		
		1. Check your current version:
		lsb_release -a
		
		2. Update Raspberry Pi Repositories List
		sudo apt update
		
		3. Upgrade Packages on Raspberry Pi
		sudo apt dist-upgrade -y
		
		4. Upgrade Firmware
		sudo rpi-update
		sudo reboot
		
		5. Edit Raspberry Pi Source Lists
		In /etc/apt/sources.list replace all instances of "buster" by "bullseye".
		sudo nano /etc/apt/sources.list or use WinSCP
		Do the same for any other sources lists in the /etc/apt/sources.list.d directory:
		sudo nano /etc/apt/sources.list.d/raspi.list
		sudo nano /etc/apt/sources.list.d/vscode.list
		
		6. Update package lists
		sudo apt update
		
		7. Install missing packages:
		sudo apt install -y libgcc-8-dev gcc-8-base
		(enter yes if asked if services have to be automatically restarted)
		
		8. Upgrade to Bullseye
		sudo apt full-upgrade
		
			Ran into Error:
			E: Failed to fetch http://mirror.serverius.net/raspbian/raspbian/pool/main/p/protobuf-c/libprotobuf-c1_1.3.3-1+b2_armh              f.deb  Undetermined Error [IP: 5.255.95.70 80]
			E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
			Decided to rerun command, second time it worked OK.
			
			Installer reported change conflict in 3 files. Backed up all three and let installler use new versions.
			/etc/default/useradd
			/etc/sudoers
			/etc/mosquitto/mosquitto.conf

			Decided not to do anything about useradd
			Fixed /etc/sudoers with command sudo visudo (see snap configuration above)
			Fixed conflict in /etc/mosquitto/mosquitto.conf by adding the two missing lines (see mosquitto configuration above)
		
		9. Fix error in /etc/systemd/system/dhcpcd.service.d/wait.conf
		Check if file exists:
		sudo stat /etc/systemd/system/dhcpcd.service.d/wait.conf
		Edit the file
		sudo nano /etc/systemd/system/dhcpcd.service.d/wait.conf
		Replace
		ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -w
		by
		ExecStart=/usr/sbin/dhcpcd -q -w
		Exit and save.
		
		10. Modify Boot config
		sudo nano /boot/config.txt
		In section [pi4] comment out dtoverlay=vc4-fkms-v3d (put a # in front of the line)
		In section [all] add the line dtoverlay=vc4-kms-v3d
		Exit and save.
		
		11. Reboot Pi.
		sudo reboot
		
		12. Check version to see if upgrade was successful:
		lsb_release -a
		
		13. Cleanup (if everything works)
		sudo apt autoclean

		14. Reinstall dev version of Python plugin to get plugins working
		sudo apt install python3-dev

		15. Restart Domoticz
		sudo service domoticz restart

I hope this guide is useful for anyone who still has to do the same upgrade. If you like it, please let me know.

I follow your step by step tutorial to migrate from OZW to ZWave-JS-UI, everything goes well until the configuration of serial port. When i launch the command

Code: Select all

sudo snap interface serial-port

the reply is the following

Code: Select all

name:    serial-port
summary: allows accessing a specific serial port
plugs:
  - zwave-js-ui
slots:
  - core:rfxtrx433
  - core

It seems my Aeon Labs ZW0990 Z-Stick Gen5 EU is not detected ? I'm able to launch ZWave-JS-UI with sudo zwave-js-ui.enable and browse the service on port 8091. My RPi is already running Bullseye, and i'm using OZW on Domoticz 2023.1 for now and everything is working, but as OZW is depreciate i want to anticipate and migrate to ZWave-JS-UI, so if anyone as face the same issue of not seeing the ZStick maybe you can help me.

[thomasbaetge]

Your stick can connect to one instance at a time only.
So while OZW is running, zwJS can not access it (and vice versa)

[d3smo]

Your stick can connect to one instance at a time only.
So while OZW is running, zwJS can not access it (and vice versa)

I've try disable OZW in Domoticz devices and stop domoticz service from terminal, the result is the same stick is not visible from command

Code: Select all

sudo snap interface serial-port

[thomasbaetge]

it may require a reboot.
but frankly I don't understand why you're using snap in the first place.
Why not Docker?

[d3smo]

Same behavior after a reboot, and why Snap because the tutorial seems easier to me, but i'll try with docker. Do you have a step by step installation guide of ZWave-JS-UI that include the docker installation and setup ? I found a lot of thing but nothing explaining the complete process step by step like the tutorial i link in my first post did for snap method

[MikeyMan]

Yup, same reason I went for snap. Docker seems to be quite heavy on config and resources on a pi.

Although I didnt have any issues with that step of the snap manual.

Did you try the home assistant manual for this part of the install?