Python plugin: Presence detection from wireless router

[Ray-Man]

python3-cryptography was already installed, but python3-paramiko wasn't. installed it, rebooted system, no luck.

Btw using Ubuntu 20.04.6 TLS

purged pip, paramiko, cryptography (random order)

installed pip -> sudo apt install pyhon3-pip
installed cryptography -> sudo apt install pyhon3-cryptography
installed paramiko -> sudo apt install pyhon3-paramiko
Rebooted system, no luck

WoW, somewhere did a sudo apt autoremove and reenabled the plugin: succes!

[Ray-Man]

Will do a rollback to a clean basic Domoticz system, make sure to update Ubuntu and try to reinstall iDetect with given suggestions. I'll post the proper install sequence ones I have a working iDetect again.

Thank for your help so fare!

Much appreciated!

[Ray-Man]

sudo apt update
sudo apt upgrade
cd /home/pi/domoticz/plugins
git clone https://github.com/d-EScape/Domoticz_iDetect.git iDetect
restart domoticz, added hardware iDetect and configured it: errors as expected
sudo apt install python3-pip
restart domoticz, errors as expected
sudo apt install python3-cryptography
restart domoticz, errors as expected
sudo apt install python3-paramiko
restart domoticz, Succeeded!

Proper sequence must be:

• sudo apt update

• sudo apt upgrade

• sudo apt install python3-pip

• sudo apt install python3-cryptography

• sudo apt install python3-paramiko

• cd /home/pi/domoticz/plugins

• git clone https://github.com/d-EScape/Domoticz_iDetect.git iDetect

• restart domoticz

• Configure iDetect

[EscApe]

<snip>
Proper sequence must be:
<snip>

Thanks/bedankt! That's very helpful. I have updated the installation instructions.

[DeKraker]

Netgear Orbi RBR350 running fw. V4.4.1.29_3.10.80

The response you where getting in your previous log is very different from the one I'm getting form the rbr50. Maybe it needs another approach. Since I don't have access to a RBR350 I cannot diagnose this any further. Can you experiment on your router?

Tha basic commands in Python3 to get the device list from a rbr50 is:

Code: Select all

import requests
http_session = requests.Session()
http_session.auth = ("admin","<<<<<your password >>>>>")
orbi_url = 'https://{}:{}/DEV_device_info.htm'.format("192.168.11.1", "443")
resp = http_session.get(orbi_url, verify=False, timeout=8)
print(resp.text)

Maybe it needs another url than /DEV_device_info.htm ?

If someone can give me access to a rbr350 (remotely over vpn should work) I would gladly take a look.

HTTP response seems to be similar to the login page of the router, using the url 'https://192.168.11.1:443/DEV_device_info.htm'.
However, when log on via a browser, I can succesfully perform a HTTP GET request using the URL .../DEV_device_info.htm in the browser session.
Callback then seems to be right (see below).

Code: Select all

{ "GUI_only_hijack_process": "3", "info": { "device_changed": 1, "device": [ { "ip": "192.168.11.4", "mac": "98:D8:63:95:63:BC", "contype": "wired", "attachtype": "0", "devtype": "21", "devtype_name": "Router", "model": "Unknown", "name": "Shanghai High-Flying Electronics", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.100", "mac": "F4:6D:04:D9:93:B1", "contype": "wired", "attachtype": "0", "devtype": "1", "devtype_name": "Computer (Generic)", "model": "Asus", "name": "MEDIA-PC", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.6", "mac": "2C:30:33:DE:86:08", "contype": "wired", "attachtype": "0", "devtype": "21", "devtype_name": "Router", "model": "R700", "name": "Netgear-R700", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.105", "mac": "90:0E:B3:01:C1:D6", "contype": "wired", "attachtype": "0", "devtype": "1", "devtype_name": "Computer (Generic)", "model": "Unknown", "name": "Shenzhen Amediatech Technology", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.104", "mac": "B8:27:EB:0C:CB:50", "contype": "wired", "attachtype": "0", "devtype": "19", "devtype_name": "Network (Generic)", "model": "Raspberry Pi", "name": "Raspberry Pi-Raspberry Pi", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.37", "mac": "C0:B5:D7:B7:02:0B", "contype": "5G", "attachtype": "1", "devtype": "2", "devtype_name": "Laptop", "model": "HP", "name": "LAPTOP-VLK7LDJ5", "accsta": "0", "conn_orbi_name": "Orbi Satellite-2", "conn_orbi_mac": "9C:C9:EB:1D:29:34", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.101", "mac": "B8:27:EB:B2:51:0A", "contype": "wired", "attachtype": "0", "devtype": "19", "devtype_name": "Network (Generic)", "model": "Raspberry Pi", "name": "Raspberry Pi-Raspberry Pi", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.102", "mac": "B8:27:EB:54:AE:84", "contype": "wired", "attachtype": "1", "devtype": "19", "devtype_name": "Network (Generic)", "model": "Raspberry Pi", "name": "Raspberry Pi-Raspberry Pi", "accsta": "0", "conn_orbi_name": "Orbi Satellite-1", "conn_orbi_mac": "9C:C9:EB:1D:29:FD", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.3", "mac": "9C:C9:EB:1D:29:FD", "contype": "wired", "attachtype": "2", "devtype": "0", "devtype_name": "Router", "model": "RBS350", "name": "Orbi Satellite-1", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "RBS350" }, { "ip": "192.168.11.2", "mac": "9C:C9:EB:1D:29:34", "contype": "wired", "attachtype": "2", "devtype": "0", "devtype_name": "Router", "model": "RBS350", "name": "Orbi Satellite-2", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "RBS350" }, { "ip": "192.168.11.74", "mac": "00:22:61:D6:AC:6C", "contype": "2.4G", "attachtype": "1", "devtype": "8", "devtype_name": "Smart Speaker", "model": "Frontier Silicon Ltd", "name": "SOPHIE", "accsta": "0", "conn_orbi_name": "Orbi Satellite-1", "conn_orbi_mac": "9C:C9:EB:1D:29:FD", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.9", "mac": "1C:53:F9:3C:91:0A", "contype": "5G", "attachtype": "0", "devtype": "6", "devtype_name": "Media Streamer", "model": "Unknown", "name": "Google", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.7", "mac": "74:F9:CA:90:B8:85", "contype": "5G", "attachtype": "0", "devtype": "7", "devtype_name": "Gaming", "model": "Nintendo", "name": "Nintendo", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.10", "mac": "3A:1B:F1:E2:CE:C2", "contype": "5G", "attachtype": "1", "devtype": "18", "devtype_name": "Tablet", "model": "iPad", "name": "Apple-iPad", "accsta": "0", "conn_orbi_name": "Orbi Satellite-2", "conn_orbi_mac": "9C:C9:EB:1D:29:34", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.23", "mac": "00:22:61:D9:00:E0", "contype": "2.4G", "attachtype": "1", "devtype": "8", "devtype_name": "Smart Speaker", "model": "Frontier Silicon Ltd", "name": "SARAH'S KAMER", "accsta": "0", "conn_orbi_name": "Orbi Satellite-2", "conn_orbi_mac": "9C:C9:EB:1D:29:34", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.8", "mac": "3A:5F:51:8D:4F:1C", "contype": "5G", "attachtype": "1", "devtype": "17", "devtype_name": "Smart Phone (Generic)", "model": "Galaxy A52", "name": "GALAXY-A52", "accsta": "0", "conn_orbi_name": "Orbi Satellite-2", "conn_orbi_mac": "9C:C9:EB:1D:29:34", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.13", "mac": "CA:7F:A8:D8:52:7D", "contype": "5G", "attachtype": "1", "devtype": "17", "devtype_name": "Smart Phone (Generic)", "model": "Galaxy A52", "name": "A52-VAN-SARAH", "accsta": "0", "conn_orbi_name": "Orbi Satellite-2", "conn_orbi_mac": "9C:C9:EB:1D:29:34", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" }, { "ip": "192.168.11.67", "mac": "1C:86:9A:A2:20:CA", "contype": "wired", "attachtype": "0", "devtype": "5", "devtype_name": "TV", "model": "Samsung", "name": "SAMSUNG", "accsta": "0", "conn_orbi_name": "Orbi Router", "conn_orbi_mac": "9C:C9:EB:1D:31:5C", "backhaul_sta": "Good", "ledstate": "0", "led_func": "0", "sync_btn": "0", "uprate": "0.00", "downrate": "0.00", "voice_orbi": "0", "voice_lwauserid": "", "ceiling_power": "not support", "module_name": "" } ] } }

[EscApe]

HTTP response seems to be similar to the login page of the router, using the url 'https://192.168.11.1:443/DEV_device_info.htm'.
However, when log on via a browser, I can succesfully perform a HTTP GET request using the URL .../DEV_device_info.htm in the browser session.
Callback then seems to be right (see below).

Do you mean the response in your log is similar to what you see when you open the login page in the browser? That is expected since it is the same page. I meant it looked different from the page on the orbi rbk50.

Anyway... I just updated GitHub with a completely different approach, using the soap interface of most Netgear routers instead of the admin user interface. This is completely experimental. I am not even able to test the complete plugin myself, because it only works on the local network and I don't have a local orbi. I have tried the commands remotely.

Please let me know if it works

To use it:

Code: Select all

sudo pip3 install pynetgear
update the iDetect plugin (git pull)
change the tracker config to 192.168.11.1#type=netgear-soap
restart the plugin (or Domoticz)

Edit: The host address is actually ignored for now (you still need it for the config to be valid). The pynetgear module will find the router on its own. Somehow specifying a host address doesn't work in my test setup.

[EscApe]

Netgear Orbi RBR350 running fw. V4.4.1.29_3.10.80

I have completed a new tracker type for Netgear Orbi and other Netgear routers supporting the soap api (most modern Netgear router).

Since the first (experimental) version I have added support for custom host and port addresses and usernames. There is also an option to autodetect the router bij specifying the tracker as: autodetect#type=netgear-soap
I was able to test the tracker with a remote Orbi router through a proxy, so I know it's working.

Steps te enable it are the same...

Code: Select all

sudo pip3 install pynetgear
update the iDetect plugin (git pull)
change the tracker config to 192.168.11.1#type=netgear-soap
restart the plugin (or Domoticz)

[hestia]

Hi @hestia,

Could it be the Domoticz host, network or router are so busy that the ssh session is getting a time-out? There are some pretty short hardcoded timeouts in ssh_tracker.py that you could try and change.

@Escape
sorry for the late answer, I wasn't notified...
I don't thing so
host:

Screenshot 2023-03-26 160827.png (31.31 KiB) Viewed 2340 times

network or router : at the same time, I have another domoticz running well with the plugin
And if I disable and enable the plugin, it's working fine!

Screenshot 2023-03-26 161452.png (82.35 KiB) Viewed 2340 times

[EscApe]

network or router : at the same time, I have another domoticz running well with the plugin
And if I disable and enable the plugin, it's working fine!

Sorry, I don’t know what is going on. If I understand correctly the plug-in is running fine on another domoticz setup? What is the difference between the two?

One other thing. The cpu on your domoticz system was not the only possible bottleneck I mentioned. The router(s) can also be to busy. Some routers only allow one connection at a time. In that case your second domoticz could be blocking requests from the first.

What make and model is your router?

[hestia]

make and model is your router?

3 ASUS RT-AC68U configured as WAP only (AP Mode)
less than 20 clients, 5 connected usually
when I do the top command on one of them I've got

Code: Select all

Mem: 61616K used, 194084K free, 860K shrd, 1072K buff, 8028K cached
CPU:  1.2% usr  3.5% sys  0.0% nic 94.5% idle  0.0% io  0.0% irq  0.5% sirq

For other bottlenecks:
Domoticz in on a 1Gb Ethernet

the plug-in is running fine on another domoticz setup?

Yes, I've just restart domoticz on the other server, and it's running fine.

And for the bottlenecks, while there are issues on one dz instance, the other one is running well, and when I restart the plugin, both are running well. So no specific issue with bottlenecks.

Could you give me the command in the asus so that I could check, there is no issue with the router?

What is the difference between the two

the old one, where the pluging is running fine

Code: Select all

RPI buster
Version: 2022.2 (build 15022)
Build Hash: af4316cb7
Compile Date: 2023-02-03 16:33:14
dzVents Version: 3.1.8
Python Version: 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
iDetect version 2.0

the new one where this is the issue

Code: Select all

RPI bullseye
Version: 2023.1 (build 15098)
Build Hash: 867c4e8fa
Compile Date: 2023-02-24 07:56:50
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Mar 12 2021, 04:06:34) [GCC 10.2.1 20210110]
iDetect version 2.1

EDIT: just upgrade to

Code: Select all

Version: 2023.1 (build 15177)
Build Hash: dfd9b9203
Compile Date: 2023-03-25 07:24:57

same issue

[hestia]

+
new install on the same RPI in another folder with an empty database
=> only 2 plugins
- iDetect version 2.2
- zigbeeForDomoticz

same issue

[EscApe]

There are some hardcoded timeouts in ssh_tracker.py. Can you try ans Increase them? Especially line 91:

Code: Select all

def getfromssh(self, tracker_cli, alltimeout=5, sshtimeout=3): 

These times are very short (on purpose), but maybe it is just too close for your environment.

[hestia]

done,

Screenshot 2023-03-26 203620.png (3.62 KiB) Viewed 2313 times

no change

Screenshot 2023-03-26 203435.png (24.03 KiB) Viewed 2313 times

I think there is no specific limit on my env, as it works after disable/enable the plugin

Screenshot 2023-03-26 203950.png (22.76 KiB) Viewed 2313 times

[EscApe]

Okay… something else to try:
You are polling three routers and all of them every 15 seconds. Can you change the poll-interval to 30 seconds. If that doesn’t help change it to 1 minute. Just to diagnose this issue.

[hestia]

directly with 1 router and 60 secondes
same error, but 1 min after!

Code: Select all

2023-03-26 21:16:09.211 Status: Domoticz V2023.1 (build 15177) (c)2012-2023 GizMoCuz
2023-03-26 21:16:09.213 Status: Build Hash: dfd9b9203, Date: 2023-03-25 07:24:57
2023-03-26 21:16:09.214 Status: Startup Path: /home/pihestia/domoticz/
2023-03-26 21:16:09.328 Status: PluginSystem: Started, Python version '3.9.2', 3 plugin definitions loaded.
2023-03-26 21:16:09.340 Status: WebServer(HTTP) started on address: :: with port 8080
2023-03-26 21:16:09.373 Status: WebServer(SSL) started on address: :: with port 443
2023-03-26 21:16:11.439 ePresence: Worker thread started.
2023-03-26 21:16:11.439 zigbeeE: Worker thread started.
2023-03-26 21:16:11.439 Status: ePresence: Started.
2023-03-26 21:16:11.439 Status: ePresence: Entering work loop.
2023-03-26 21:16:11.439 Status: zigbeeE: Entering work loop.
2023-03-26 21:16:11.440 Status: zigbeeE: Started.
2023-03-26 21:16:11.440 Status: NotificationSystem: thread started...
2023-03-26 21:16:11.441 Status: EventSystem: reset all events...
...
2023-03-26 21:16:11.734 Status: ePresence: Initialized version 2.2, author 'ESCape'
2023-03-26 21:16:11.763 Status: PluginSystem: 2 plugins started.
2023-03-26 21:16:12.859 Status: Python EventSystem: Initializing event module.
2023-03-26 21:16:12.860 Status: EventSystem: Started
2023-03-26 21:16:12.860 Status: EventSystem: Queue thread started...
2023-03-26 21:16:14.375 Status: ePresence: Starting tracker:wapsalon, user:admin, class:ssh_autodetect and poll interval:60
...
2023-03-26 21:16:33.685 Status: zigbeeE: Plugin with Zigpy, Coordinator Unknown Silicon Labs firmware UNK - 6.10.3.0 build 297 correctly initialized
2023-03-26 21:17:19.399 ePresence: wapsalon Could not be polled
2023-03-26 21:17:19.399 Status: ePresence: wapsalon ====> SSH resetting connection
2023-03-26 21:17:19.396 Error: ePresence: wapsalon ====> SSH failed with exception: Timeout opening channel.

[EscApe]

Hi @hestia,

From the different logs you have shared I can't really see a pattern. Do I summarize correctly that:
- sometimes it works for two out of three routers but fails later on?
- it is not always the same routers that do work and that fail?
- after a restart of the plugin It works again?
- in you last log it doesn't work at all?
- on a different domoticz installation it works without any problems?

To make future diagnostics a bit easier please share complete debug logs from the moment the plugin starts until after the first poll (attempt) and alle messages around the failure. I don't think I need other logs right now (if the above is correct), but when testing in the future. Otherwise it is a bit of a puzzle ... and the issue is puzzling enough as it is

The problem still might be some weird congestion or connecting failure caused by a network driver, different python version or modules (like ssh and paramiko), other tasks that block the router from responding in time etc etc etc....

I no longer use a Asus (or other ssh based) tracker myself, so it will take me some time to build a test setup using one of my old Asus routers. In the meantime you could try a fresh install using the same python version and modules (version).

A ran the ssh tracker with two Asus routers without any problems until little over a year ago. You could also try running it on only one or two routers for a little while. Just for diagnostics.

[hestia]

Hi @EscApe
some answers ("yes" = always right, if not "no")
- sometimes it works for two out of three routers but fails later on?
no (it fails always after a restart of domoticz)

- it is not always the same routers that do work and that fail?
no (I don't think there is a difference between routers, now there is only one router, I could test them separatly, but I think this issue is not relative to the router)

- after a restart of the plugin It works again?
yes

- in you last log it doesn't work at all?
no (it was working after a restart of the plugin)

- on a different domoticz installation it works without any problems?
yes (on an old installation with buster)

[EscApe]

Another thing just sprung to mind.... By 'restart of domoticz' do you mean the entire domoticz host or just the domoticz application? If you meant the domoticz host then it is very likely that Domoticz is starting before the network interface is (completely) up.

You can solve this by modifying the startup order, which can be complicated... or just put a delay in the Domoticz startup script.

[hestia]

+
I've given log in the post with the debug on
viewtopic.php?p=301063#p301063
If you need more log, could you give me what to do please.

A ran the ssh tracker with two Asus routers without any problems until little over a year ago. You could also try running it on only one or two routers for a little while. Just for diagnostics.

On the new env. (w the issue) there only 1 router

Perhaps, I could use only one router always, because in the GUI they all have the same client list

[hestia]

Another thing just sprung to mind.... By 'restart of domoticz' do you mean the entire domoticz host or just the domoticz application? If you meant the domoticz host then it is very likely that Domoticz is starting before the network interface is (completely) up.

You can solve this by modifying the startup order, which can be complicated... or just put a delay in the Domoticz startup script.

Code: Select all

sudo service domoticz.sh restart

or

Code: Select all

sudo service domoticz.sh stop
+
sudo service domoticz.sh start