Python plugin: Presence detection from wireless router

[sapnho]

Thanks escApe, your release frequency is most impressive!

Seems to work fine, no errors so far in the (now very exhaustive) log.

[wizzard72]

@wizzard72
The override button could be disabled in the plugin settings. What is your current 'override mode'?

It was "Do not allow override". Changed it. Thanks for pointing out.

[wizzard72]

Now new features or fixes, but i have tweaked the logging a bit

**version 0.6.6**
- Improved: Debug logging for the ssh session now hides the ssh password.
- Improved: Presence changes are now logged (without needing debug mode), so you can see something is going on.
- Improved: Some more log tweaking (removed some redundancies and changed log levels for some messages)

Looking good. No more errors in the log. Keep up the good work!

[sapnho]

What is actually the best way to transfer the iDetect settings to another Domoticz installation? Or is it saved in the Domoticz settings when you do a backup?

[EscApe]

All settings are stored in the database and should be included in your Domoticz backup. Just put the plugin.py and .zip files on the new installation.

[sapnho]

Ok, perfect, thanks.

I am having trouble with the restore function in Domoticz (4.9999) at the moment but I guess that will be resolved soon.

[sapnho]

Two new error message at start up after a soft reboot. No problem otherwise.

2018-09-12 08:26:40.238 Error: iDetect iPhones hardware (5) thread seems to have ended unexpectedly
2018-09-12 08:26:40.238 Error: iDetect RPi Fnatenrahmen hardware (6) thread seems to have ended unexpectedly

[EscApe]

@saphno
Can't really tell something by just these messages, but my guess is that either domoticz or its python plugin framework got stuck/overloaded. That could be explained if you poll many routers and your network, router of domoticz slows down. If timeouts start occurring every poll will take max 2 seconds * number of routers. So 6 seconds with your three routers. If you poll every 10 seconds and maybe have some other python plugins installed things might get ugly. It should recover though, and hopefully not happen frequently.

All this is pure speculation on my part, but for now i see no reason to be alarmed.

[sapnho]

Ok, good, thanks!

[EscApe]

version 0.7.0 is available on GitHub - some major changes under the hood

Added: option to preconfigure the router commands (skipping auto detection) and a way to easily find out what to ‘preconfigure’. See github for instructions.
Fixed: Router capability detection (compatibility with some firmwares that limit the ssh argument size).

Important: Please TEST the new version (watch te log) and leave a message on the forum if it fails where the previous version succeeded. There are changes in this version that i simply cannot test, because i don’t own every brand and model of router.

This release introduces a completely rewritten function to detect the router capabilities. Why fix something that doesn’t seem broken? Well, it turns out the original router (shell) part of the script was to long for some routers. In that case the ssh session would fail because the script was longer than the maximum length allowed for an ssh argument. So i used a shorter notation and broke it up into de detection of available commands and the detection of the interfaces to query. The new approach is also a bit easier to maintain (add or modify chipset support). Using separate sessions for capabilities detection introduces some additional overhead, but only when the plugin/domoticz starts. It has zero impact on the poll times, since it generates the same poll commands as before.

An indication of the changed startup performance: The old detection function would take one session, which took 0.35 seconds on my Asus AC86. It will now take 2 sessions = 0.7 seconds. A router with two chipsets, like the AC87 will take 1+2 sessions = around 1 seconds. A mesh setup with three AC87’s will take 3 + 3x2 sessions, which translates into a little over 3 seconds. With the old function that same mesh setup would take around one second. If timeouts occur the wait time will quickly add up, so i lowered them to two seconds (per ssh session). That should not be a problem if your network, router and Domoticz host are in good shape, but please let me know if it proofs a little to close to the edge.

Thanks to mvzut for testing and for investigating why the capabilities detection was failing on his router!

[QSKONE]

Hi , it is my router Xiaomi Router 3 poor device for this?

Code: Select all

Error: (iDetect) Failed 30 times to get capabilities for 192.168.31.1. Retry interval automatically reduced for this router.

[mvzut]

Is ssh enabled on your router and do you know the ssh login name and password?

It's difficult to tell what the problem is from this log message only, if you turn on debugging we can probably give more relevant feedback.

[QSKONE]

Thanks debugging helps me. Plugin works but phones are not connected.
Full log

Code: Select all

(iDetect) Started.
(iDetect) Entering work loop.
(iDetect) Initialized version 0.7.0, author 'ESCape'
(iDetect) Using generic brctl command on router 192.168.31.1. Will respond slower and on some routers a little less reliable to absence
(iDetect) Awaiting confirmation on absence of 2 devices
(iDetect) Connection restored for 192.168.31.1
(iDetect) SSH command on 192.168.31.1 took 776 milliseconds.
(iDetect) Found these devices connected:[]
(iDetect) Considered absent: Phone2
(iDetect) Considered absent: Phone1
(iDetect) Polling presense data from 192.168.31.1
(iDetect) Using password instead of ssh public key authentication for 192.168.31.1 (less secure!)
(iDetect) Fetching data from 192.168.31.1 using: sshpass -p **secret** ssh -o StrictHostKeyChecking=no -o ConnectTimeout=2 [email protected] export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH;brctl showmacs br0 | grep '..:..:..:..:..:..' | awk '{print $ 2}';exit
(iDetect) ssh command on 192.168.31.1 returned:b''
(iDetect) SSH command on 192.168.31.1 took 780 milliseconds.
(iDetect) Found these devices connected:[]
(iDetect) Considered absent: Phone2
(iDetect) Considered absent: Phone1

[EscApe]

I have no experience with the Xiaomi router. It seems to be based on a Ralink MT7620A chipset. Is anyone else on this forum using the plugin with a chipset like that? does it return anything on the brctl command?

Seems weird that it isn't returning anything on the xiaomi. Brctl should even return wired devices.

@QSKONE
Do you know your way around the command line? Could you check if the following commands return anything if you run them on the router yourself?

Code: Select all

brctl showmacs br0

(try different names if br0 is not the right name for the xiaomi. You can find the names using: brctl show)

Code: Select all

arp -a

[QSKONE]

Brctl cannot work

Code: Select all

root@XiaoQiang:~# brctl showmacs br0
brctl: invalid argument 'showmacs' to 'brctl'
root@XiaoQiang:~#  brcl show
-ash: brcl: not found

Comand arp -a shows my connected devices with ip and mac adress.

Code: Select all

? (192.168.31.11) at mac adress [ether]  on br-lan
  ...

[EscApe]

@QSKONE

you could try configuring the router in the plugin settings as: 192.168.31.1=arp
This will force the plugin to use arp on the router. I do not see any Mac addresses in the output you shared. If that was the real output, then that is not going to work either.

Brctl is behaving differtly on your router, so that’s not an option. (The command is available, but you left out the letter ‘t’ on you second attempt)

[QSKONE]

@EscApe thanks you very much for help =arp works well on Xiaomi Router 3

[mikeoo]

Trying to get your plugin to work with an Unify EdgeRouter X with SSH enable and 2 Unifi AP Lite and 1 Unify Pro

But no success for the moment.

I can SSH to the router, but for SSH i am using a different port number (9325), so don't know how the configure this in the plugin and if this is a problem. I tried only the router -ip adres and routerip:portnumber both not wokring.

Don't use a SSH key for now, but put my username and password which i use for putty to SSH to my router.
I can SSH from the raspberry to my router.

Hope you have some tips for me.

Debug Log

Code: Select all

 2018-09-25 16:34:40.726 Status: (iDetect) Started.
2018-09-25 16:34:40.939 Status: (iDetect) Entering work loop.
2018-09-25 16:34:40.940 Status: (iDetect) Initialized version 0.7.0, author 'ESCape'
2018-09-25 16:34:41.018 (iDetect) SSH subprocess for 192.168.0.1 failed with error (255):b''
2018-09-25 16:34:41.018 (iDetect) SSH command on 192.168.0.1 took 30 milliseconds.
2018-09-25 16:34:41.018 (iDetect) Could not retreive available commands on 192.168.0.1
2018-09-25 16:34:41.018 (iDetect) Router initialized as:{'192.168.0.1': {'user': '<user account for SSH>', 'errorcount': 1, 'cmd': '', 'prospone': datetime.datetime(2018, 9, 25, 16, 34, 53, 18616), 'initialized': False}}
2018-09-25 16:34:41.018 (iDetect) iDetect - Phone1 is stil in use
2018-09-25 16:34:41.018 (iDetect) iDetect - Phone2 is stil in use
2018-09-25 16:34:41.018 (iDetect) iDetect - Phone3 is stil in use
2018-09-25 16:34:41.019 (iDetect) Devicenames and their Domoticz (subdevice) index: {'Hans': 2, 'Nellie': 3, 'Michel': 4}
2018-09-25 16:34:41.019 (iDetect) MAC addresses to monitor: {'Phone1': {'ignore': False, 'mac': 'D0:B1:28:37:4E:23', 'lastseen': None}, 'Phone2': {'ignore': False, 'mac': '44:C3:46:48:E8:2B', 'lastseen': None}, 'Phone3': {'ignore': False, 'mac': '64:A2:F9:28:5A:A1', 'lastseen': None}}
2018-09-25 16:34:41.019 (iDetect) Plugin initialization done.
2018-09-25 16:34:50.988 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:02.030716
2018-09-25 16:34:50.988 (iDetect) Did not retreive WLAN information from any router
2018-09-25 16:35:00.954 (iDetect) 192.168.0.1 was not properly initialized. Retrying to get router capabilities (and skipping this poll round).
2018-09-25 16:35:00.955 (iDetect) Using password instead of ssh public key authentication for 192.168.0.1 (less secure!)
2018-09-25 16:35:00.955 (iDetect) Fetching data from 192.168.0.1 using: sshpass -p **secret** ssh -o StrictHostKeyChecking=no -o ConnectTimeout=2 [email protected]
2018-09-25 16:35:00.955 #!/bin/sh
2018-09-25 16:35:00.955 export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH
2018-09-25 16:35:00.955 type wl > /dev/null 2>&1 && printf "~wl"
2018-09-25 16:35:00.955 type iwinfo > /dev/null 2>&1 && printf "~iwinfo"
2018-09-25 16:35:00.955 type wlanconfig > /dev/null 2>&1 && printf "~wlanconfig"
2018-09-25 16:35:00.955 type qcsapi_sockrpc > /dev/null 2>&1 && printf "~qcsapi_sockrpc"
2018-09-25 16:35:00.955 type brctl > /dev/null 2>&1 && printf "~brctl"
2018-09-25 16:35:00.955 type arp > /dev/null 2>&1 && printf "~arp"
2018-09-25 16:35:00.955 [ -f /proc/net/arp ] && printf "~procarp"
2018-09-25 16:35:00.955 exit
2018-09-25 16:35:00.955
2018-09-25 16:35:01.045 (iDetect) SSH subprocess for 192.168.0.1 failed with error (255):b''
2018-09-25 16:35:01.045 (iDetect) SSH command on 192.168.0.1 took 90 milliseconds.
2018-09-25 16:35:01.045 (iDetect) Could not retreive available commands on 192.168.0.1
2018-09-25 16:35:01.046 (iDetect) Routerinfo:{'user': '<user account for SSH>', 'errorcount': 2, 'cmd': '', 'prospone': datetime.datetime(2018, 9, 25, 16, 36, 1, 46001), 'initialized': False}
2018-09-25 16:35:01.046 (iDetect) Did not retreive WLAN information from any router 
 2018-09-25 16:35:10.964 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:50.081134
2018-09-25 16:35:10.965 (iDetect) Did not retreive WLAN information from any router
2018-09-25 16:35:20.984 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:40.061711
2018-09-25 16:35:20.984 (iDetect) Did not retreive WLAN information from any router
2018-09-25 16:35:30.957 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:30.088598
2018-09-25 16:35:30.957 (iDetect) Did not retreive WLAN information from any router 
 

[EscApe]

@mikeoo

That's because 0.7.0 didn't support custom port numbers. The 0.7.1 iteration i just uploaded to GitHub does
You can now specify router:portnumber.

[mikeoo]

@mikeoo

That's because 0.7.0 didn't support custom port numbers. The 0.7.1 iteration i just uploaded to GitHub does
You can now specify router:portnumber.

Thanks for the quick update, now it is working
Needs ARP because WL is missing on the router.

Only problem i see in the log is the when my Phone goes offline is works fine. As soon as i turn on Wifi a few seconds later i get first a few times an error (iDetect) Did not retreive WLAN information from any router
After that is can connect again and turn the switch back to ON.

Sometimes it works fine without errors and sometimes i get this error.
Could this be a problem because i use username and password instead of SSH Key.

So it is working but my problem is that i use the phones to turn on/off light when we arrive or left the house. Going away is no problem but coming home will turn on the light later because of the delay with the errors.

But most important a big compliment for your plugin and support. Love the way it is working and hoping we find more options to add to it

Code: Select all

2018-09-26 08:30:11.164 (iDetect) Considered absent: Miche-GSMl
2018-09-26 08:30:15.671 (iDetect) Polling presense data from 192.168.0.1
2018-09-26 08:30:16.219 (iDetect) Using password instead of ssh public key authentication for 192.168.0.1 (less secure!)
2018-09-26 08:30:16.220 (iDetect) Fetching data from 192.168.0.1 using: sshpass -p **secret** ssh -o StrictHostKeyChecking=no -o ConnectTimeout=2 -p9325 <username router>@192.168.0.1 export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH;arp -a | grep '..:..:..:..:..:..' | awk '{print $ 4}';exit
2018-09-26 08:30:18.240 (iDetect) SSH subprocess for 192.168.0.1 timeout
2018-09-26 08:30:18.240 (iDetect) SSH command on 192.168.0.1 took 20 milliseconds.
2018-09-26 08:30:18.241 (iDetect) Routerinfo:{'user': '<username router>', 'initialized': True, 'port': 9325, 'prospone': datetime.datetime(2018, 9, 26, 8, 30, 48, 241000), 'cmd': "export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH;arp -a | grep '..:..:..:..:..:..' | awk '{print $ 4}';exit", 'errorcount': 1}
2018-09-26 08:30:18.241 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:22.751 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:25.489953
2018-09-26 08:30:22.751 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:27.759 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:20.481861
2018-09-26 08:30:27.759 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:32.768 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:15.472683
2018-09-26 08:30:32.768 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:37.787 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:10.453300
2018-09-26 08:30:37.788 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:42.683 MQTT: Topic: domoticz/in, Message: {"idx":3417,"nvalue":0,"svalue":"26.8"}
2018-09-26 08:30:47.754 (iDetect) Prosponed connection to 192.168.0.1 for 0:00:00.486358
2018-09-26 08:30:47.754 (iDetect) Did not retreive WLAN information from any router
2018-09-26 08:30:52.769 (iDetect) Polling presense data from 192.168.0.1
2018-09-26 08:30:52.769 (iDetect) Using password instead of ssh public key authentication for 192.168.0.1 (less secure!)
2018-09-26 08:30:52.769 (iDetect) Fetching data from 192.168.0.1 using: sshpass -p **secret** ssh -o StrictHostKeyChecking=no -o ConnectTimeout=2 -p9325 <username router>@192.168.0.1 export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH;arp -a | grep '..:..:..:..:..:..' | awk '{print $ 4}';exit
2018-09-26 08:30:53.742 (iDetect) ssh command on 192.168.0.1 returned:b'00:23:eb:78:00:d9\n5c:cf:7f:af:bd:a8\nd0:17:c2:ba:15:19\nc0:3f:d5:69:c5:19\n60:01:94:29:df:03\n4c:32:75:35:5e:6f\n5c:cf:7f:4c:52:27\n2c:3a:e8:2e:89:6d\n00:11:32:2c:12:f1\nf0:9f:c2:a6:cc:96\ne0:50:8b:32:5b:34\nd0:e7:82:da:08:fd\n44:c3:46:48:e8:2b\n00:1c:85:0a:41:f3\n8c:89:a5:cc:85:df\nd0:b1:28:37:4e:23\nb8:27:eb:16:63:96\n00:11:32:0f:0a:dc\n28:6c:07:85:e7:60\n5c:cf:7f:58:42:f1\nac:cf:23:6d:3e:58\n64:a2:f9:28:5a:a1\nf0:9f:c2:a6:c7:18\n00:11:32:2c:12:f2\n5c:cf:7f:f9:09:7a\n78:8a:20:83:e1:27\n'
2018-09-26 08:30:53.742 (iDetect) SSH command on 192.168.0.1 took 972 milliseconds.
2018-09-26 08:30:53.743 (iDetect) Found these devices connected:['00:23:EB:78:00:D9', '5C:CF:7F:AF:BD:A8', 'D0:17:C2:BA:15:19', 'C0:3F:D5:69:C5:19', '60:01:94:29:DF:03', '4C:32:75:35:5E:6F', '5C:CF:7F:4C:52:27', '2C:3A:E8:2E:89:6D', '00:11:32:2C:12:F1', 'F0:9F:C2:A6:CC:96', 'E0:50:8B:32:5B:34', 'D0:E7:82:DA:08:FD', '44:C3:46:48:E8:2B', '00:1C:85:0A:41:F3', '8C:89:A5:CC:85:DF', 'D0:B1:28:37:4E:23', 'B8:27:EB:16:63:96', '00:11:32:0F:0A:DC', '28:6C:07:85:E7:60', '5C:CF:7F:58:42:F1', 'AC:CF:23:6D:3E:58', '64:A2:F9:28:5A:A1', 'F0:9F:C2:A6:C7:18', '00:11:32:2C:12:F2', '5C:CF:7F:F9:09:7A', '78:8A:20:83:E1:27']
2018-09-26 08:30:53.743 Status: (iDetect) Connection restored for 192.168.0.1
2018-09-26 08:30:53.758 Status: (iDetect) Changing presence of iDetect - Miche-GSMl to On 

[/code]