Python plugin: Presence detection from wireless router

[EscApe]

@mideoo

Thanks!

I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.

Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).

You could experiment by increasing them a little in line 84 of the plugin:

Code: Select all

def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):

I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.

The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.

[EscApe]

Just published version 0.7.2

Enhanced: Found a different way to prevent multiple router configurations from overloading the (single threaded) python plugin framework when timeouts occur.
The timeouts per ssh session are a little more relaxed now and the plugin will just skip a poll if a previous poll cycle could not complete in time.
It will also warn you if a poll cycle takes over 50% of the poll interval time, because that kind of (wait) times could indicate that something is wrong (slowed down by cpu, network, router). On my single router setup it takes about 3% of a 10 second interval and i would expect most configurations to stay below 15%, even with multiple routers. Higher percentages might have a performance impact on Domoticz or at least it’s single threaded python plugin framework. If you just want to check the percentages on your setup you can see them in debug mode.

[mikeoo]

@mideoo

Thanks!

I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.

Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).

You could experiment by increasing them a little in line 84 of the plugin:

Code: Select all

def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):

I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.

The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.

I run it now for 24 hours and it look like it is working fine. I use an other system that also use ARP and running it next to each other the test how the plugin is doing. Until now your plugin is appropriable 30 sec quicker with detecting a phone then the older one so that look great and it respond much better.

Only problem i have with ARP based function that sometimes the phones goes offline/online for a short while.

This is one phone log with the plugin active and the other is the log of the same phone and with the old ARP system. So with your plugin it goes less Offline/Online but still not found a solution for it. Battery saving and Wifi is always active on the Android phone but i think it is a problem with Android to try the save battery and kill things.

So i will test with Grace period if i can set this longer to maybe fix the on/off situation for about 2 min
You mean to adjust alltimeout=2. Is the time set per minute ?

Plugin

Code: Select all

2018-09-27 06:47:21	On
2018-09-27 06:45:29	Off
2018-09-27 06:20:06	On
2018-09-27 06:18:13	Off

Old ARP system

Code: Select all

2018-09-27 07:52:20	Off
2018-09-27 06:47:28	On
2018-09-27 06:47:16	Off
2018-09-27 06:20:08	On
2018-09-27 06:19:56	Off
2018-09-27 05:44:25	On
2018-09-27 05:32:57	Off
2018-09-27 02:05:41	On
2018-09-27 02:04:35	Off

Unifi use a Unify Controller to control the AP and if you don't use that you need to use a App for phone/tablet
You can't login in directly to a AP but it is centralized with a DB running on my Raspberry. The edgerouter X have it's one Gui but there is also an UniFy secure Gateway which is a bit the same as the Edgerouter but that one is allso using the Unify Controller software. So you can change settings for everything with one piece of software.

But need more digging to see if what option there are for this but it is quit complex so the take a while

Found Bug:
I can't switch the iDetect - Override to On. When i click on it a pop-up said Switching On but it stays Off. Also nothing in the logfile
For testing I removed everything so plugin was gone in Hardware and noting left in Devices.
Reboot Raspberry
Add the plugin-in again
Still same problem

[EscApe]

@mikeoo

I published 0.7.2 last night, which has more relaxed timeouts. You should not need to modify them in the plugin.

Unfortunately there are still devices that temporarily drop the wifi connection completely. My iPhone 5 would (occasionally) but later iPhones do keep the wifi connection intact. At least for the chipset based detection methods... It is thinkable that they are dropped from the arp tables if the don't send any data through the router/switch (i don't know the inner workings of arp that well). A longer 'grace period' is indeed the right way to ignore those dropouts. Devices will still be detected quickly if they get home, but it takes longer to detect that they have left.
Using arp on two routers seems a bit redundant at first, but it might indeed speed up the detection if devices start communicating through one of the two routers only.

I can't switch the iDetect - Override to On

Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess

[mikeoo]

@mikeoo

I can't switch the iDetect - Override to On

Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess

Fixed indeed when you choose the right options

Love it

[hjzwiers]

Just updated to version 0.7.2
Using a Asus Ac86U and Ac68u in AImesh mode.
Was curious if the updates would work without the #forcegeneric use. This is not the case, without the #forcegeneric iDetect only sees the phones connected to the main router (and not phones connected to the mesh)

The logging without #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using chipset specific wl command on router 192.168.1.1 for interfaces eth5 & eth6 (=wl eth5 eth6)

With #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using generic brctl command on router 192.168.1.1. Will respond slower and on some routers a little less reliable to absence

Please do not remove this feature, in AImesh it is still needed

[EscApe]

Strange. I didn’t remove that option yet, but if it is somehow ignored anyway you can use the new approach. Add =brctl or =arp after the router address eg, 192.168.0.1=brctl

This should have the same effect as forcegeneric with one difference: you now specify it on a per server basis.

[hjzwiers]

=brctl seems to work fine

what is the difference between =brctl and =arp ?

[EscApe]

I am no network expert, so if you want know more then you will have to find a better source. Basically both give you information about your network layout, but on a different level. Arp is about discovering corresponding mac addresses from ip addresses. De bridge has to know what mac addresses are on which (virtual) port.

In my experience with a single router setup the information from the bridge (brctl) is quicker in detecting when a device leaves. On the downside it is also quicker to forget a device that has temporarily dropped the connection (power saving on some phones). Arp is to unreliable on my asus router. In some cases it would keep devices listed even if they had disconnected hours ago.

Bottom line: both are good in detecting when you get home. Brctl is quicker (and more reliable on my router) to detect that you have left.

[hjzwiers]

Tried again to run without #forcegeneric or =brctl because idetect stopped logging the phones

Before 0,7.2 I did not get these, but this is in the loging:

2018-10-04 19:50:13.079 Error: (iDetect) 'onHeartbeat' failed 'TypeError':'getrouter() got an unexpected keyword argument 'port''.
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 656 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 584 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 287 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function getactivemacs

after the initial errors this message

2018-10-04 19:50:58.069 Status: (iDetect) Warning! Skipping this poll cycle because the previous session has not finished yet (investigate what is slowing things down if you frequently get this message)

Update with =brctl started things back up ok.

I am on AC86U and AC68U in AImesh config.

[EscApe]

I have quickly corrected the type error directly on github. That should fix the redetection from crashing. It was an oversight on my part.

The fact that tou ran into this error means the router capabilities were not detected on script startup. The plugin will then retry to get the capabilities on the next heartbeat (that’s why i didn’t catch my mistake when testing 0.7.2

The debug log should tell you more about why it fails to get the capabilities.

[hjzwiers]

Updated to 07.3. All I get now is:

Error: (iDetect) Failed 3 times to get capabilities for 192.168.1.1. Retry interval automatically reduced for this router.

Just also noticed the switches have not updated since 30 Sept .....

Error: (iDetect) Polling 192.168.1.1 has failed 3 times. Poll interval automatically reduced for this router.

Don't see the connection in the router log anymore (Pub key). Strange

[EscApe]

There is not much to tell without a debug log, apart from the plugin not being able to get info from the router. Any changes on the 30/9?

Have you tried ssh from the command line (from the same user account that is running Domoticz.)
New router firmware could lead to the router having a new (host) key.

[hjzwiers]

Thanks for the suggestion, there was a firmware update. Will check the host key.

[hjzwiers]

I needed to add the router to known hosts again, was the reason for failing.

Just to report. I tested without =brctl again. iDetect then only sees the mac addresses on the main router, not on the mesh router. With =brctl it working again as it was

[EscApe]

For detection using the wifi chipset the plugin must connect to the router containing that chipset, so in your mesh setup you will have to add both mesh nodes to the plugin.

arp and brctl do not rely on the information from the wifi chipset, but on discovery protocols on the entire network. That's why they will see every device on every router, but are a bit slower to respond to changes.

[hjzwiers]

With Asus AImesh putting in both IP addresses does not work, still only sees the main router.

=arp fails, =brctl works

Just giving the info, I'll stick to brctl thnx

[EscApe]

Thanks for sharing the info. If you ever want to investigate why the chipset approach doesn’t work just let me know (forum or pm). I’m interested to find out how the aimesh works.

[AurelioF]

I confirm that it works with Unifi AP Pro
Thanks for sharing

[mikeoo]

I confirm that it works with Unifi AP Pro
Thanks for sharing

Do you use ARP with the Unifi. I use also Unifi and interesting how you use it.