Page 20 of 55
Re: Python plugin: Presence detection from wireless router
Posted: Wednesday 26 September 2018 11:40
by EscApe
@mideoo
Thanks!
I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.
Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).
You could experiment by increasing them a little in line 84 of the plugin:
Code: Select all
def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):
I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.
The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.
Re: Python plugin: Presence detection from wireless router
Posted: Wednesday 26 September 2018 20:52
by EscApe
Just published version 0.7.2
Enhanced: Found a different way to prevent multiple router configurations from overloading the (single threaded) python plugin framework when timeouts occur.
The timeouts per ssh session are a little more relaxed now and the plugin will just skip a poll if a previous poll cycle could not complete in time.
It will also warn you if a poll cycle takes over 50% of the poll interval time, because that kind of (wait) times could indicate that something is wrong (slowed down by cpu, network, router). On my single router setup it takes about 3% of a 10 second interval and i would expect most configurations to stay below 15%, even with multiple routers. Higher percentages might have a performance impact on Domoticz or at least it’s single threaded python plugin framework. If you just want to check the percentages on your setup you can see them in debug mode.
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 27 September 2018 8:18
by mikeoo
EscApe wrote: ↑Wednesday 26 September 2018 11:40
@mideoo
Thanks!
I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.
Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).
You could experiment by increasing them a little in line 84 of the plugin:
Code: Select all
def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):
I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.
The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.
I run it now for 24 hours and it look like it is working fine. I use an other system that also use ARP and running it next to each other the test how the plugin is doing. Until now your plugin is appropriable 30 sec quicker with detecting a phone then the older one so that look great and it respond much better.
Only problem i have with ARP based function that sometimes the phones goes offline/online for a short while.
This is one phone log with the plugin active and the other is the log of the same phone and with the old ARP system. So with your plugin it goes less Offline/Online but still not found a solution for it. Battery saving and Wifi is always active on the Android phone but i think it is a problem with Android to try the save battery and kill things.
So i will test with Grace period if i can set this longer to maybe fix the on/off situation for about 2 min
You mean to adjust
alltimeout=2. Is the time set per minute ?
Plugin
Code: Select all
2018-09-27 06:47:21 On
2018-09-27 06:45:29 Off
2018-09-27 06:20:06 On
2018-09-27 06:18:13 Off
Code: Select all
2018-09-27 07:52:20 Off
2018-09-27 06:47:28 On
2018-09-27 06:47:16 Off
2018-09-27 06:20:08 On
2018-09-27 06:19:56 Off
2018-09-27 05:44:25 On
2018-09-27 05:32:57 Off
2018-09-27 02:05:41 On
2018-09-27 02:04:35 Off
Unifi use a Unify Controller to control the AP and if you don't use that you need to use a App for phone/tablet
You can't login in directly to a AP but it is centralized with a DB running on my Raspberry. The edgerouter X have it's one Gui but there is also an UniFy secure Gateway which is a bit the same as the Edgerouter but that one is allso using the Unify Controller software. So you can change settings for everything with one piece of software.
But need more digging to see if what option there are for this but it is quit complex so the take a while
Found Bug:
I can't switch the
iDetect - Override to On. When i click on it a pop-up said Switching On but it stays Off. Also nothing in the logfile
For testing I removed everything so plugin was gone in Hardware and noting left in Devices.
Reboot Raspberry
Add the plugin-in again
Still same problem
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 27 September 2018 17:33
by EscApe
@mikeoo
I published 0.7.2 last night, which has more relaxed timeouts. You should not need to modify them in the plugin.
Unfortunately there are still devices that temporarily drop the wifi connection completely. My iPhone 5 would (occasionally) but later iPhones do keep the wifi connection intact. At least for the chipset based detection methods... It is thinkable that they are dropped from the arp tables if the don't send any data through the router/switch (i don't know the inner workings of arp that well). A longer 'grace period' is indeed the right way to ignore those dropouts. Devices will still be detected quickly if they get home, but it takes longer to detect that they have left.
Using arp on two routers seems a bit redundant at first, but it might indeed speed up the detection if devices start communicating through one of the two routers only.
I can't switch the iDetect - Override to On
Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess
Re: Python plugin: Presence detection from wireless router
Posted: Friday 28 September 2018 8:12
by mikeoo
EscApe wrote: ↑Thursday 27 September 2018 17:33
@mikeoo
I can't switch the iDetect - Override to On
Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess
Fixed indeed when you choose the right options
Love it
Re: Python plugin: Presence detection from wireless router
Posted: Friday 28 September 2018 22:56
by hjzwiers
Just updated to version 0.7.2
Using a Asus Ac86U and Ac68u in AImesh mode.
Was curious if the updates would work without the #forcegeneric use. This is not the case, without the #forcegeneric iDetect only sees the phones connected to the main router (and not phones connected to the mesh)
The logging without #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using chipset specific wl command on router 192.168.1.1 for interfaces eth5 & eth6 (=wl eth5 eth6)
With #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using generic brctl command on router 192.168.1.1. Will respond slower and on some routers a little less reliable to absence
Please do not remove this feature, in AImesh it is still needed
Re: Python plugin: Presence detection from wireless router
Posted: Friday 28 September 2018 23:09
by EscApe
Strange. I didn’t remove that option yet, but if it is somehow ignored anyway you can use the new approach. Add =brctl or =arp after the router address eg, 192.168.0.1=brctl
This should have the same effect as forcegeneric with one difference: you now specify it on a per server basis.
Re: Python plugin: Presence detection from wireless router
Posted: Saturday 29 September 2018 9:39
by hjzwiers
=brctl seems to work fine
what is the difference between =brctl and =arp ?
Re: Python plugin: Presence detection from wireless router
Posted: Sunday 30 September 2018 10:33
by EscApe
I am no network expert, so if you want know more then you will have to find a better source. Basically both give you information about your network layout, but on a different level. Arp is about discovering corresponding mac addresses from ip addresses. De bridge has to know what mac addresses are on which (virtual) port.
In my experience with a single router setup the information from the bridge (brctl) is quicker in detecting when a device leaves. On the downside it is also quicker to forget a device that has temporarily dropped the connection (power saving on some phones). Arp is to unreliable on my asus router. In some cases it would keep devices listed even if they had disconnected hours ago.
Bottom line: both are good in detecting when you get home. Brctl is quicker (and more reliable on my router) to detect that you have left.
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 04 October 2018 20:00
by hjzwiers
Tried again to run without #forcegeneric or =brctl because idetect stopped logging the phones
Before 0,7.2 I did not get these, but this is in the loging:
2018-10-04 19:50:13.079 Error: (iDetect) 'onHeartbeat' failed 'TypeError':'getrouter() got an unexpected keyword argument 'port''.
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 656 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 584 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 287 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function getactivemacs
after the initial errors this message
2018-10-04 19:50:58.069 Status: (iDetect) Warning! Skipping this poll cycle because the previous session has not finished yet (investigate what is slowing things down if you frequently get this message)
Update with =brctl started things back up ok.
I am on AC86U and AC68U in AImesh config.
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 04 October 2018 20:49
by EscApe
I have quickly corrected the type error directly on github. That should fix the redetection from crashing. It was an oversight on my part.
The fact that tou ran into this error means the router capabilities were not detected on script startup. The plugin will then retry to get the capabilities on the next heartbeat (that’s why i didn’t catch my mistake when testing 0.7.2
The debug log should tell you more about why it fails to get the capabilities.
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 04 October 2018 21:40
by hjzwiers
Updated to 07.3. All I get now is:
Error: (iDetect) Failed 3 times to get capabilities for 192.168.1.1. Retry interval automatically reduced for this router.
Just also noticed the switches have not updated since 30 Sept .....
Error: (iDetect) Polling 192.168.1.1 has failed 3 times. Poll interval automatically reduced for this router.
Don't see the connection in the router log anymore (Pub key). Strange
Re: Python plugin: Presence detection from wireless router
Posted: Friday 05 October 2018 7:04
by EscApe
There is not much to tell without a debug log, apart from the plugin not being able to get info from the router. Any changes on the 30/9?
Have you tried ssh from the command line (from the same user account that is running Domoticz.)
New router firmware could lead to the router having a new (host) key.
Re: Python plugin: Presence detection from wireless router
Posted: Friday 05 October 2018 7:09
by hjzwiers
Thanks for the suggestion, there was a firmware update. Will check the host key.
Re: Python plugin: Presence detection from wireless router
Posted: Saturday 06 October 2018 9:12
by hjzwiers
I needed to add the router to known hosts again, was the reason for failing.
Just to report. I tested without =brctl again. iDetect then only sees the mac addresses on the main router, not on the mesh router. With =brctl it working again as it was
Re: Python plugin: Presence detection from wireless router
Posted: Saturday 06 October 2018 9:40
by EscApe
For detection using the wifi chipset the plugin must connect to the router containing that chipset, so in your mesh setup you will have to add both mesh nodes to the plugin.
arp and brctl do not rely on the information from the wifi chipset, but on discovery protocols on the entire network. That's why they will see every device on every router, but are a bit slower to respond to changes.
Re: Python plugin: Presence detection from wireless router
Posted: Saturday 06 October 2018 19:37
by hjzwiers
With Asus AImesh putting in both IP addresses does not work, still only sees the main router.
=arp fails, =brctl works
Just giving the info, I'll stick to brctl thnx
Re: Python plugin: Presence detection from wireless router
Posted: Saturday 06 October 2018 20:48
by EscApe
Thanks for sharing the info. If you ever want to investigate why the chipset approach doesn’t work just let me know (forum or pm). I’m interested to find out how the aimesh works.
Re: Python plugin: Presence detection from wireless router
Posted: Thursday 11 October 2018 15:10
by AurelioF
I confirm that it works with Unifi AP Pro
Thanks for sharing
Re: Python plugin: Presence detection from wireless router
Posted: Friday 12 October 2018 7:20
by mikeoo
AurelioF wrote: ↑Thursday 11 October 2018 15:10
I confirm that it works with Unifi AP Pro
Thanks for sharing
Do you use ARP with the Unifi. I use also Unifi and interesting how you use it.