Python plugin: Presence detection from wireless router

[EscApe]

I would be happy to add iwinfo as one of the commands to detect. To really add it to the plug in i need to know how it behaves. That would take some experimenting. Maybe you can provide some input on it? From the openwrt wiki i understand that a query would look like "iwinfo wlan0/wl0/ath0 assoclist". I'm guessing the wiki example has an atheros chipset. Is the iwinfo command specific to the atheros chipset?

I'll need to know:

- How to detect the wireless interaces in use on the router? (it will probably not always be wlan0/wl0/ath0?)
- The output (exact format) of the iwinfo command (so i can make the plugin can interpret it)

Caveats: These kind of things take a lot of trial and error. There are a lot of different routers with different chipset and different firmwares. I would love to make the plugin that flexible but cannot do that without owning every type of router. I will have to depend on you and others to provide information and preferably your modifications to the plugin to get it to work, so i can integrate all different examples into one smart and flexible plugin.

[mrkrabs]

thanks for your interest, EscApe! To know the wireless interface i think it's possible to use output from ifconfig command:

Code: Select all

[spoiler]root@ArcherC7:~# ifconfig
br-lan    Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          inet addr:10.0.10.200  Bcast:10.0.10.255  Mask:255.255.255.0
          inet6 addr: xx::724f:57ff:fe15:46fc/64 Scope:Link
          inet6 addr: xx:64a4:7076::1/60 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4717565 errors:0 dropped:62 overruns:0 frame:0
          TX packets:671295 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3596807348 (3.3 GiB)  TX bytes:84913051 (80.9 MiB)

eth0      Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          inet6 addr: xx::724f:57ff:fe15:46fc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:198961547 errors:0 dropped:0 overruns:930 frame:0
          TX packets:292922987 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3803219329 (3.5 GiB)  TX bytes:2401686031 (2.2 GiB)
          Interrupt:4

eth0.1    Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:198961445 errors:0 dropped:0 overruns:0 frame:0
          TX packets:292420520 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:223560081831 (208.2 GiB)  TX bytes:310297882886 (288.9 GiB)

eth0.2    Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          inet6 addr: xx::724f:57ff:fe15:46fc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:502453 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:169755149 (161.8 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3625 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3625 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:489440 (477.9 KiB)  TX bytes:489440 (477.9 KiB)

wlan0     Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          inet6 addr: xx::724f:57ff:fe15:46fb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14779947 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26167110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6394952925 (5.9 GiB)  TX bytes:33844936894 (31.5 GiB)

wlan1     Link encap:Ethernet  HWaddr 70:4F:57:15:46:xx
          inet6 addr: xx::724f:57ff:fe15:46fc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:67817896 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45769125 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:56869588111 (52.9 GiB)  TX bytes:9601538782 (8.9 GiB)[/spoiler]

then i changed your script, the routerscript part to use this line

Code: Select all

ifconfig | cut -d ' ' -f1| tr ':' '\n' | grep wlan

and it's lists 2 wlan interfaces (wlan0 - 2.4ghz wifi and wlan1 - 5ghz wifi). It seems to work successfully, here's the output from log (i already changed wl to iwinfo, so it says that in the lof ):

Code: Select all

2018-05-09 13:01:18.146 (Presence detection) Using iwinfo command on router (best method) for interfaces ['wlan0', 'wlan1']

and here comes the tricky part for me - the output of iwinfo wlan0 assoclist (note that there should not be the "-i" option as otherwise the command fails):

Code: Select all

root@ArcherC7:~# iwinfo wlan0 assoclist
98:DE:D0:xx:xx:xx  -70 dBm / -104 dBm (SNR 34)  10 ms ago
        RX: 650.0 MBit/s, VHT-MCS 7, 80MHz, VHT-NSS 2    772389 Pkts.
        TX: 6.0 MBit/s                               1582496 Pkts.
        expected throughput: unknown

F4:37:B7:xx:xx:xx  -49 dBm / -104 dBm (SNR 55)  210 ms ago
        RX: 24.0 MBit/s                               548041 Pkts.
        TX: 6.0 MBit/s                                814975 Pkts.
        expected throughput: unknown

So as you can see it's blocked in the so-called groups of 4 lines, where each line starts with a MAC address of the device. And to my shame - i don't think i know how to strip off everything apart from MAC addresses from the list.

Apart from stripping everything out - your scripts seems to work fine with the minimum changes i described above!

EDIT: can you please let me know on how the list with MAC addresses should look after this function?:

Code: Select all

list=[]
for item in sshdata.splitlines():
	list.append(item.decode("utf-8").split(" ")[-1].upper())
return list

as i came up with this function :

Code: Select all

list=[]
for item in sshdata.split():
    if item.count(':') == 5:
	    list.append(item.decode('UTF-8').upper())
retunr list

that creats a list that looks like [u'xx:xx:86:15:6D:1a', u'xx:xx:90:13:5B:3A'] but it still doesn't seem to work, the log says 'TypeError':'a bytes-like object is required, not 'str''

[EscApe]

It's best to leave the parsing of the output as is unless it is impossible to get some pre-formatted output from the router. Otherwise the plugin will become less flexible. The section you are referring to parses the output one line at a time and in case the wl command is used it will strip the "assoclist " prefix from every line. In hindsight i should have followed my own advice and strip those in the routercommand :-$. In case arp is used it will have no effect.

The output of the function is a python list of uppercase strings, containing the mac addresses eg: ["11:22:33:44:55:66", "11:22:33:44:55:77", "11:22:33:44:55:88"].

I will not change that part of the plugin because it would break the current function of the plugin.
We need to somehow get a list of clean mac addresses from the router. I'm currently researching how that can be accomplished using awk and grep on the output of iwinfo. If anyone out there knows how to accomplish this, please pitch in!

Come to think of it: Since your router does not offer the wl command the plugin should fall back to the arp command. I would expect that to just work in the current version of the plugin. Can you try "arp -a" on the router and share the output?

In the current version the full arp command is:

Code: Select all

arp -a | grep br0 | awk '{print $ 4}'

You could try to change that to

Code: Select all

arp -a | awk '{print $ 4}'

Just incase br0 means nothing to your router

[mrkrabs]

and again - thanks. here's the output of my arp -a:

Code: Select all

arp -a
IP address       HW type     Flags       HW address            Mask     Device
10.0.10.12       0x1         0x2         xx:xx:86:03:6d:0d     *        br-lan
10.0.10.76       0x1         0x2         xx:xx:d0:be:18:41     *        br-lan
10.0.10.75       0x1         0x2         xx:xx:95:24:a4:92     *        br-lan
10.0.10.27       0x1         0x2         xx:xx:a9:d0:4e:a2     *        br-lan
10.0.10.3        0x1         0x2         xx:xx:90:66:5b:3a     *        br-lan
10.0.10.19       0x1         0x2         xx:xx:0e:08:8d:9a     *        br-lan
10.0.10.1        0x1         0x2         xx:xx:11:96:d9:cb     *        br-lan
10.0.10.30       0x1         0x2         xx:xx:eb:4a:fa:31     *        br-lan

The output of the function is a python list of uppercase strings, containing the mac addresses eg: ["11:22:33:44:55:66", "11:22:33:44:55:77", "11:22:33:44:55:88"].

that's strange, since i can get this output with the .split command i posted earlier, but still, i got that TypeError error in log.

[EscApe]

Did you enable debug option on the hardware configuration page of the plugin? It will give a lot more info about what is going on, including raw output from the router.

I might have found a way to parse the iwinfo output. This should give you a clean list of mac adresses connected to wlan0:

Code: Select all

iwinfo wlan0 assoclist | grep '^..:..:..:..:..:.. ' | cut -d' ' -f1

Can you conform this?

Also the arp command should indeed be changed as described in my previous post. Your router doesn't state br0, but br-lan. Since it is nog really necessary we can just remove the grep statement.

[mrkrabs]

yes, i confirm it, the command you provide gives out a clean column on MAC addresses like this:

Code: Select all

11:22:D0:BE:18:41
22:33:86:03:6D:0D
33:44:90:66:5B:3A

[EscApe]

Now for the real test .... this should theoretically work without breaking the plugin for already supported routers.... <holding my breath>

Can you replace the plugin.py with the attached version. This should be considered a beta

Edit: There are a few more changes than just using the iwinfo command.
I have followed my own advice and now get a cleaned up list from the router (no more splitting in python other than individual lines)
Any interface name starting with wlan is included. Not just eth*

[mrkrabs]

finally! it works! thank you very much! i actually managed to make it work myself with your stripped grep command from post above, but my version got wlan0 and wlan1 hardcoded, but your now show that it checks all interfaces:

Code: Select all

Using iwinfo command on router (best method) for interfaces ['eth0', 'eth0.1', 'eth0.2', 'wlan0', 'wlan1']

once again thanks for the great plugin and all your help!

[EscApe]

I'm glad it works. Thank you for providing the info. It helped make the plugin more universal.

Your log item interests me. The plugin should only check interfaces that are actually in use, to minimize the amount of work on every poll. In your case it will check 5 interfaces but probably not all of them are needed. Are the wlan interfaces just aliases for the eth0.1 en eth0.2 maybe? Are eth0, eth0.1 and eth0.2 all wireless interfaces? Does the plugin return multiple instances of the same mac address (should show in the log if debug mode is enabled)?

I would like to see if we can optimize this. Doesn't need to be today though
When polling every 5 or 10 seconds every fraction of a second counts because the ssh command will block the (already busy) Domoticz event system. And well... i just like things working as good as they can

The plugin will upon its initialization try all interfaces that have names starting with eth or wlan. On my asus it will use the wl command and the wl command will exit with a return code other than 0 if the interface is not an active wireless interface. How can we determine which wireless interfaces are really in use using the iwinfo command and ignore aliases/links?

[EscApe]

I have moved the files and instructions for this plugin to GitHub. The forum system would keep messing up my attachments every time i modified the top post with the latest version.

The version on GitHub is the latest version, including the iwinfo command as a method for detection. So now wl, iwinfo and arp are automatically detected, in an effort to support as many routers as possible.

[mrkrabs]

i am not sure if eth0.1 and eth0.2 are actual aliases (could be something related to IPv6 connections?) but they are empty and do not have any active MACs attached to them, none of the interfaces actually have any MACs apart from wlan0 and wlan1. what i found is that if you change this command from your script :

Code: Select all

ifconfig | cut -d ' ' -f1| tr ':' '\n' | grep '^eth\|^wlan'

to this one:

Code: Select all

iwinfo | cut -d ' ' -f1| tr ':' '\n' | grep '^eth\|^wlan'

which basically means replacing ifconfig with iwinfo it will only show wlan0 and wlan1, which are the only active interfaces. I hope that helps For your reference, here are 2 outputs of the above mentioned commands from my OpenWrt router:

Code: Select all

root@ArcherC7:~# ifconfig | cut -d ' ' -f1| tr ':' '\n' | grep '^eth\|^wlan'
eth0
eth0.1
eth0.2
wlan0
wlan1

Code: Select all

root@ArcherC7:~# iwinfo | cut -d ' ' -f1| tr ':' '\n' | grep '^eth\|^wlan'
wlan0
wlan1

If you could modify the script somehow to run the latter command on the OpenWrt routers that might solve the issue with unused interfaces. Please let me know if you need any more info from me and my router And thanks again for your help!

EDIT: and no, none of the MACs are duplicated in the domoticz debug log and i don't think eth0/1 etc are wireless interfaces, they are likely wired ones, but i dont have anything connected to that router via lan ports

[EscApe]

That's great info. Since we already know which command to use before the interfaces are queried it is a minor change that shouldn't affect other router types.

I already published the modification on https://github.com/d-EScape/Domoticz_iDetect

Please let me know if it works or breaks anything

[mrkrabs]

seems like all working good with the latest version from github, and it only detects the active interfaces wlan0 and wlan1:

Code: Select all

2018-05-10 13:07:14.348 (Presence detection) Using iwinfo command on router (best method) for interfaces ['wlan0', 'wlan1']

there's one more thing that i find personally for me not working as expected (not related to the interface detection method) but still quite annoying for my setup, with the 2 devices renamed (and i personally do that) and the option Show obsolete devices as unavailable or delete obsolete devices selected, after the restart the plugin just recreates 2 new devices with the default names and removes the old ones (or leaves them as inactive). It would be good if the plugin could remember the renamed devices and not recreate them after the restart (possibly remember them by their ID's and not by their names?)

It is especially frustrating since i create notifications for devices and have to recreate them (cause new devices are added) everytime when domoticz restarts.

UPDATE: figured out that i have to use the same name for the Device itself and in "MAC addresses to monitor:" field before the actual MAC for the device not to be removed during restart

[EscApe]

That is (or was) indeed an annoyance i had learned to live with. Turns out (with my current experience in developing plugins) it was easily fixed. A new version has been published on GitHub.

You can now change the (display) names of devices... At a small price though: Your existing devices will be deleted and replaced by new ones. You will loose any log (history) of those devices.

Please keep the names in the configuration short. I have not yet found any technical limitations for the names, but they will make the device list (under settings) look ugly.

[mattia81]

I don't use the router to release IP addresses but I have installed DHCP server on my Raspberry (the same that hosts Domoticz)

The DHCP server set STATIC IP based on Mac address to all my devices.

How could I use this script?

[EscApe]

The plugin doesn't use ip addresses and it should not matter that your raspberry is running the dhcp server. Presence is detected by directly querying the wireless interface on your router for connected clients. You should use the plugin just as described.

[Sushukka]

Hello,
Testing here with Telewell brand router:
https://www.telewell.fi/en/product/inte ... uter-cat-4
Marvell is the manufacturer of the 3G/4G chip and system chip seems to be Broadcom BMIPS4350 V8.0

I can login with ssh and it has wl and ifconfig. FIlesystem is protected and "full" so cannot transfer any keys etc --> have to use user/password with sshpass. When manually logging in, it starts with dummy prompt (>). I have to write manually sh before being able to access the Linux side and commands like wl on the router. Could it be the reason why I get following errors to the log:

Code: Select all

2018-05-12 00:45:56.997 (Wifi presence detection) command: ['sshpass', '-p', 'password', 'ssh', '-o', 'StrictHostKeyChecking=no', '-o', 'ConnectTimeout=2', '[email protected]', 'sh \n #!/bin/sh\n\t\t\t\t\texport PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH\n\t\t\t\t\ttest=$(which wl > /dev/null 2>&1)\n\t\t\t\t\tif [ $? == 0 ]; then\n\t\t\t\t\t\t\tprintf "wl@"\n\t\t\t\t\t\t\tfor iface in $(ifconfig | cut -d \' \' -f1| tr \':\' \'\n\' | grep \'^eth\\|^wlan\')\n\t\t\t\t\t\t\tdo\n\t\t\t\t\t\t\t\t\ttest=$(wl -i $iface assoclist > /dev/null 2>&1)\n\t\t\t\t\t\t\t\t\tif [ $? == 0 ]; then\n\t\t\t\t\t\t\t\t\t\t\tprintf "$iface "\n\t\t\t\t\t\t\t\t\tfi\n\t\t\t\t\t\t\tdone\n\t\t\t\t\t\t\texit\n\t\t\t\t\tfi\n\t\t\t\t\ttest=$(which iwinfo > /dev/null 2>&1)\n\t\t\t\t\tif [ $? == 0 ]; then\n\t\t\t\t\t\t\tprintf "iwinfo@"\n\t\t\t\t\t\t\tfor iface in $(iwinfo | cut -d \' \' -f1| tr \':\' \'\n\' | grep \'^eth\\|^wlan\')\n\t\t\t\t\t\t\tdo\n\t\t\t\t\t\t\t\t\ttest=$(iwinfo wlan0 assoclist > /dev/null 2>&1)\n\t\t\t\t\t\t\t\t\tif [ $? == 0 ]; then\n\t\t\t\t\t\t\t\t\t\t\tprintf "$iface "\n\t\t\t\t\t\t\t\t\tfi\n\t\t\t\t\t\t\tdone\n\t\t\t\t\t\t\texit\n\t\t\t\t\tfi\t\t\t\t\t\n\t\t\t\t\ttest=$(which arp > /dev/null 2>&1)\n\t\t\t\t\tif [ $? == 0 ]; then\n\t\t\t\t\t\t\tprintf "arp"\n\t\t\t\t\tfi']
2018-05-12 00:45:56.997 Error: (Wifi presence detection) 'onStart' failed 'AttributeError'.
2018-05-12 00:45:56.997 Error: (Wifi presence detection) ----> Line 330 in /home/pi/domoticz/plugins/iDetect/plugin.py, function onStart
2018-05-12 00:45:56.997 Error: (Wifi presence detection) ----> Line 233 in /home/pi/domoticz/plugins/iDetect/plugin.py, function onStart
2018-05-12 00:45:56.997 Error: (Wifi presence detection) ----> Line 147 in /home/pi/domoticz/plugins/iDetect/plugin.py, function routercommand
2018-05-12 00:45:56.997 Error: (Wifi presence detection) ----> Line 107 in /home/pi/domoticz/plugins/iDetect/plugin.py, function getfromssh
2018-05-12 00:45:56.997 (Wifi presence detection) Processing 'PollIntervalDirective' message
2018-05-12 00:45:56.997 (Wifi presence detection) Heartbeat interval set to: 10.
2018-05-12 00:46:02.683 (MySensors Ethernet Gateway) Usage (Usage)
2018-05-12 00:46:06.493 (Wifi presence detection) Pushing 'onHeartbeatCallback' on to queue
2018-05-12 00:46:06.515 (Wifi presence detection) Processing 'onHeartbeatCallback' message
2018-05-12 00:46:06.515 (Wifi presence detection) Calling message handler 'onHeartbeat'.
2018-05-12 00:46:06.515 (Wifi presence detection) devid2domid: {}
2018-05-12 00:46:06.515 Error: (Wifi presence detection) 'onHeartbeat' failed 'AttributeError':''BasePlugin' object has no attribute 'routercmdline''.
2018-05-12 00:46:06.515 Error: (Wifi presence detection) ----> Line 334 in /home/pi/domoticz/plugins/iDetect/plugin.py, function onHeartbeat
2018-05-12 00:46:06.515 Error: (Wifi presence detection) ----> Line 292 in /home/pi/domoticz/plugins/iDetect/plugin.py, function onHeartbeat

[EscApe]

If the ssh server is not behaving like it should, that could be indeed be a reason for the plugin to fail. I guess you could do some experimenting by changing the getfromssh function. One thing i can think of is this:

Edit: The suggestion i made (below) doesn't make sense, sorry. That would invoke shell on the Domoticz host, not the router.
I don't have a solution and don't know if a router like that can ever be supported.

change line 81 of the script
completed=subprocess.run(cmd, stdout=subprocess.PIPE, timeout=alltimeout)
to
completed=subprocess.run(cmd, stdout=subprocess.PIPE, timeout=alltimeout, shell=True)/s]

[mrkrabs]

That is (or was) indeed an annoyance i had learned to live with. Turns out (with my current experience in developing plugins) it was easily fixed. A new version has been published on GitHub.

tried and tested - all good! thanks again

[Innovator]

@Escape yesterday attempted to activate the latest version of the python i-detect script on my new ASUS BRT-AC828.

Unfortunately, during testing I found that my phone, despite being active on Wifi, did not jump on.
When I look at the debug log, I see that the "arp command" is not activated. When I press on the command line of the AC828 wl I get Usage: wl clm_da_ver.

The result of that command is BRT-AC828 10.4-0.0.2217.273 (IPQ8064.ILQ.3.1 CSU3).
I think that the detection runs wrong on wl ... I have added an attachment from the debug log to the message. At this moment I solved it by putting # in the script for method = gotinfo [0], under which method = "arp" turned on.

Then it works fine! Thanks for the nice work!