Python plugin: Presence detection from wireless router

[McMelloW]

@EscApe
Hoi Escape,

Digging more in the subject of "Presence Detection" I discovered the Linux command arp. See the first bit of the man page of arp

Code: Select all

NAME
       arp - manipulate the system ARP cache

DESCRIPTION
       Arp  manipulates or displays the kernel's IPv4 network neighbour cache. It can add entries to the table, delete one or display the current content.

       ARP stands for Address Resolution Protocol, which is used to find the media access control address of  a  network neighbour for a given IPv4 Address.

MODES
       arp with no mode specifier will print the current content of the table. It is possible to limit the number of entries printed, by specifying an hardware address type, interface name or host address.

As it reads the network neighbour cache table, there is hardly any impact on the device. I guess this would be an easy and very fast tracker method local on my Raspberry Pi.
What do you think

[EscApe]

@EscApe
Hoi Escape,

Digging more in the subject of "Presence Detection" I discovered the Linux command arp. See the first bit of the man page of arp

Code: Select all

NAME
       arp - manipulate the system ARP cache

DESCRIPTION
       Arp  manipulates or displays the kernel's IPv4 network neighbour cache. It can add entries to the table, delete one or display the current content.

       ARP stands for Address Resolution Protocol, which is used to find the media access control address of  a  network neighbour for a given IPv4 Address.

MODES
       arp with no mode specifier will print the current content of the table. It is possible to limit the number of entries printed, by specifying an hardware address type, interface name or host address.

As it reads the network neighbour cache table, there is hardly any impact on the device. I guess this would be an easy and very fast tracker method local on my Raspberry Pi.
What do you think

Arp is already supported by the plugin. In fact if the autodetection does not find any hardware specific command it will fall back on generic commands like arp. Even then it is the third choice. It works and detects presence pretty fast, but -depending on the router- it can take (very) long for a disconnected device to be removed from the arp table. So it can show up as ‘present’ longer than desired. Running arp on the domoticz host will not always show all devices connected to the network, so it stil needs te be run on the router using ssh.

[McMelloW]

Arp is already supported by the plugin. In fact if the autodetection does not find any hardware specific command it will fall back on generic commands like arp. Even then it is the third choice. It works and detects presence pretty fast, but -depending on the router- it can take (very) long for a disconnected device to be removed from the arp table. So it can show up as ‘present’ longer than desired. Running arp on the domoticz host will not always show all devices connected to the network, so it stil needs te be run on the router using ssh.

There is a difference in arp and arp-scan. I am talking about arp
On my RPi, I did some test by switching off/on wifi on my mobile and arp shows only (incomplete) after the IP-address within seconds. You have to look for the MAC-address and not the IP-address.
I haven't figure out yet how to refresh this IPv4 network neighbour cache easely.
Is there a way to force iDetect to use arp as a 1st choice in an 'arp-tracker'

[EscApe]

Arp is already supported by the plugin. In fact if the autodetection does not find any hardware specific command it will fall back on generic commands like arp. Even then it is the third choice. It works and detects presence pretty fast, but -depending on the router- it can take (very) long for a disconnected device to be removed from the arp table. So it can show up as ‘present’ longer than desired. Running arp on the domoticz host will not always show all devices connected to the network, so it stil needs te be run on the router using ssh.

There is a difference in arp and arp-scan. I am talking about arp
On my RPi, I did some test by switching off/on wifi on my mobile and arp shows only (incomplete) after the IP-address within seconds. You have to look for the MAC-address and not the IP-address.
I haven't figure out yet how to refresh this IPv4 network neighbour cache easely.
Is there a way to force iDetect to use arp as a 1st choice in an 'arp-tracker'

You can run (force) any command you like over ssh. If you want to try it on your domoticz host then the plugin must connect to its own host using ssh (as a work around). Just configure it like <trackerhost>#ssh=arp -a (see the readme on github)

What do you mean by “ arp shows only (incomplete) after the IP-address within seconds”?
The plugin already looks for Mac not ip. I think there needs to be traffic between the host and the device for the device to show up in the hosts arp table. Broadcasts are also traffic. Anyway I abandoned arp on the domoticz host as an option long ago because devices would not always show up. If you find a reliable way then I will reconsider. I do appreciate your effort to find other options!

[McMelloW]

You can run (force) any command you like over ssh. If you want to try it on your domoticz host then the plugin must connect to its own host using ssh (as a work around). Just configure it like <trackerhost>#ssh=arp -a (see the readme on github)

Thanks, for your reply.

What do you mean by “ arp shows only (incomplete) after the IP-address within seconds”?

When enter arp on the command line, the response is the following table with a device not connected (192.168.1.14)

Code: Select all

pi@mcmellow-pi:~ $ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.134           ether   dc:4f:22:2d:db:06   C                     eth0
192.168.1.13            ether   d0:f8:8c:38:0e:96   C                     eth0
192.168.1.131           ether   dc:4f:22:2d:c3:fe   C                     eth0
192.168.1.154           ether   9c:80:df:54:87:3c   C                     eth0
192.168.1.18            ether   20:34:fb:f8:d8:d6   C                     eth0
192.168.1.14            (incomplete)
192.168.1.120           ether   a8:40:41:17:8b:1d   C                     eth0
192.168.1.132           ether   dc:4f:22:2d:bb:43   C                     eth0
192.168.1.113           ether   b4:6d:83:ae:06:1e   C                     eth0
192.168.1.117           ether   00:27:02:12:cc:fe   C                     eth0
192.168.1.133           ether   dc:4f:22:2d:87:5d   C                     eth0

The plugin already looks for Mac not ip. I think there needs to be traffic between the host and the device for the device to show up in the hosts arp table. Broadcasts are also traffic. Anyway I abandoned arp on the domoticz host as an option long ago because devices would not always show up. If you find a reliable way then I will reconsider. I do appreciate your effort to find other options!

I did some tests with <trackerhost>#ssh=arp -a. But you are right. arp is not very reliable. Sometimes you see a change within seconds and sometimes it looks like nothing is changed. This is my current version of arp which is a part of net-tools

Code: Select all

pi@mcmellow-pi:~ $ arp --version
net-tools 2.10-alpha
+I18N +SELINUX
AF: (inet) +UNIX +INET +INET6 +IPX +AX25 +NETROM +X25 +ATALK +ECONET +ROSE -BLUETOOTH
HW: (ether) +ETHER +ARC +SLIP +PPP +TUNNEL -TR +AX25 +NETROM +X25 +FR +ROSE +ASH +SIT +FDDI +HIPPI +HDLC/LAPB +EUI64

But still searching for other options in my local network.

[EscApe]

@McMelloW

But still searching for other options in my local network.

I think the http approach is still your best bet for reliable and (near) real-time info. If it is possible for homeassistant then it should be possible as a tracker for this plugin.

[McMelloW]

@McMelloW

But still searching for other options in my local network.

I think the http approach is still your best bet for reliable and (near) real-time info. If it is possible for homeassistant then it should be possible as a tracker for this plugin.

Yes that must be possible, but I am not a professional programmer. Scripting is not new for me, but python is very new. I have to learn the specific Python details and scripting run flow, etc.
The http approach is rather difficult with the experia boxes. You have to login remote and scrape the webpage for the correct information. The correct webpage has two frames and I only need the contents of 1 frame. I did get the total interactive in python and it was a very large part of text.

My thought of looking into a way to let my Raspberry check for connected devices on my network. The Raspberry has a direct LAN-cable to the router.

Do you have some info about this line in ssh_brctl.py How does this work?

Code: Select all

self.trackerscript = "brctl showmacs br0"

[EscApe]

The http approach is rather difficult with the experia boxes. You have to login remote and scrape the webpage for the correct information. The correct webpage has two frames and I only need the contents of 1 frame. I did get the total interactive in python and it was a very large part of text.

You can probably just return the entire html page to the plugin. The plugin should sort out the Mac-addresses without any intelligent scraping required.

Do you have some info about this line in ssh_brctl.py How does this work?
CODE: SELECT ALL
self.trackerscript = "brctl showmacs br0"

That's comparable to arp. It queries the ethernet bridge on the router over ssh.

[fireport]

Good mornig to all,
I installed this plugin in my domoticz installation, but the detection happens only once after the first setup.
The plugin doesn't seem to run. I tried to change the schedule time without result.
The domoticz log is clear and doesn't report any errors.

[EscApe]

Good mornig to all,
I installed this plugin in my domoticz installation, but the detection happens only once after the first setup.
The plugin doesn't seem to run. I tried to change the schedule time without result.
The domoticz log is clear and doesn't report any errors.

Please include a debug log so we can see what’s going on (and what isn’t )

[fireport]

After a systemctl restart, the plugin started normally. But after that, I had another problem.
The Mac address list was very very partial (I have more than 30 devices) so the present detection was not very reliable.

Code: Select all

2020-03-13 10:15:28.138  (iDetect) 192.168.1.1 Timed poll starting like clockwork
2020-03-13 10:15:28.138  (iDetect) 192.168.1.1 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x7ff6a4040690>>
2020-03-13 10:15:28.138  (iDetect) 192.168.1.1 ====> SSH Fetching data using: /sbin/wlanconfig ath0 list
/sbin/wlanconfig ath1 list
/sbin/wlanconfig ath2 list
exit
2020-03-13 10:15:28.168  (iDetect) 192.168.1.1 ====> SSH returned (decoded):ADDR               AID CHAN TXRATE RXRATE RSSI MINRSSI MAXRSSI IDLE  TXSEQ  RXSEQ  CAPS        ACAPS     ERP    STATE MAXRATE(DOT11) HTCAPS ASSOCTIME    IEs   MODE                   PSMODE RXNSS TXNSS
3c:71:bf:2c:8e:56    2   11  57M     48M   32      26      34    1      0   65535   EPS         0          f              0               P 35:23:39  RSN WME IEEE80211_MODE_11NG_HT20   0        1     1
b4:e6:2d:5a:51:df    1   11  72M     54M   32      27      36    2      0   65535   EPS         0          f              0               P 35:22:51  RSN WME IEEE80211_MODE_11NG_HT20   0        1     1
b0:41:1d:5c:c4:47    3   11 130M    117M   39      30      40    0      0   65535    EP         0          f              0               P 23:03:07  RSN WME IEEE80211_MODE_11NG_HT20   0        2     2
b4:e6:2d:3c:1a:1d    4   11  72M     54M   44      43      46    0      0   65535   EPS         0          d              0               P 00:08:02  RSN IEEE80211_MODE_11NG_HT20   1        1     1
ADDR               AID CHAN TXRATE RXRATE RSSI MINRSSI MAXRSSI IDLE  TXSEQ  RXSEQ  CAPS        ACAPS     ERP    STATE MAXRATE(DOT11) HTCAPS ASSOCTIME    IEs   MODE                   PSMODE RXNSS TXNSS
16:7b:44:d3:08:8a    1  100 180M    240M   28       0      41    0      0   65535   EPs         0          b              0           AWPSM 51:13:46  RSN WME IEEE80211_MODE_11AC_VHT80   0        2     2
b8:27:eb:1f:a0:a9    4  100 150M    180M   27      20      37    0      0   65535    EP         0          b              0            AWQS 43:07:42  RSN WME IEEE80211_MODE_11NA_HT40   0        1     1
58:20:59:70:6e:26    2  100 433M    292M   21      12      40    1      0   65535    EP         0          b              0           AWPGS 01:36:26  RSN WME IEEE80211_MODE_11AC_VHT80   1        1     1
68:ec:c5:4b:bb:d0    3  100 433M    292M   34      23      40    0      0   65535   EPs         0          b              0           AWQSM 01:14:37  RSN WME IEEE80211_MODE_11AC_VHT80   0        1     1
ADDR               AID CHAN TXRATE RXRATE RSSI MINRSSI MAXRSSI IDLE  TXSEQ  RXSEQ  CAPS        ACAPS     ERP    STATE MAXRATE(DOT11) HTCAPS ASSOCTIME    IEs   MODE                   PSMODE RXNSS TXNSS
c8:7b:5b:b0:0d:16    1   40  65M     19M   24      15      52    2      0   65535    EP         0          b              0               P 40:11:29  RSN WME IEEE80211_MODE_11NA_HT20   1        1     1
c4:95:00:24:28:bb    2   40 585M    526M   37      24      45    0      0   65535    EP         0          b              0           AWPSM 39:02:59  RSN WME IEEE80211_MODE_11AC_VHT80   0        2     2
b0:41:1d:5c:d8:d1    4   40 300M    162M   23       0      30    0      0   65535    EP         0          b              0             WPS 38:54:21  RSN WME IEEE80211_MODE_11NA_HT40   0        2     2
74:40:bb:13:7b:cf    3   40 325M    263M   19      15      39    2      0   65535   EPs         0          b              0            WPSM 01:26:11  RSN WME IEEE80211_MODE_11AC_VHT80   0        1     1

2020-03-13 10:15:28.169  (iDetect) 192.168.1.1 ====> SSH session took 30 milliseconds.
2020-03-13 10:15:28.169  (iDetect) Inbound data from: 192.168.1.1 containing ['3C:71:BF:2C:8E:56', 'B4:E6:2D:5A:51:DF', 'B0:41:1D:5C:C4:47', 'B4:E6:2D:3C:1A:1D', '16:7B:44:D3:08:8A', 'B8:27:EB:1F:A0:A9', '58:20:59:70:6E:26', '68:EC:C5:4B:BB:D0', 'C8:7B:5B:B0:0D:16', 'C4:95:00:24:28:BB', 'B0:41:1D:5C:D8:D1', '74:40:BB:13:7B:CF']
2020-03-13 10:15:28.169  (iDetect) 1 devices are present (excluding ignored devices)

The brct showmacs br0 seems to show the perfect output

Code: Select all

admin@Lyra:/tmp/home/root# brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     04:92:26:7b:48:70       no                 9.50
  1     08:31:8b:38:0d:8c       no                 1.46
  1     10:7b:44:d2:7a:ab       no                 0.68
  2     10:7b:44:d2:f1:60       yes                0.00
  3     10:7b:44:d2:f1:62       yes                0.00
  1     10:7b:44:d2:f1:63       yes                0.00
  4     10:7b:44:d2:f1:64       yes                0.00
  3     10:7b:44:d3:08:8b       no                 0.20
  2     3c:71:bf:2c:8e:56       no                14.41
  1     52:54:00:f7:0e:49       no                 5.03
  3     58:20:59:70:6e:26       no                48.36
  1     5c:b1:3e:eb:c1:41       no                39.86
  1     5c:cf:7f:20:ce:8d       no                 5.01
  3     60:01:94:73:ea:e9       no                 0.20
  3     68:ec:c5:4b:bb:d0       no                 0.21
  1     68:f7:28:7a:46:66       no                 0.08
  1     70:85:c2:b4:53:ca       no                 0.00
  4     74:40:bb:13:7b:cf       no                 0.20
  3     84:f3:eb:4a:df:c2       no                 0.17
  1     84:f3:eb:b0:c0:9d       no                69.31
  3     a4:cf:12:b5:e9:d7       no                 0.05
  3     b0:41:1d:52:df:b6       no                 0.00
  2     b0:41:1d:5c:c4:47       no                 0.00
  4     b0:41:1d:5c:d8:d1       no                 0.01
  2     b4:e6:2d:3c:1a:1d       no                 4.72
  2     b4:e6:2d:5a:51:df       no                 3.56
  3     b8:27:eb:1f:a0:a9       no                 0.33
  4     c4:95:00:24:28:bb       no                 0.02
  1     c4:95:00:5e:9c:4c       no                 0.02
  4     c8:7b:5b:b0:0d:16       no               197.79
  1     d0:50:99:36:9a:bb       no                 0.00
  1     ec:e5:12:12:54:30       no                 8.57

so I switched the tracker command to 192.168.1.1#ssh=brctl showmacs br0
and now detection it's ok.
For further reference: I have an ASUS Lyra mesh
Thanks to all

[EscApe]

After a systemctl restart, the plugin started normally. But after that, I had another problem.
The Mac address list was very very partial (I have more than 30 devices) so the present detection was not very reliable.

I would not be surprised if running wlanconfig on one node of a mesh will only return the devices directly connected to that nodes wifi chipset. In that case you would only see the other directly connected nodes and end-devices. Since I don not use a mesh setup I can only speculate. The most reliable methods to query a mesh will probably be (in descending order of preference):
- Write a tracker for a vendor specific api for the mesh
- Query each node with hardware specific commands (like the plugin does be default)
- Use a command like brctl that looks beyond the wireless interfaces (less responsive/reliable when a device goes offline/becomes 'absent')

[fireport]

EscAPe you are right when you think that wlanconfig atX lists only the devices directly connected to node.
Looking around router html pages I found that web interface call /usr/sbin/networkmap. The command source code https://github.com/RMerl/asuswrt-merlin ... tworkmap.c report that this generates/tmp/clientlist.json that contains all the information's that I need.
So I changed the tracker command to 192.168.1.1#ssh=cat /tmp/clientlist.json and this is the working output.
Just a question. Why did you say that brctl is less responsive/reliable when a device goes offline/becomes 'absent' ?

Code: Select all

 2020-03-13 16:13:02.240 (iDetect) 192.168.1.1 ====> SSH returned (decoded):{"10:7B:44:D3:08:8B":{"2G":{"60:01:94:73:EA:E9":{"ip":"","rssi":"-45"},"84:F3:EB:4A:DF:C2":{"ip":"","rssi":"-68"},"A4:CF:12:B5:E9:D7":{"ip":"","rssi":"-53"},"10:44:00:1C:BF:02":{"ip":"","rssi":"-55"}},"5G1":{"B0:41:1D:52:DF:B6":{"ip":"","rssi":"-75"},"74:40:BB:13:7B:CF":{"ip":"","rssi":"-61"}}},"10:7B:44:D2:7A:AB":{"2G":{"2C:F4:32:5B:36:56":{"ip":"","rssi":"-48"},"2C:F4:32:5B:45:CA":{"ip":"","rssi":"-57"},"60:01:94:4E:F5:2C":{"ip":"","rssi":"-53"},"5C:CF:7F:20:CE:8D":{"ip":"","rssi":"-65"},"84:F3:EB:B0:C0:9D":{"ip":"","rssi":"-73"},"5C:CF:7F:20:D0:10":{"ip":"","rssi":"-60"}},"5G":{"68:EC:C5:4B:BB:D0":{"ip":"","rssi":"-77"}},"5G1":{"C4:95:00:5E:9C:4C":{"ip":"","rssi":"-64"},"04:92:26:7B:48:70":{"ip":"","rssi":"-44"},"08:31:8B:38:0D:8C":{"ip":"","rssi":"-69"}},"wired_mac":{"10:44:00:1C:BF:02":{"ip":""},"10:7B:44:D2:F1:63":{"ip":""},"10:7B:44:D3:08:8B":{"ip":""},"38:8C:50:28:7A:6C":{"ip":""},"3C:71:BF:2C:8E:56":{"ip":""},"52:54:00:F7:0E:49":{"ip":""},"5C:B1:3E:EB:C1:41":{"ip":""},"68:F7:28:7A:46:66":{"ip":""},"70:85:C2:B4:53:CA":{"ip":""},"74:40:BB:13:7B:CF":{"ip":""},"B4:E6:2D:5A:51:DF":{"ip":""},"B8:27:EB:1F:A0:A9":{"ip":""},"C4:95:00:24:28:BB":{"ip":""}}},"10:7B:44:D2:F1:63":{"2G":{"3C:71:BF:2C:8E:56":{"ip":"192.168.1.246","rssi":"-66"},"B4:E6:2D:5A:51:DF":{"ip":"192.168.1.79","rssi":"-63"},"B0:41:1D:5C:C4:47":{"ip":"192.168.1.5","rssi":"-61"},"B4:E6:2D:3C:1A:1D":{"ip":"192.168.1.196","rssi":"-53"}},"5G":{"16:7B:44:D3:08:8A":{"ip":"","rssi":"-72"},"B8:27:EB:1F:A0:A9":{"ip":"192.168.1.11","rssi":"-64"},"58:20:59:70:6E:26":{"ip":"192.168.1.9","rssi":"-64"}},"5G1":{"C8:7B:5B:B0:0D:16":{"ip":"192.168.1.33","rssi":"-72"},"C4:95:00:24:28:BB":{"ip":"192.168.1.88","rssi":"-57"},"B0:41:1D:5C:D8:D1":{"ip":"192.168.1.4","rssi":"-72"}},"wired_mac":{"04:92:26:7B:48:70":{"ip":"192.168.1.10"},"08:31:8B:38:0D:8C":{"ip":"192.168.1.187"},"10:7B:44:D2:7A:AB":{"ip":"192.168.1.39"},"2C:F4:32:5B:36:56":{"ip":"192.168.1.197"},"38:8C:50:28:7A:6C":{"ip":"192.168.1.131"},"52:54:00:F7:0E:49":{"ip":"192.168.1.140"},"5C:B1:3E:EB:C1:41":{"ip":"192.168.1.134"},"68:EC:C5:4B:BB:D0":{"ip":"192.168.1.35"},"68:F7:28:7A:46:66":{"ip":"192.168.1.107"},"70:85:C2:B4:53:CA":{"ip":"192.168.1.2"},"C4:95:00:5E:9C:4C":{"ip":"192.168.1.167"},"D0:50:99:36:9A:BB":{"ip":"192.168.1.7"},"EC:E5:12:12:54:30":{"ip":"192.168.1.101"}}}}
2020-03-13 16:13:02.240 (iDetect) 192.168.1.1 ====> SSH session took 20 milliseconds.
2020-03-13 16:13:02.240 (iDetect) Inbound data from: 192.168.1.1 containing ['10:7B:44:D3:08:8B', '60:01:94:73:EA:E9', '84:F3:EB:4A:DF:C2', 'A4:CF:12:B5:E9:D7', '10:44:00:1C:BF:02', 'B0:41:1D:52:DF:B6', '74:40:BB:13:7B:CF', '10:7B:44:D2:7A:AB', '2C:F4:32:5B:36:56', '2C:F4:32:5B:45:CA', '60:01:94:4E:F5:2C', '5C:CF:7F:20:CE:8D', '84:F3:EB:B0:C0:9D', '5C:CF:7F:20:D0:10', '68:EC:C5:4B:BB:D0', 'C4:95:00:5E:9C:4C', '04:92:26:7B:48:70', '08:31:8B:38:0D:8C', '10:44:00:1C:BF:02', '10:7B:44:D2:F1:63', '10:7B:44:D3:08:8B', '38:8C:50:28:7A:6C', '3C:71:BF:2C:8E:56', '52:54:00:F7:0E:49', '5C:B1:3E:EB:C1:41', '68:F7:28:7A:46:66', '70:85:C2:B4:53:CA', '74:40:BB:13:7B:CF', 'B4:E6:2D:5A:51:DF', 'B8:27:EB:1F:A0:A9', 'C4:95:00:24:28:BB', '10:7B:44:D2:F1:63', '3C:71:BF:2C:8E:56', 'B4:E6:2D:5A:51:DF', 'B0:41:1D:5C:C4:47', 'B4:E6:2D:3C:1A:1D', '16:7B:44:D3:08:8A', 'B8:27:EB:1F:A0:A9', '58:20:59:70:6E:26', 'C8:7B:5B:B0:0D:16', 'C4:95:00:24:28:BB', 'B0:41:1D:5C:D8:D1', '04:92:26:7B:48:70', '08:31:8B:38:0D:8C', '10:7B:44:D2:7A:AB', '2C:F4:32:5B:36:56', '38:8C:50:28:7A:6C', '52:54:00:F7:0E:49', '5C:B1:3E:EB:C1:41', '68:EC:C5:4B:BB:D0', '68:F7:28:7A:46:66', '70:85:C2:B4:53:CA', 'C4:95:00:5E:9C:4C', 'D0:50:99:36:9A:BB', 'EC:E5:12:12:54:30']
2020-03-13 16:13:02.241 (iDetect) 3 devices are present (excluding ignored devices) 

Just a question. Why did you say that brctl is less responsive/reliable when a device goes offline/becomes 'absent' ?

Thank you

[EscApe]

EscAPe you are right when you think that wlanconfig atX lists only the devices directly connected to node.
Looking around router html pages I found that web interface call /usr/sbin/networkmap. The command source code https://github.com/RMerl/asuswrt-merlin ... tworkmap.c report that this generates/tmp/clientlist.json that contains all the information's that I need.
So I changed the tracker command to 192.168.1.1#ssh=cat /tmp/clientlist.json and this is the working output.
Just a question. Why did you say that brctl is less responsive/reliable when a device goes offline/becomes 'absent' ?

<<>>

Just a question. Why did you say that brctl is less responsive/reliable when a device goes offline/becomes 'absent' ?

Thank you

Great find! The clientlist.json approach seems to work fine, even on a single Asus router. The information seems to be accurate, but it does have a downside. It seems the file is repopulated at a fixed interval, so there will be a delay before a new connection shows up and before a disconnected device disappears from the list. I did some (limited) testing and is was around 15-20 seconds (maybe the interval is 30 seconds?). So the maximum delay before presence is detected will be Asus interval + plugin poll interval.

Brctl will see presence immediately, but there is a delay before absence is detected. On my Asus router it takes 5 minutes ('aging timer' on brctl), but during my initial testing (long ago) it would sometimes keep Mac addresses for hours(!) That's what I meant by "less responsive/reliable"

Querying the chipset using hardware specific tools like wl/wlanconfig will always be instant, so you only need to take the poll interval into account.

I hope this gives you some idea of why I prefer one approach over another. For me the detection of new connections (presence) is the most important, because I use it to automate stuff when I get home. I don't mind a delay in detecting absence, but 5 minutes to (occasionally) hours is not great.

[McMelloW]

@EscApe.
There is no way to get in a ExperiaBox V8. Telfort has closed everything up. So, I do throw in the towe and I have to stick to #type=dummy and a ping.
My smartphones has a fixed ip-address setup in the router.

Does somebody has an example dzVent or Lua script what you do when the Anyone Home device changes?

[EscApe]

@EscApe.
There is no way to get in a ExperiaBox V8. Telfort has closed everything up. So, I do throw in the towe and I have to stick to #type=dummy and a ping.
My smartphones has a fixed ip-address setup in the router.

Does somebody has an example dzVent or Lua script what you do when the Anyone Home device changes?

All devices created by the plugin are on/off switches. They can be used in scripts like any other on/off switch.

[gikah]

Hi. At first - thank you for making such complex and professional addon.
It works for me very well when using routeros tracker.

The problem is, that Mikrotik router works in my home with AirPort Express (1st gen) acting as AP to cover all rooms with wifi.
So when I'm using routeros tracker it can't see my phone connected to AP. My first solution for this was change of tracker to routeros-arp (with it all my phones, even connected to airport are visible) but that method as we know is not precise, sometimes my phone stays in online status for 20 minutes.

My second solution - I activated ssh on Airport Express, iDevice plugin can connect and take data from it but (for my knowledge) only using old arp method. The biggest issue is that AP keeping arp table all the time(?), even if my phone is disconnected from it for few hours.

So I can't use two trackers (one from mikrotik and second from AirPort) because it shows my phone online 'all the time'.

Why I'm writing this? Maybe someone here knew a way to force AirPort Express to work with other - more precise tracker? AirPort runs on NetBSD so maybe there is a chance for some software tweaks?

[Melotron]

Greetings.

I have installed a new router a home, a AX88U and reinstalled the script.
But now I'am getting a ssh error.

Code: Select all

 2020-03-21 09:33:17.307 Status: (iDetect) Started.
2020-03-21 09:33:17.596 Status: (iDetect) Entering work loop.
2020-03-21 09:33:17.599 Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-03-21 09:33:19.991 (iDetect) Debug logging mask set to: PYTHON
2020-03-21 09:33:19.991 (iDetect) Operation system is: linux
2020-03-21 09:33:20.020 (iDetect) The OS user profile running domoticz is: root
2020-03-21 09:33:20.020 (iDetect) Parsing user and optional keyfile from:admin
2020-03-21 09:33:20.021 (iDetect) start get or create wireless device
2020-03-21 09:33:20.021 (iDetect) phone1 monitor tag_id:A4:50:46:E2:FD:63, domoticz unit:2
2020-03-21 09:33:20.022 (iDetect) start get or create wireless device
2020-03-21 09:33:20.022 (iDetect) phone2 monitor tag_id:AA:D8:AC:5C:AB:CE, domoticz unit:3
2020-03-21 09:33:20.022 (iDetect) start get or create wireless device
2020-03-21 09:33:20.022 (iDetect) phone3 monitor tag_id:A8:DB:03:5E:23:83, domoticz unit:4
2020-03-21 09:33:20.022 (iDetect) Monitoring {'A4:50:46:E2:FD:63': <plugin.tag_device object at 0x64ec6bd0>, 'AA:D8:AC:5C:AB:CE': <plugin.tag_device object at 0x64f62070>, 'A8:DB:03:5E:23:83': <plugin.tag_device object at 0x64f62930>} for presence.
2020-03-21 09:33:20.022 (iDetect) Tracker configuration:192.168.1.1
2020-03-21 09:33:20.023 (iDetect) tracker:192.168.1.1
2020-03-21 09:33:20.023 (iDetect) options:
2020-03-21 09:33:20.027 (iDetect) 192.168.1.1 Tracker is of the ssh kind
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH Fetching data using:
2020-03-21 09:33:20.028 export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH
2020-03-21 09:33:20.028 type wl
2020-03-21 09:33:20.028 type iwinfo
2020-03-21 09:33:20.028 type wlanconfig
2020-03-21 09:33:20.028 type wl_atheros
2020-03-21 09:33:20.028 type qcsapi_sockrpc
2020-03-21 09:33:20.028 type ip
2020-03-21 09:33:20.028 type brctl
2020-03-21 09:33:20.028 type arp
2020-03-21 09:33:20.028 [ -f /proc/net/arp ] && type cat
2020-03-21 09:33:20.028 echo 0
2020-03-21 09:33:20.028 exit
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH not connected ... connecting
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH start connect
2020-03-21 09:33:20.246 (iDetect) 192.168.1.1 ====> SSH returned (decoded):wl is /usr/sbin/wl
2020-03-21 09:33:20.246 iwinfo: not found
2020-03-21 09:33:20.246 wlanconfig: not found
2020-03-21 09:33:20.246 wl_atheros: not found
2020-03-21 09:33:20.246 qcsapi_sockrpc: not found
2020-03-21 09:33:20.246 ip is /usr/sbin/ip
2020-03-21 09:33:20.246 brctl is /bin/brctl
2020-03-21 09:33:20.246 arp is /sbin/arp
2020-03-21 09:33:20.246 cat is /bin/cat
2020-03-21 09:33:20.246 0
2020-03-21 09:33:20.246
2020-03-21 09:33:20.246 (iDetect) 192.168.1.1 ====> SSH session took 218 milliseconds.
2020-03-21 09:33:20.248 (iDetect) Available commands on 192.168.1.1:{'wl': '/usr/sbin/wl', 'arp': '/sbin/arp', 'cat': '/bin/cat', 'brctl': '/bin/brctl', 'ip': '/usr/sbin/ip'}
2020-03-21 09:33:20.249 (iDetect) 192.168.1.1 ====> SSH Fetching data using:
2020-03-21 09:33:20.249 for iface in $(ifconfig | cut -d ' ' -f1| tr ':' '
2020-03-21 09:33:20.249 ' | grep -E '^eth|^wlan|^wl');do
2020-03-21 09:33:20.249 /usr/sbin/wl -i $iface assoclist > /dev/null 2>&1 && printf "~$iface"
2020-03-21 09:33:20.249 done
2020-03-21 09:33:20.249 exit
2020-03-21 09:33:20.304 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:20.304 (iDetect) 192.168.1.1 ====> SSH session took 55 milliseconds.
2020-03-21 09:33:20.308 (iDetect) 192.168.1.1 Available interfaces for wl: ['']
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 Prepared to poll using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:20.309 exit
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 tracker will autodetect ssh cli
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x64ebb2d0>>
2020-03-21 09:33:20.309 (iDetect) Trackers initialized as:{'192.168.1.1': <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>}
2020-03-21 09:33:20.309 (iDetect) Plugin initialization done.
2020-03-21 09:33:20.226 Status: (iDetect) 192.168.1.1 ====> SSH connection established
2020-03-21 09:33:20.309 Status: (iDetect) Tracker activated:192.168.1.1 port 22, user: admin, type: default and options: {}
2020-03-21 09:33:20.304 Error: (iDetect) 192.168.1.1 ====> SSH returned error:sh: ifconfig: not found
2020-03-21 09:33:20.304
2020-03-21 09:33:29.828 (iDetect) onHeartbeat called
2020-03-21 09:33:29.828 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:34.313 (RFXCOM) Temp + Humidity (Sovrum Temp)
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 Timed poll starting like clockwork
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>>
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 ====> SSH Fetching data using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:35.028 exit
2020-03-21 09:33:35.046 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:35.046 (iDetect) 192.168.1.1 ====> SSH session took 17 milliseconds.
2020-03-21 09:33:35.056 (iDetect) Inbound data from: 192.168.1.1 containing []
2020-03-21 09:33:35.056 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:35.046 Error: (iDetect) 192.168.1.1 ====> SSH returned error:type -h <optional command> for help
2020-03-21 09:33:35.046
2020-03-21 09:33:39.847 (iDetect) onHeartbeat called
2020-03-21 09:33:39.848 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:49.821 (iDetect) onHeartbeat called
2020-03-21 09:33:49.821 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:50.059 (iDetect) 192.168.1.1 Timed poll starting like clockwork
2020-03-21 09:33:50.060 (iDetect) 192.168.1.1 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>>
2020-03-21 09:33:50.060 (iDetect) 192.168.1.1 ====> SSH Fetching data using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:50.060 exit
2020-03-21 09:33:50.076 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:50.077 (iDetect) 192.168.1.1 ====> SSH session took 16 milliseconds.
2020-03-21 09:33:50.079 (iDetect) Inbound data from: 192.168.1.1 containing []
2020-03-21 09:33:50.080 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:50.076 Error: (iDetect) 192.168.1.1 ====> SSH returned error:type -h <optional command> for help
2020-03-21 09:33:50.076 
 

I have updated the ssh key on both sides.
I can ssh to the router with no problem from my raspberry and same from router to raspberry.
I also have a tried a custom tracker string:

Code: Select all

192.168.1.1#port=XXXX&type=routeros&interval=10&user=admin&password=X

Does it work on Asus default firmware, or do I need to update it to Merlins ??
Or do I need to change the routeros string or anything else ??

added Domoticz ver info:

Code: Select all

About Domoticz
Version: 4.10717
Build Hash: b38b49e5
Compile Date: 2019-05-09 13:04:08
dzVents Version: 2.4.19
Python Version: 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516]

//Magnus

[EscApe]

Greetings.

I have installed a new router a home, a AX88U and reinstalled the script.
But now I'am getting a ssh error.

Code: Select all

 2020-03-21 09:33:17.307 Status: (iDetect) Started.
2020-03-21 09:33:17.596 Status: (iDetect) Entering work loop.
2020-03-21 09:33:17.599 Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-03-21 09:33:19.991 (iDetect) Debug logging mask set to: PYTHON
2020-03-21 09:33:19.991 (iDetect) Operation system is: linux
2020-03-21 09:33:20.020 (iDetect) The OS user profile running domoticz is: root
2020-03-21 09:33:20.020 (iDetect) Parsing user and optional keyfile from:admin
2020-03-21 09:33:20.021 (iDetect) start get or create wireless device
2020-03-21 09:33:20.021 (iDetect) phone1 monitor tag_id:A4:50:46:E2:FD:63, domoticz unit:2
2020-03-21 09:33:20.022 (iDetect) start get or create wireless device
2020-03-21 09:33:20.022 (iDetect) phone2 monitor tag_id:AA:D8:AC:5C:AB:CE, domoticz unit:3
2020-03-21 09:33:20.022 (iDetect) start get or create wireless device
2020-03-21 09:33:20.022 (iDetect) phone3 monitor tag_id:A8:DB:03:5E:23:83, domoticz unit:4
2020-03-21 09:33:20.022 (iDetect) Monitoring {'A4:50:46:E2:FD:63': <plugin.tag_device object at 0x64ec6bd0>, 'AA:D8:AC:5C:AB:CE': <plugin.tag_device object at 0x64f62070>, 'A8:DB:03:5E:23:83': <plugin.tag_device object at 0x64f62930>} for presence.
2020-03-21 09:33:20.022 (iDetect) Tracker configuration:192.168.1.1
2020-03-21 09:33:20.023 (iDetect) tracker:192.168.1.1
2020-03-21 09:33:20.023 (iDetect) options:
2020-03-21 09:33:20.027 (iDetect) 192.168.1.1 Tracker is of the ssh kind
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH Fetching data using:
2020-03-21 09:33:20.028 export PATH=/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:$PATH
2020-03-21 09:33:20.028 type wl
2020-03-21 09:33:20.028 type iwinfo
2020-03-21 09:33:20.028 type wlanconfig
2020-03-21 09:33:20.028 type wl_atheros
2020-03-21 09:33:20.028 type qcsapi_sockrpc
2020-03-21 09:33:20.028 type ip
2020-03-21 09:33:20.028 type brctl
2020-03-21 09:33:20.028 type arp
2020-03-21 09:33:20.028 [ -f /proc/net/arp ] && type cat
2020-03-21 09:33:20.028 echo 0
2020-03-21 09:33:20.028 exit
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH not connected ... connecting
2020-03-21 09:33:20.028 (iDetect) 192.168.1.1 ====> SSH start connect
2020-03-21 09:33:20.246 (iDetect) 192.168.1.1 ====> SSH returned (decoded):wl is /usr/sbin/wl
2020-03-21 09:33:20.246 iwinfo: not found
2020-03-21 09:33:20.246 wlanconfig: not found
2020-03-21 09:33:20.246 wl_atheros: not found
2020-03-21 09:33:20.246 qcsapi_sockrpc: not found
2020-03-21 09:33:20.246 ip is /usr/sbin/ip
2020-03-21 09:33:20.246 brctl is /bin/brctl
2020-03-21 09:33:20.246 arp is /sbin/arp
2020-03-21 09:33:20.246 cat is /bin/cat
2020-03-21 09:33:20.246 0
2020-03-21 09:33:20.246
2020-03-21 09:33:20.246 (iDetect) 192.168.1.1 ====> SSH session took 218 milliseconds.
2020-03-21 09:33:20.248 (iDetect) Available commands on 192.168.1.1:{'wl': '/usr/sbin/wl', 'arp': '/sbin/arp', 'cat': '/bin/cat', 'brctl': '/bin/brctl', 'ip': '/usr/sbin/ip'}
2020-03-21 09:33:20.249 (iDetect) 192.168.1.1 ====> SSH Fetching data using:
2020-03-21 09:33:20.249 for iface in $(ifconfig | cut -d ' ' -f1| tr ':' '
2020-03-21 09:33:20.249 ' | grep -E '^eth|^wlan|^wl');do
2020-03-21 09:33:20.249 /usr/sbin/wl -i $iface assoclist > /dev/null 2>&1 && printf "~$iface"
2020-03-21 09:33:20.249 done
2020-03-21 09:33:20.249 exit
2020-03-21 09:33:20.304 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:20.304 (iDetect) 192.168.1.1 ====> SSH session took 55 milliseconds.
2020-03-21 09:33:20.308 (iDetect) 192.168.1.1 Available interfaces for wl: ['']
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 Prepared to poll using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:20.309 exit
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 tracker will autodetect ssh cli
2020-03-21 09:33:20.309 (iDetect) 192.168.1.1 Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x64ebb2d0>>
2020-03-21 09:33:20.309 (iDetect) Trackers initialized as:{'192.168.1.1': <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>}
2020-03-21 09:33:20.309 (iDetect) Plugin initialization done.
2020-03-21 09:33:20.226 Status: (iDetect) 192.168.1.1 ====> SSH connection established
2020-03-21 09:33:20.309 Status: (iDetect) Tracker activated:192.168.1.1 port 22, user: admin, type: default and options: {}
2020-03-21 09:33:20.304 Error: (iDetect) 192.168.1.1 ====> SSH returned error:sh: ifconfig: not found
2020-03-21 09:33:20.304
2020-03-21 09:33:29.828 (iDetect) onHeartbeat called
2020-03-21 09:33:29.828 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:34.313 (RFXCOM) Temp + Humidity (Sovrum Temp)
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 Timed poll starting like clockwork
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>>
2020-03-21 09:33:35.028 (iDetect) 192.168.1.1 ====> SSH Fetching data using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:35.028 exit
2020-03-21 09:33:35.046 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:35.046 (iDetect) 192.168.1.1 ====> SSH session took 17 milliseconds.
2020-03-21 09:33:35.056 (iDetect) Inbound data from: 192.168.1.1 containing []
2020-03-21 09:33:35.056 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:35.046 Error: (iDetect) 192.168.1.1 ====> SSH returned error:type -h <optional command> for help
2020-03-21 09:33:35.046
2020-03-21 09:33:39.847 (iDetect) onHeartbeat called
2020-03-21 09:33:39.848 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:49.821 (iDetect) onHeartbeat called
2020-03-21 09:33:49.821 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:50.059 (iDetect) 192.168.1.1 Timed poll starting like clockwork
2020-03-21 09:33:50.060 (iDetect) 192.168.1.1 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x64f620b0>>
2020-03-21 09:33:50.060 (iDetect) 192.168.1.1 ====> SSH Fetching data using: /usr/sbin/wl -i assoclist
2020-03-21 09:33:50.060 exit
2020-03-21 09:33:50.076 (iDetect) 192.168.1.1 ====> SSH returned (decoded):
2020-03-21 09:33:50.077 (iDetect) 192.168.1.1 ====> SSH session took 16 milliseconds.
2020-03-21 09:33:50.079 (iDetect) Inbound data from: 192.168.1.1 containing []
2020-03-21 09:33:50.080 (iDetect) 0 devices are present (excluding ignored devices)
2020-03-21 09:33:50.076 Error: (iDetect) 192.168.1.1 ====> SSH returned error:type -h <optional command> for help
2020-03-21 09:33:50.076 
 

I have updated the ssh key on both sides.
I can ssh to the router with no problem from my raspberry and same from router to raspberry.
I also have a tried a custom tracker string:

Code: Select all

192.168.1.1#port=XXXX&type=routeros&interval=10&user=admin&password=X

Does it work on Asus default firmware, or do I need to update it to Merlins ??
Or do I need to change the routeros string or anything else ??

added Domoticz ver info:

Code: Select all

About Domoticz
Version: 4.10717
Build Hash: b38b49e5
Compile Date: 2019-05-09 13:04:08
dzVents Version: 2.4.19
Python Version: 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516]

//Magnus

It looks like ssh is working just fine and the autodetection does find a suitable command to use on your router (wl). However, it does not find an interface te use the command on. Maybe the ax88u has some unusual interface names that I need to add to the plugin?

Your custom tracker string is using the routeros tracker for an ASUS router. ASUS is not routeros. The plugin should work with your ASUS router and you do not need the Merlin firmware for that (but the Merlin firmware is still great for other reasons).

[Melotron]

It looks like ssh is working just fine and the autodetection does find a suitable command to use on your router (wl). However, it does not find an interface te use the command on. Maybe the ax88u has some unusual interface names that I need to add to the plugin?

Your custom tracker string is using the routeros tracker for an ASUS router. ASUS is not routeros. The plugin should work with your ASUS router and you do not need the Merlin firmware for that (but the Merlin firmware is still great for other reasons).

I see the phones with a arp on the router, but they all are on a same interface as the rest of the devices.

Code: Select all

Phones:
? (192.168.1.65) at a8:db:03:5e:23:83 [ether]  on br0
? (192.168.1.246) at aa:d8:ac:5c:ab:ce [ether]  on br0
? (192.168.1.102) at a4:50:46:e2:fd:63 [ether]  on br0

Domoticz (wired)
? (192.168.1.101) at b8:27:eb:ad:ba:2e [ether]  on br0

What kind of logs or command output from the router do you need ?