Python plugin: Presence detection from wireless router

[EscApe]

@Franzie

Initially I saw everything was fine, phones are registered in and out. That works well!
But now I have in my log, every 50 minutes:

Error: (iDetect) 192.168.1.1 ====> SSH failed with exception:

I poll every 15 secs and grace period is 30 secs.

I get those messages too every once in a while. Don’t worry about them, the connection will recover. I should probably make this a warning instead of an error if it only occurs once. It might be worth investigating why you get them every 50 minutes (exactly?). Does your router or domoticz host run a cpu intensive task every 50 minutes or is there a high network load around that time?

[MrEnergy]

Hi All,

I'm long time Domoticz user, currently on latest stable on Pi3b. OS is Raspbian Buster.

Looking for a good presence detection, as ping is often failing to mobiles.

Until I've found your iDetect plugin, however the Experiabox has no SSH or telnet access. Only option to scrape from webpage.

I've been using a python script to login to my KPN Experia V100 Box and retreive ip and mac of connected devices.

Works like expected.

As i'm not a real programmer I don't know to to make a new tracker from my python script. Would It be possible to write a tracker for the ExperiaBox V10 using this code ?

Code: Select all

import requests
import re
from datetime import datetime, timezone
from hashlib import sha256
import xml.etree.ElementTree as ET
from collections import namedtuple

Device = namedtuple('Device', ['mac', 'name', 'ip'])

session = requests.Session()
ts = round(datetime.now(timezone.utc).timestamp() * 1000)
username = 'Admin'
pwd = '<<PASSWORD>>'

token_url = 'http://192.168.2.254/function_module/login_module/login_page/logintoken_lua.lua'
login_url = 'http://192.168.2.254'
data_url = 'http://192.168.2.254/common_page/home_AssociateDevs_lua.lua?AccessMode=WLAN&_={}'

session.get(login_url)
result = session.get(token_url)

session.post(login_url, data = {
        "Username": username,
        "Password":sha256((pwd + re.findall(r'\d+',result.text)[0]).encode('utf-8')).hexdigest(),
        "action": "login"
})

data_page = session.get(data_url)

result_root = ET.fromstring(data_page.text)

logout_url = 'http://192.168.2.254'
log_out_page = session.post(logout_url, data = {
        "IF_LogOff": 1,
        "IF_LanguageSwitch": "",
        "IF_ModeSwitch": ""
})

if result:
        device_list = result_root.find('OBJ_ACCESSDEV_ID')

        if device_list is None:
                print ('Geen devices gevonden')
                # return false

        results = []
        for device in device_list:
                keys = device.findall('ParaName')
                values = device.findall('ParaValue')

                result = {}
                for index, key in enumerate(keys):
                        value = values[index]

                        if key.text in ['HostName', 'MACAddress', 'IPAddress']:
                                result[key.text] = value.text or ''
                                print (key.text)
                                print (value.text)
                results.append(Device(result['MACAddress'].upper(),result['HostName'], result['IPAddress']))

        last_results = results

Lots of code from HASS module for Experiabox V10

results are in key.text and value.text.

If needed I can debug

[EscApe]

@MrEnergie

Some tips for an experiabox owner that wants to take on this challenge: you do not need to scrape the webpage. If the tracker returns the entire html result (as a string) to the base tracker it should filter all the Mac addresses from it. I would also connect once and only reconnect when needed, to reduce overhead. Logout when the plugin stops. For examples have a look in the tracker directory.

[Rusla]

It works good for me, but sometimes it shows wrong "out of home" status for a few seconds, so I get some kind of chattering on tracker switch and it sends me wrong notifications - could I avoid that somehow using plugin settings or dzvents?
Also I got "(Tracker) 192.168.1.1 Not (yet) ready for polling" every 15 sec (pulling settings of the plugin) in the log file - what does it mean? I would think router doesnt accept pings, but trackers with no problem (except sometime chattering I mentioned) works all the time

[EscApe]

Hi Rusla,

You can set a longer grace period and make sure there are several polls within the grace period. This should circumvent most temporary connection drops.

If you need some help with the error message please share a full debug log (enable debug mode) from the start of the plugin until the first few errors.

[vamp]

@vamp and other Mirotik owners...

If somebody use this plugin with Mikrotik and CAPsMAN it need to change this line to working:

from
CODE: SELECT ALL
custom['routeros']="interface wireless registration-table print"
to
CODE: SELECT ALL
custom['routeros']="caps-man registration-table print"

It only need to check the capsman manager not need to check all APs!


it would be great that this feature will be build in to the next release. (like "routeros-capsman")

Should the caps-man approach replace the existing method (is it better, faster, more reliable)? Or is this an additional method that might not work on every microtik router?

It is a additional method. Caps-man is not setting up default. the other method simply not work, if we use caps-man. So if somebody use caps-man, only way that use my modification.

[Rusla]

If you need some help with the error message please share a full debug log (enable debug mode) from the start of the plugin until the first few errors.

Hi
This is log from tracker start.

Code: Select all

2020-01-17 10:08:57.464 Status: (Tracker) Started.
2020-01-17 10:08:57.671 Status: (Tracker) Entering work loop.
2020-01-17 10:08:57.673 Status: (Tracker) Initialized version 2.0, author 'ESCape'
2020-01-17 10:08:59.711 (Tracker) Debug logging mask set to: PYTHON
2020-01-17 10:08:59.711 (Tracker) Operation system is: linux
2020-01-17 10:08:59.725 (Tracker) The OS user profile running domoticz is: root
2020-01-17 10:08:59.725 (Tracker) Parsing user and optional keyfile from:*******
2020-01-17 10:08:59.725 (Tracker) start get or create wireless device
2020-01-17 10:08:59.725 (Tracker) Ruslan monitor tag_id:192.168.1.120, domoticz unit:2
2020-01-17 10:08:59.725 (Tracker) Will use local ping to monitor presence for: 192.168.1.120
2020-01-17 10:08:59.726 (Tracker) local pinger Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x709a57f0>>
2020-01-17 10:08:59.726 (Tracker) local pinger Starting ping timer for 192.168.1.120
2020-01-17 10:08:59.727 (Tracker) start get or create wireless device
2020-01-17 10:08:59.727 (Tracker) Vika monitor tag_id:192.168.1.122, domoticz unit:3
2020-01-17 10:08:59.727 (Tracker) Will use local ping to monitor presence for: 192.168.1.122
2020-01-17 10:08:59.727 (Tracker) local pinger Starting ping timer for 192.168.1.122
2020-01-17 10:08:59.728 (Tracker) Monitoring {'192.168.1.120': <plugin.tag_device object at 0x6f7fd7d0>, '192.168.1.122': <plugin.tag_device object at 0x67c0bd90>} for presence.
2020-01-17 10:08:59.728 (Tracker) Tracker configuration:192.168.1.1
2020-01-17 10:08:59.728 (Tracker) tracker:192.168.1.1
2020-01-17 10:08:59.728 (Tracker) options:
2020-01-17 10:08:59.728 (Tracker) 192.168.1.1 Tracker is of the ssh kind
2020-01-17 10:08:59.729 (Tracker) 192.168.1.1 Could not retreive available commands
2020-01-17 10:08:59.729 (Tracker) 192.168.1.1 tracker will autodetect ssh cli
2020-01-17 10:08:59.729 (Tracker) 192.168.1.1 Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x709a57f0>>
2020-01-17 10:08:59.729 (Tracker) Trackers initialized as:{'192.168.1.1': <trackers.ssh_autodetect.ssh_autodetect object at 0x67c0be50>, 'local pinger': <trackers.ping_tracker.ping_tracker object at 0x67c01770>}
2020-01-17 10:08:59.729 (Tracker) Plugin initialization done.
2020-01-17 10:08:59.726 Status: (Tracker) Tracker activated: Local ping (for the ip address tags you configured)
2020-01-17 10:08:59.729 Status: (Tracker) Tracker activated:192.168.1.1 port 22, user: *******, type: default and options: {}
2020-01-17 10:08:59.729 Error: (Tracker) Missing paramiko module required for ssh. Install it using: sudo pip3 install paramiko
2020-01-17 10:09:09.245 (Tracker) onHeartbeat called
2020-01-17 10:09:09.245 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:09.734 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:09:10.142 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:09:10.146 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:09:10.147 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:10.754 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:09:19.263 (Tracker) onHeartbeat called
2020-01-17 10:09:19.263 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:19.746 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:09:20.592 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:09:20.594 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:09:20.594 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:21.780 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:09:29.281 (Tracker) onHeartbeat called
2020-01-17 10:09:29.282 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:29.726 (Tracker) local pinger Timed poll starting like clockwork
2020-01-17 10:09:29.747 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:09:31.619 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 256
2020-01-17 10:09:32.808 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:09:39.249 (Tracker) onHeartbeat called
2020-01-17 10:09:39.250 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:39.748 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:09:41.702 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:09:41.704 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:09:41.704 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:43.824 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:09:49.267 (Tracker) onHeartbeat called
2020-01-17 10:09:49.268 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:49.750 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:09:51.925 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:09:51.927 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:09:51.928 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:54.851 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:09:59.285 (Tracker) onHeartbeat called
2020-01-17 10:09:59.286 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:09:59.728 (Tracker) local pinger Timed poll starting like clockwork
2020-01-17 10:09:59.753 (Tracker) 192.168.1.1 Not (yet) ready for polling
2020-01-17 10:10:02.942 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 256
2020-01-17 10:10:05.877 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:10:09.253 (Tracker) onHeartbeat called
2020-01-17 10:10:09.253 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:10:13.095 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:10:13.098 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:10:13.099 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:10:16.904 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:10:19.271 (Tracker) onHeartbeat called
2020-01-17 10:10:19.272 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:10:23.568 (Tracker) Tried pinging tag: 192.168.1.122 --> error_level (0 means online): 0
2020-01-17 10:10:23.570 (Tracker) Inbound data from: local pinger containing ['192.168.1.122']
2020-01-17 10:10:23.570 (Tracker) 1 devices are present (excluding ignored devices)
2020-01-17 10:10:27.930 (Tracker) Tried pinging tag: 192.168.1.120 --> error_level (0 means online): 256
2020-01-17 10:10:29.289 (Tracker) onHeartbeat called
2020-01-17 10:10:29.290 (Tracker) 1 devices are present (excluding ignored devices)

It's 10 seconds poll and grace period is 40 seconds. There was 6 wrong notifications tonight. I can't post log till first error cause there no such old debug info in the log when I connect to domoticz few hours after, but I will try to stay connected to see that error message.

It also says "Missing paramiko module required for ssh. Install it using: sudo pip3 install paramiko", but it's installed already, this is response:

Code: Select all

pi@raspberrypi:~ $ sudo pip3 install paramiko
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: paramiko in /usr/local/lib/python3.6/site-packages (2.7.1)
Requirement already satisfied: cryptography>=2.5 in /usr/local/lib/python3.6/site-packages (from paramiko) (2.6.1)
Requirement already satisfied: bcrypt>=3.1.3 in /usr/local/lib/python3.6/site-packages (from paramiko) (3.1.7)
Requirement already satisfied: pynacl>=1.0.1 in /usr/local/lib/python3.6/site-packages (from paramiko) (1.3.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.6/site-packages (from cryptography>=2.5->paramiko) (1.12.3)
Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python3.6/site-packages (from cryptography>=2.5->paramiko) (1.12.0)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/local/lib/python3.6/site-packages (from cryptography>=2.5->paramiko) (0.24.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.5->paramiko) (2.19)

[EscApe]

@vamp

it would be great that this feature will be build in to the next release. (like "routeros-capsman")

It has been added to the newest version. Please let me know if there are any issues.

[EscApe]

@Rusla

It's 10 seconds poll and grace period is 40 seconds. There was 6 wrong notifications tonight.

This seems to be the case for the devices you are pinging. Not for wireless devices, since the router (tracker) never gets prepared for polling. Could your network or those devices be very busy around the times they fail? With ping devices you can set individual grace periods and intervals. If only on devices is unreliable you could increase its grace period.
Maybe I should increase the ping timeout(?) It is now just one second to minimize the performance impact on absent devices.

I can't post log till first error cause there no such old debug info in the log when I connect to domoticz few hours after, but I will try to stay connected to see that error message.

Domoticz can also log to a file. I think it is a command line argument.

It also says "Missing paramiko module required for ssh. Install it using: sudo pip3 install paramiko", but it's installed already, this is response:

Are you sure that Domoticz is using the same Python environment on your system? You can have several versions of python on one system with different libraries installed. Have a look at the Domoticz 'About' page. Does it show python 3.6? Ik might still be using the 3.5 (or different) environment.
The "(Tracker) 192.168.1.1 Not (yet) ready for polling" message indicates that router could not be configured for polling... probably because Paramiko is missing from the environment Domoticz is using.

[airmarshall]

Hi EscApe,

Awesome plugin, thank you very much.

I've just combined a homelab upgrade to edgerouter + unifi APs with an upgrade to the latest version of your script.

I've been through the thread for relevant info but still need some clarity.

Firsty, it looks like you've catered already for the setup so how did you design the script to work with edgerouters and/or unifi wi-fi?

My progress with the Edgerouter (Doesn't Work):

Tracker: 10.10.0.1
Username and password set globally. SSH connection verified working with putty.

Code: Select all

2020-01-20 21:59:58.897 Status: (iDetect) Started.
2020-01-20 21:59:59.158 Status: (iDetect) Entering work loop.
2020-01-20 21:59:59.161 Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-01-20 22:00:01.690 (iDetect) Debug logging mask set to: PYTHON
2020-01-20 22:00:01.690 (iDetect) Operation system is: linux
2020-01-20 22:00:01.718 (iDetect) The OS user profile running domoticz is: root
2020-01-20 22:00:01.718 (iDetect) Parsing user and optional keyfile from:admin
2020-01-20 22:00:01.719 (iDetect) start get or create wireless device
2020-01-20 22:00:01.719 (iDetect) XXX monitor tag_id:7C:D6:61:XX:0A:5D, domoticz unit:2
2020-01-20 22:00:01.720 (iDetect) start get or create wireless device
2020-01-20 22:00:01.720 (iDetect) XXX monitor tag_id:48:XX:XX:68:45:34, domoticz unit:3
2020-01-20 22:00:01.720 (iDetect) Monitoring {'7C:D6:61:XX:0A:5D': <plugin.tag_device object at 0x7074a990>, '48:2C:A0:68:45:34': <plugin.tag_device object at 0x707e1d30>} for presence.
2020-01-20 22:00:01.720 (iDetect) Tracker configuration:10.10.0.1
2020-01-20 22:00:01.720 (iDetect) tracker:10.10.0.1
2020-01-20 22:00:01.720 (iDetect) options:
2020-01-20 22:00:01.722 (iDetect) 10.10.0.1 Tracker is of the ssh kind
2020-01-20 22:00:01.723 (iDetect) 10.10.0.1 ====> SSH Fetching data using:
2020-01-20 22:00:01.723 (iDetect) 10.10.0.1 ====> SSH not connected ... connecting
2020-01-20 22:00:01.723 (iDetect) 10.10.0.1 ====> SSH start connect
2020-01-20 22:00:02.157 (iDetect) 10.10.0.1 Could not retreive available commands
2020-01-20 22:00:02.157 (iDetect) 10.10.0.1 tracker will autodetect ssh cli
2020-01-20 22:00:02.157 (iDetect) 10.10.0.1 Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x66791970>>
2020-01-20 22:00:02.158 (iDetect) Trackers initialized as:{'10.10.0.1': <trackers.ssh_autodetect.ssh_autodetect object at 0x707e1d10>}
2020-01-20 22:00:02.158 (iDetect) Plugin initialization done.
2020-01-20 22:00:02.157 Status: (iDetect) Tracker activated:10.10.0.1 port 22, user: admin, type: default and options: {}
2020-01-20 22:00:02.157 Error: (iDetect) 10.10.0.1 SSH Could not connect (with os key). Exception: No authentication methods available
2020-01-20 22:00:11.677 (iDetect) onHeartbeat called
2020-01-20 22:00:11.680 (iDetect) 2 devices are present (excluding ignored devices)
2020-01-20 22:00:16.723 (iDetect) 10.10.0.1 Not (yet) ready for polling
2020-01-20 22:00:21.701 (iDetect) onHeartbeat called

My progress with direct connect to UAP-PRO (Works):

Tracker: 10.10.0.20
Username and password set globally. SSH connection verified working with putty.

Code: Select all

2020-01-20 22:10:40.813 Status: (iDetect) Started.
2020-01-20 22:10:41.058 Status: (iDetect) Entering work loop.
2020-01-20 22:10:41.061 Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-01-20 22:10:43.419 (iDetect) Debug logging mask set to: PYTHON
2020-01-20 22:10:43.420 (iDetect) Operation system is: linux
2020-01-20 22:10:43.446 (iDetect) The OS user profile running domoticz is: root
2020-01-20 22:10:43.446 (iDetect) Parsing user and optional keyfile from:fAr_adMin
2020-01-20 22:10:43.447 (iDetect) start get or create wireless device
2020-01-20 22:10:43.447 (iDetect) Dan monitor tag_id:7C:D6:61:BB:0A:5D, domoticz unit:2
2020-01-20 22:10:43.447 (iDetect) start get or create wireless device
2020-01-20 22:10:43.447 (iDetect) Elisa monitor tag_id:48:2C:A0:68:45:34, domoticz unit:3
2020-01-20 22:10:43.447 (iDetect) Monitoring {'48:2C:A0:68:45:34': <plugin.tag_device object at 0x672f90f0>, '7C:D6:61:BB:0A:5D': <plugin.tag_device object at 0x6f476110>} for presence.
2020-01-20 22:10:43.447 (iDetect) Tracker configuration:10.10.0.20
2020-01-20 22:10:43.448 (iDetect) tracker:10.10.0.20
2020-01-20 22:10:43.448 (iDetect) options:
2020-01-20 22:10:43.450 (iDetect) 10.10.0.20 Tracker is of the ssh kind
2020-01-20 22:10:43.450 (iDetect) 10.10.0.20 ====> SSH Fetching data using:
2020-01-20 22:10:43.450 (iDetect) 10.10.0.20 ====> SSH not connected ... connecting
2020-01-20 22:10:43.450 (iDetect) 10.10.0.20 ====> SSH start connect
2020-01-20 22:10:43.994 (iDetect) 10.10.0.20 ====> SSH returned (decoded):wl: not found
2020-01-20 22:10:43.994 (iDetect) 10.10.0.20 ====> SSH session took 543 milliseconds.
2020-01-20 22:10:43.997 (iDetect) Available commands on 10.10.0.20:{'cat': '/bin/cat', 'ip': '/usr/sbin/ip', 'arp': '/sbin/arp', 'wlanconfig': '/usr/sbin/wlanconfig', 'brctl': '/usr/sbin/brctl'}
2020-01-20 22:10:43.997 (iDetect) 10.10.0.20 ====> SSH Fetching data using:
2020-01-20 22:10:43.956 Status: (iDetect) 10.10.0.20 ====> SSH connection established
2020-01-20 22:10:44.178 (iDetect) 10.10.0.20 ====> SSH returned (decoded):~ath0~ath1~ath2~ath4~ath5~ath6~eth0~eth0.20~eth0.30
2020-01-20 22:10:44.178 (iDetect) 10.10.0.20 ====> SSH session took 180 milliseconds.
2020-01-20 22:10:44.181 (iDetect) 10.10.0.20 Available interfaces for wlanconfig: ['ath0', 'ath1', 'ath2', 'ath4', 'ath5', 'ath6', 'eth0', 'eth0.20', 'eth0.30']
2020-01-20 22:10:44.182 (iDetect) 10.10.0.20 Prepared to poll using: /usr/sbin/wlanconfig ath0 list
2020-01-20 22:10:44.182 (iDetect) 10.10.0.20 tracker will autodetect ssh cli
2020-01-20 22:10:44.182 (iDetect) 10.10.0.20 Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x673168d0>>
2020-01-20 22:10:44.182 (iDetect) Trackers initialized as:{'10.10.0.20': <trackers.ssh_autodetect.ssh_autodetect object at 0x672f9750>}
2020-01-20 22:10:44.182 (iDetect) Plugin initialization done.
2020-01-20 22:10:44.182 Status: (iDetect) Tracker activated:10.10.0.20 port 22, user: fAr_adMin, type: default and options: {}
2020-01-20 22:10:53.703 (iDetect) onHeartbeat called
2020-01-20 22:10:53.703 (iDetect) 2 devices are present (excluding ignored devices)
2020-01-20 22:10:58.450 (iDetect) 10.10.0.20 Timed poll starting like clockwork
2020-01-20 22:10:58.450 (iDetect) 10.10.0.20 Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_autodetect.ssh_autodetect object at 0x672f9750>>
2020-01-20 22:10:58.450 (iDetect) 10.10.0.20 ====> SSH Fetching data using: /usr/sbin/wlanconfig ath0 list
2020-01-20 22:10:58.519 (iDetect) 10.10.0.20 ====> SSH returned (decoded):ADDR AID CHAN TXRATE RXRATE RSSI IDLE TXSEQ RXSEQ CAPS STATE MAXRATE(DOT11) DHCP TIME_TO_IP HTCAPS MODE PSMODE ASSOCTIME IEs
2020-01-20 22:10:58.519 (iDetect) 10.10.0.20 ====> SSH session took 68 milliseconds.
2020-01-20 22:10:58.528 (iDetect) Inbound data from: 10.10.0.20 containing ['34:CE:00:84:B9:33', '34:CE:00:8B:4C:31', '34:CE:00:88:B8:00', 'EC:FA:BC:8A:1F:8C', '04:CF:8C:78:BE:61', 'DC:4F:22:37:74:25', '5C:CF:7F:B4:61:4F', '40:31:3C:35:51:63', '04:CF:8C:3A:65:A0', '40:31:3C:AA:65:F8', '2C:3A:E8:11:61:A6', '48:2C:A0:68:45:34', '7C:D6:61:BB:0A:5D', '18:0C:AC:FB:6C:B4', 'E4:F0:42:28:B3:80', 'B0:41:1D:E1:54:F6', '00:22:15:A5:03:04', '02:42:0A:0A:00:FE', '02:42:0A:0A:00:FA', '78:7B:8A:9F:47:28', '38:EA:A7:A1:00:20', '20:CF:30:6E:D7:2A', 'D4:F5:47:22:B3:99', 'E4:F0:42:21:A5:05']
2020-01-20 22:10:58.529 (iDetect) 2 devices are present (excluding ignored devices)
2020-01-20 22:10:58.518 Error: (iDetect) 10.10.0.20 ====> SSH returned error:Not supported

However, should I be using the unifi tracker? If so, should I point it at the controller IP rather than the UAP and please can you offer me the correct syntax to use it as:

Tracker:

Code: Select all

110.10.0.10#port=8443&type=unifi&interval=30&user=adMin&password=password

throws the following:

Code: Select all

2020-01-20 22:23:33.481 Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-01-20 22:23:36.016 (iDetect) Debug logging mask set to: PYTHON
2020-01-20 22:23:36.017 (iDetect) Operation system is: linux
2020-01-20 22:23:36.044 (iDetect) The OS user profile running domoticz is: root
2020-01-20 22:23:36.044 (iDetect) Parsing user and optional keyfile from:adMin
2020-01-20 22:23:36.045 (iDetect) start get or create wireless device
2020-01-20 22:23:36.045 (iDetect) Dan monitor tag_id:7C:D6:61:BB:0A:5D, domoticz unit:2
2020-01-20 22:23:36.045 (iDetect) start get or create wireless device
2020-01-20 22:23:36.045 (iDetect) Elisa monitor tag_id:48:2C:A0:68:45:34, domoticz unit:3
2020-01-20 22:23:36.046 (iDetect) Monitoring {'48:2C:A0:68:45:34': <plugin.tag_device object at 0x670b6a10>, '7C:D6:61:BB:0A:5D': <plugin.tag_device object at 0x6f448210>} for presence.
2020-01-20 22:23:36.046 (iDetect) Tracker configuration:10.10.0.10#port=8443&type=unifi&interval=30&user=adMin&password=password
2020-01-20 22:23:36.046 (iDetect) tracker:10.10.0.10
2020-01-20 22:23:36.046 (iDetect) options:port=8443&type=unifi&interval=30&user=adMin&password=password
2020-01-20 22:23:36.047 Error: (iDetect) 'onStart' failed 'KeyError'.
2020-01-20 22:23:36.047 Error: (iDetect) ----> Line 427 in '/home/pi/domoticz/plugins/iDetect/plugin.py', function onStart
2020-01-20 22:23:36.047 Error: (iDetect) ----> Line 350 in '/home/pi/domoticz/plugins/iDetect/plugin.py', function onStart

Thanks for all your hard work. It's awesome!

[EscApe]

Hi airmarshall,

Firsty, it looks like you've catered already for the setup so how did you design the script to work with edgerouters and/or unifi wi-fi?

The plugin has three possibilities:
1. not specifying a type will make the plugin try to auto-detect the correct command available on the router through ssh (it will obviously only detect commands the plugin knows).
2. specifying a 'type' for which a specific tracker is available in the plugin (see the __init__.py for available tracker types)
3. specifying ssh= in the tracker configuration to use any ssh command you need

My progress with the Edgerouter (Doesn't Work):
Tracker: 10.10.0.1
Username and password set globally. SSH connection verified working with putty.

Make sure you are running putty under the same user profile that is running domoticz (root). I don't see a password in the tracker activation, so it seems you are using key based authentication, which must then be configured for root.

However, should I be using the unifi tracker? If so, should I point it at the controller IP rather than the UAP and please can you offer me the correct syntax to use it as:

Tracker:
CODE: SELECT ALL
110.10.0.10#port=8443&type=unifi&interval=30&user=adMin&password=password

unifi is not an existing type of tracker. You can use 'unifiusg-arp' (over ssh) or 'unifi-http' (over http).

Note that using arp on any router will eventually show all devices in the same LAN, so if you are using arp it might be enough to just query the edgerouter. If you want to monitor wireless devices only the most reliable and responsive method would probably be to only query the AP's using the unify-http type. But other unifi owners might have better advice to offer, since I am not familiar with them.

[MrEnergy]

Hi,

Had some spare time, looked into programming python3, well it's not native to me and hard to learn.

Untill now created a http_experiaboxv10.py in trackers :

Code: Select all

# Simple tracker for Experiabox v10
# Trying to get it working
# Author: MrEnergy

import Domoticz
from trackers.tracker_base import tracker
import requests
import re
from datetime import datetime, timezone
from hashlib import sha256
import xml.etree.ElementTree as ET
from collections import namedtuple

class http_experiaboxv10(tracker):
	def __init__(self, tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval):
		super().__init__(tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval)
		self.login_url = 'http://' + self_tracker_ip
		self.token_url = 'http://' + self_tracker_ip + "/function_module/login_module/login_page/logintoken_lua.lua"
		self.data_url = 'http://' + self_tracker_ip + "/common_page/home_AssociateDevs_lua.lua?AccessMode=WLAN&_={}"
		Domoticz.Debug(self.tracker_ip + ' Initializing: ')
		self.http_session = None
		self.prepare_for_polling()

	def poll_present_tag_ids(self):
		try:
			if not self.http_session:
				self.connect()
			response = self.http_session.get(self.data_url)
		except Exception as e:
			Domoticz.Error(self.tracker_ip + ' Polling error: ' + str(e))
			self.close_connection()
			
		if (response.status_code == 401):
			self.close_connection()
			Domoticz.Error(self_tracker_ip + 'Invalid login, or login has expired') 
			return
			
		http_response = self.http_session.get(self.data_url)
		raw_data = http_response.text
		Domoticz.Debug(self.tracker_ip + ' Returned: ' + raw_data)
		
		self.receiver_callback(raw_data)

	def connect(self):
		Domoticz.Debug(self.tracker_ip + ' Starting pull')
		self.http_session = requests.Session()
		ts = round(datetime.now(timezone.utc).timestamp() * 1000)
		# Login to get cookie
		self.http_session.get(self.login_url)
		result = self.http_session.get(self.token_url)
		response = self.http_session.post(
			self.login_url, data = {
				"Username": self.tracker_user,
				"Password":sha256((self.tracker_password + re.findall(r'\d+',result.text)[0]).encode('utf-8')).hexdigest(),
				"action": "login"
			}
		)

		if response.status_code == 400:
			Domoticz.Error(self.tracker_ip + 'Failed to login to Experiabox')
			return False
		Domoticz.Status(self.tracker_ip + ' Initialized as Experiabox v10')
		return True

	def disconnect(self):
		try:
			self.http_session_get("{}/logout".format(self.login_url))
		finally:
			self.close.connection()
 
    def close_connection(self):
        try:
            self.http_session.close()
            self.http_session = None
            Domoticz.Debug(self.tracker_ip + ' HTTP session closed')
        except Exception as e:
            Domoticz.Debug(self.tracker_ip + ' Close session exception: ' + str(e))
        return
       
    def prepare_for_polling(self):
        if self.connect():
            self.is_ready = True

    def stop_now(self):
        self.is_ready = False
        self.disconnect()

        super().stop_now()

It seems to be accepted but somehow indent got messed up. Cant find it using notepad++ Maybe somebody else can find the indent error ?

Domoticz log:

Code: Select all

2020-01-23 21:54:16.227  Status: (iDetect) Initialized version 2.0, author 'ESCape'
2020-01-23 21:54:16.576  Error: (iDetect) 'onStart' failed 'IndentationError'.
2020-01-23 21:54:16.576  Error: (iDetect) ----> Line 427 in 'C:\Program Files (x86)\Domoticz\plugins\Domoticz-iDetect\plugin.py', function onStart
2020-01-23 21:54:16.576  Error: (iDetect) ----> Line 212 in 'C:\Program Files (x86)\Domoticz\plugins\Domoticz-iDetect\plugin.py', function onStart
2020-01-23 21:54:16.576  Error: (iDetect) ----> Line 12 in 'C:\Program Files (x86)\Domoticz\plugins\Domoticz-iDetect\trackers\__init__.py', function <module>

(Yes developing on my Windows 10 laptop )

[EscApe]

@MrEnergy

Kudos for taking on this challenge yourself!
Python is very picky about indentation. Eg. a tab is not the same as a series of spaces.

The tracker might be more efficient if it would not connect and disconnect on every poll.
I would not mind experimenting a little to make this work, but unfortunately I don’t have access to an experiabox.

[EscApe]

@MrEnergy

Here is your code with the indentation corrected.

Code: Select all

# Simple tracker for Experiabox v10
# Trying to get it working
# Author: MrEnergy

import Domoticz
from trackers.tracker_base import tracker
import requests
import re
from datetime import datetime, timezone
from hashlib import sha256
import xml.etree.ElementTree as ET
from collections import namedtuple

class http_experiaboxv10(tracker):
	def __init__(self, tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval):
		super().__init__(tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval)
		self.login_url = 'http://' + self_tracker_ip
		self.token_url = 'http://' + self_tracker_ip + "/function_module/login_module/login_page/logintoken_lua.lua"
		self.data_url = 'http://' + self_tracker_ip + "/common_page/home_AssociateDevs_lua.lua?AccessMode=WLAN&_={}"
		Domoticz.Debug(self.tracker_ip + ' Initializing: ')
		self.http_session = None
		self.prepare_for_polling()

	def poll_present_tag_ids(self):
		try:
			if not self.http_session:
				self.connect()
			response = self.http_session.get(self.data_url)
		except Exception as e:
			Domoticz.Error(self.tracker_ip + ' Polling error: ' + str(e))
			self.close_connection()
			
		if (response.status_code == 401):
			self.close_connection()
			Domoticz.Error(self_tracker_ip + 'Invalid login, or login has expired') 
			return
			
		http_response = self.http_session.get(self.data_url)
		raw_data = http_response.text
		Domoticz.Debug(self.tracker_ip + ' Returned: ' + raw_data)
		
		self.receiver_callback(raw_data)

	def connect(self):
		Domoticz.Debug(self.tracker_ip + ' Starting pull')
		self.http_session = requests.Session()
		ts = round(datetime.now(timezone.utc).timestamp() * 1000)
		# Login to get cookie
		self.http_session.get(self.login_url)
		result = self.http_session.get(self.token_url)
		response = self.http_session.post(
			self.login_url, data = {
				"Username": self.tracker_user,
				"Password":sha256((self.tracker_password + re.findall(r'\d+',result.text)[0]).encode('utf-8')).hexdigest(),
				"action": "login"
			}
		)

		if response.status_code == 400:
			Domoticz.Error(self.tracker_ip + 'Failed to login to Experiabox')
			return False
		Domoticz.Status(self.tracker_ip + ' Initialized as Experiabox v10')
		return True

	def disconnect(self):
		try:
			self.http_session_get("{}/logout".format(self.login_url))
		finally:
			self.close.connection()
 
	def close_connection(self):
		try:
			self.http_session.close()
			self.http_session = None
			Domoticz.Debug(self.tracker_ip + ' HTTP session closed')
		except Exception as e:
			Domoticz.Debug(self.tracker_ip + ' Close session exception: ' + str(e))
		return
	   
	def prepare_for_polling(self):
		if self.connect():
			self.is_ready = True

	def stop_now(self):
		self.is_ready = False
		self.disconnect()

		super().stop_now()

[Dave21w]

OK, probably a stupid question but here goes.

How does this cope with the fact that an iPhone (mine is an 8) turns its Wifi off very shortly after it locks and goes to sleep, won't the system then think that you've left ?

Currently I'm using the geofencing option in the Pilot app to report home / away status but this has a minimum radius of about 200M and sometimes doesn't report away till I'm a mile up the road so I'd like an alternative, is bluetooth also an option with this by using the bluetooth built into the Pi or adding an external dongle if need be?

Thanks

[EscApe]

How does this cope with the fact that an iPhone (mine is an 8) turns its Wifi off very shortly after it locks and goes to sleep, won't the system then think that you've left ?

The ‘grace period’ setting will deal with (short) connection drops. I have polling set to every 15 seconds and a grace period of 40 seconds. So it will have two chances to detect the device within the grace period. This also means it can take 40 seconds to detect that a device has left.

is bluetooth also an option with this

If someone would write a tracker then it would be. It’s not available right now.

[HvdW]

Hi,
Just curiosity looking at this thread.
I have an Experia V10 as well, that's why it draw my attention.
So I downloaded the python script.
It halted on import Domoticz
I seached with pip and pip3 - no Domoticz module to import.

Can you explain what I do not understand?

[marktn]

Hello,

Does this work with a Fritzbox, OS 7.12 ?

SSH Could not connect (using password). Exception: [Errno None] Unable to connect to port 22

[MrEnergy]

@HvdW,

My first script is pure python. I run it from cron output to a file what i use from another script to search for the mac addresses of connected devices. Then toggle a switch in domoticz.

The second script is a work in progress tracker for this plugin, so no standalone python script.

Today tried again. Failing to get authenticated and thus read out the results.

@EscApe, the experiabox webpage is locked to one login at a time. So i need to logoff after each query on the page. But connecting should be done only once.

Might have some more time next weekend.

[EscApe]

Hello,

Does this work with a Fritzbox, OS 7.12 ?

SSH Could not connect (using password). Exception: [Errno None] Unable to connect to port 22

If I remember correctly the Fritzbox does not offer ssh, but connected clients can be queried from a html page(?) A tracker needs to be written for the Fritzbox. This should be possible with some python knowledge and access to a Fritzbox, but so far no one has taken on this challenge.