Python plugin: Presence detection from wireless router

[ESFnl]

Solved!

You are not running the same version as a few years ago I hope? That would be a major security risk without any patches/updates.

Ehh... sorry to admit. I knew all hardware and software is chasing each other for updates until you have to buy a new laptop or mobile phone. I forgot to update my router software.

For future reference the steps I took (after SSH into router):

Code: Select all

opkg update

Code: Select all

rm /usr/sbin/brctl

Code: Select all

opkg install bridge

Now my iDetect errors are gone
Thanks for all the help!

[EscApe]

Glad you got it working. Thanks for sharing the info!

[killtoo]

Hi,
If someone want to use this plugin with Unifi Dream Machine Pro (UDM Pro), here is the change for http_unifi.py tracker :

connect() :

Code: Select all

        response = self.http_session.post(
            "{}/api/auth/login".format(self.baseurl),
            data=json.dumps(login_data),
            verify=self.verify_ssl,
            headers={"content-type": "application/json"}
        )

poll_present_tag_ids() :

Code: Select all

response = self.http_session.get(
                "{}/proxy/network/api/s/{}/stat/sta".format(self.baseurl, self.site),
                verify=self.verify_ssl, 
                data="json={}"
            )

There is a bug in the new Unifi API, when wireless client leave WiFi, it appear in LAN network (so it seems to be connected).
If you use this plugin only for wireless client, you can add this code for keep only wireless clients connected :

Code: Select all

        jData = json.loads(raw_data)
        OnlineClients_list = []

        for client in jData["data"]:
            if client["is_wired"] == False:
                OnlineClients_list.append(client['mac'])

        self.receiver_callback(OnlineClients_list)

[Harald777]

Thanks for this great plugin, it is working great for my mobile phones using MAC adres.
Much better than to ping those devices.

I have one question: I also want to use the plugin to detect is a Personal Computer is on but I can only get this to work using it's IP address, not MAC address.
It it not possibl eto check if a PC is on using its MAC address?

Greetings, Harald

[EscApe]

If you force the plugin to use arp or brctl then wired devices would probably also be detected. But why not use the IP address or hostname to ping it if it is not battery powered?

[Harald777]

If you force the plugin to use arp or brctl then wired devices would probably also be detected. But why not use the IP address or hostname to ping it if it is not battery powered?

Thanks for the reply.
I was just wondering why it did not work. Switched to IP address and all fine (before I only used the plugin for mobile devices).

One other issue, every time I reboot my Domoticz the plugin fails to login into my router.
Just hit update in the hardware tab in Domoticz solves this issue.
I had this before but after my Raspberry PI Buster update and clean re-install I don't know how I fixed that before (2 years ago).
Any tips for that?

Greetings Harald

p.s. wanted to fix the login issue but my Domoticz is constantly crashing while working on the iDetect plugin.
Following error log observed:

Spoiler: show

2020-12-19 11:45:43.093 Status: (iDetect) Stop directive received.
2020-12-19 11:45:43.093 Status: (iDetect) Exiting work loop.
2020-12-19 11:45:43.124 Error: Domoticz(pid:653, tid:657('PluginMgr')) received fatal signal 6 (Aborted)
2020-12-19 11:45:43.124 Error: siginfo address=0x28d, address=0xffffffff
2020-12-19 11:45:43.258 (Slimme Meter) P1 Smart Meter (Electriciteit)
2020-12-19 11:45:43.265 (Slimme Meter) Usage (Usage L1)
2020-12-19 11:45:43.270 (Slimme Meter) Usage (Delivery L1)
2020-12-19 11:45:48.638 (RFLink 433 Gateway) Temp (Zolderkamer)
2020-12-19 11:45:48.717 Error: Thread 4 (Thread 0x74b9d220 (LWP 657)):
2020-12-19 11:45:48.717 Error: #0 __waitpid (options=0, stat_loc=0x74b9bed8, pid=1037) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2020-12-19 11:45:48.717 Error: #1 __waitpid (pid=1037, stat_loc=0x74b9bed8, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:25
2020-12-19 11:45:48.717 Error: #2 0x002e29ac in dumpstack_gdb(bool) ()
2020-12-19 11:45:48.717 Error: #3 0x002e2f78 in signal_handler(int, siginfo_t*, void*) ()
2020-12-19 11:45:48.717 Error: #4 <signal handler called>
2020-12-19 11:45:48.717 Error: #5 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
2020-12-19 11:45:48.717 Error: #6 0x76a16230 in __GI_abort () at abort.c:79
2020-12-19 11:45:48.717 Error: #7 0x75d7c270 in ?? () from /lib/arm-linux-gnueabihf/libpython3.7m.so

[EscApe]

By default the plugin queries the wireless chipset on your router for Mac addresses. That is why wired devices, like your pc, won’t show up. The plugin uses ping to look for ip addresses.

The startup issues might be related to Domoticz starting before the network interface is up (just guessing). You could try changing the startup order or simply delay the Domoticz startup.

[Toolhead]

Hi, trying this great plugin, but confronted with a problem.
I can ssh from my raspberry pi into my router and perform the brctl showmacs command.
So far so good.

But when I enter my username and password in to the plugin page the log says it can not log in.
Error: (idetect) 192.168.1.2&type ====> SSH Could not connect (using password). Exception: [Errno -2] Name or service not known

[EscApe]

But when I enter my username and password in to the plugin page the log says it can not log in.
Error: (idetect) 192.168.1.2&type ====> SSH Could not connect (using password). Exception: [Errno -2] Name or service not known

I would checking for typos in the configuration and test the ssh connection from the same userprofile that is running Domoticz (root?).
Can you share a full debug log if it still doesn’t work?

[Toolhead]

But when I enter my username and password in to the plugin page the log says it can not log in.
Error: (idetect) 192.168.1.2&type ====> SSH Could not connect (using password). Exception: [Errno -2] Name or service not known

I would checking for typos in the configuration and test the ssh connection from the same userprofile that is running Domoticz (root?).
Can you share a full debug log if it still doesn’t work?

The Pi is normally run with pi as user.
The router only accepts root as ssh user.
So ssh from terminal with giving root as username works.
When I switch to root I can login without giving username.
But the script is probably run as pi, I do not know how to start the idetect script as root to test.

[EscApe]

Domoticz normally runs as root on Linux, so the plugin is also running as root. Testing a cli ssh command from the Domoticz host to the router under the pi user profile does not tell you anything about things like ssh key mismatches.

[Toolhead]

Done a complete new installation of Raspbian and Domoticz.
This is the log:
2020-12-20 19:45:38.298 Status: (Detect) Started.
2020-12-20 19:45:38.477 Status: (Detect) Entering work loop.
2020-12-20 19:45:38.478 Status: (Detect) Initialized version 2.0, author 'ESCape'
2020-12-20 19:45:39.528 (Detect) Debug logging mask set to: PYTHON
2020-12-20 19:45:39.528 (Detect) Operation system is: linux
2020-12-20 19:45:39.545 (Detect) The OS user profile running domoticz is: root
2020-12-20 19:45:39.545 (Detect) Parsing user and optional keyfile from:root
2020-12-20 19:45:39.545 (Detect) Getting icon requested for Anyone: idetect-home from file:ihome.zip
2020-12-20 19:45:39.550 (Detect) New image: ID: -1, Base: '', Name: , Description: ''
2020-12-20 19:45:39.552 (Detect) Getting icon requested for Override: idetect-override from file:ioverride.zip
2020-12-20 19:45:39.557 (Detect) New image: ID: -1, Base: '', Name: , Description: ''
2020-12-20 19:45:39.558 (Detect) Getting icon requested for phone1: idetect-unithome from file:iunit.zip
2020-12-20 19:45:39.562 (Detect) New image: ID: -1, Base: '', Name: , Description: ''
2020-12-20 19:45:39.564 (Detect) start get or create wireless device
2020-12-20 19:45:39.564 (Detect) phone1 monitor tag_id:A1:B1:01:01:01:01, domoticz unit:2
2020-12-20 19:45:39.564 (Detect) Monitoring {'A1:B1:01:01:01:01': <plugin.tag_device object at 0x6a94bc50>} for presence.
2020-12-20 19:45:39.564 (Detect) Tracker configuration:192.168.1.2&type=brctl
2020-12-20 19:45:39.564 (Detect) tracker:192.168.1.2&type=brctl
2020-12-20 19:45:39.564 (Detect) options:
2020-12-20 19:45:39.566 (Detect) 192.168.1.2&type Tracker is of the ssh kind
2020-12-20 19:45:39.566 (Detect) 192.168.1.2&type Data will be received and interpreted by <bound method BasePlugin.onDataReceive of <plugin.BasePlugin object at 0x6f0fcf10>>
2020-12-20 19:45:39.566 (Detect) Trackers initialized as:{'192.168.1.2&type': <trackers.ssh_brctl.ssh_brctl object at 0x6a94bcd0>}
2020-12-20 19:45:39.566 (Detect) Plugin initialization done.
2020-12-20 19:45:39.551 Status: (Detect) Created device for Anyone with unit id 1
2020-12-20 19:45:39.558 Status: (Detect) Created device for Override with unit id 255
2020-12-20 19:45:39.564 Status: (Detect) Created device for phone1 with unit id 2
2020-12-20 19:45:39.566 Status: (Detect) Tracker activated:192.168.1.2&type port 22, user: root, type: brctl and options: {}
2020-12-20 19:45:39.564 Error: (Detect) WARNING! Tracker uses depricated configuration syntax but will work ... for now (see manual): 192.168.1.2&type=brctl
2020-12-20 19:45:49.085 (Detect) onHeartbeat called
2020-12-20 19:45:49.085 (Detect) 0 devices are present (excluding ignored devices)
2020-12-20 19:45:54.566 (Detect) 192.168.1.2&type Timed poll starting like clockwork
2020-12-20 19:45:54.566 (Detect) 192.168.1.2&type Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_brctl.ssh_brctl object at 0x6a94bcd0>>
2020-12-20 19:45:54.566 (Detect) 192.168.1.2&type ====> SSH Fetching data using: brctl showmacs br0
2020-12-20 19:45:54.566 (Detect) 192.168.1.2&type ====> SSH not connected ... connecting
2020-12-20 19:45:54.566 (Detect) 192.168.1.2&type ====> SSH start connect
2020-12-20 19:45:54.568 (Detect) 192.168.1.2&type Could not be polled
2020-12-20 19:45:54.567 Error: (Detect) 192.168.1.2&type ====> SSH Could not connect (using password). Exception: [Errno -2] Name or service not known
2020-12-20 19:45:59.105 (Detect) onHeartbeat called
2020-12-20 19:45:59.106 (Detect) 0 devices are present (excluding ignored devices)
2020-12-20 19:46:09.076 (Detect) onHeartbeat called
2020-12-20 19:46:09.076 (Detect) 0 devices are present (excluding ignored devices)
2020-12-20 19:46:09.569 (Detect) 192.168.1.2&type Timed poll starting like clockwork
2020-12-20 19:46:09.569 (Detect) 192.168.1.2&type Start poll and return results to <bound method tracker.receiver_callback of <trackers.ssh_brctl.ssh_brctl object at 0x6a94bcd0>>
2020-12-20 19:46:09.569 (Detect) 192.168.1.2&type ====> SSH Fetching data using: brctl showmacs br0
2020-12-20 19:46:09.569 (Detect) 192.168.1.2&type ====> SSH not connected ... connecting
2020-12-20 19:46:09.569 (Detect) 192.168.1.2&type ====> SSH start connect
2020-12-20 19:46:09.570 (Detect) 192.168.1.2&type Could not be polled
2020-12-20 19:46:09.570 Error: (Detect) 192.168.1.2&type ====> SSH Could not connect (using password). Exception: [Errno -2] Name or service not known

[EscApe]

2020-12-20 19:45:39.564 (Detect) Tracker configuration:192.168.1.2&type=brctl
2020-12-20 19:45:39.564 (Detect) tracker:192.168.1.2&type=brctl
2020-12-20 19:45:39.564 Error: (Detect) WARNING! Tracker uses depricated configuration syntax but will work ... for now (see manual):

Your tracker configuration should be 192.168.1.2#type=brctl
The configuration syntax is explained on the GitHub page. Ampersand is used to separate different options, but a hashtag separates the router host from (all) the options. Unfortunately the log message is misleading. Your configuration looks like the old syntax, but it won’t work.

[Toolhead]

Tried the hash, no more warning depricated configuration, but still won't login.
I give up.

[mcmikev]

Hi,
If someone want to use this plugin with Unifi Dream Machine Pro (UDM Pro), here is the change for http_unifi.py tracker :

connect() :

Code: Select all

        response = self.http_session.post(
            "{}/api/auth/login".format(self.baseurl),
            data=json.dumps(login_data),
            verify=self.verify_ssl,
            headers={"content-type": "application/json"}
        )

poll_present_tag_ids() :

Code: Select all

response = self.http_session.get(
                "{}/proxy/network/api/s/{}/stat/sta".format(self.baseurl, self.site),
                verify=self.verify_ssl, 
                data="json={}"
            )

There is a bug in the new Unifi API, when wireless client leave WiFi, it appear in LAN network (so it seems to be connected).
If you use this plugin only for wireless client, you can add this code for keep only wireless clients connected :

Code: Select all

        jData = json.loads(raw_data)
        OnlineClients_list = []

        for client in jData["data"]:
            if client["is_wired"] == False:
                OnlineClients_list.append(client['mac'])

        self.receiver_callback(OnlineClients_list)

Hi Killtoo,

Can you maybe send the edited files and an example of the trackerline in the config of the plugin. I have tried your changes but it does not work (errors).
I have tried sending a PM to you about this but the messages are send it says but they remain in my outbox so not sure if they were send for real.

Nice a USG upgrading to a UDM-Pro but after that the presence detection does not work anymore.

Maybe someone else also has it running on a UDM-Pro? and is wiling to share there knowledge?

Thank you!!

[tezzlicious]

Hi,
If someone want to use this plugin with Unifi Dream Machine Pro (UDM Pro), here is the change for http_unifi.py tracker :

connect() :

Code: Select all

        response = self.http_session.post(
            "{}/api/auth/login".format(self.baseurl),
            data=json.dumps(login_data),
            verify=self.verify_ssl,
            headers={"content-type": "application/json"}
        )

poll_present_tag_ids() :

Code: Select all

response = self.http_session.get(
                "{}/proxy/network/api/s/{}/stat/sta".format(self.baseurl, self.site),
                verify=self.verify_ssl, 
                data="json={}"
            )

There is a bug in the new Unifi API, when wireless client leave WiFi, it appear in LAN network (so it seems to be connected).
If you use this plugin only for wireless client, you can add this code for keep only wireless clients connected :

Code: Select all

        jData = json.loads(raw_data)
        OnlineClients_list = []

        for client in jData["data"]:
            if client["is_wired"] == False:
                OnlineClients_list.append(client['mac'])

        self.receiver_callback(OnlineClients_list)

Hi Killtoo,

Can you maybe send the edited files and an example of the trackerline in the config of the plugin. I have tried your changes but it does not work (errors).
I have tried sending a PM to you about this but the messages are send it says but they remain in my outbox so not sure if they were send for real.

Nice a USG upgrading to a UDM-Pro but after that the presence detection does not work anymore.

Maybe someone else also has it running on a UDM-Pro? and is wiling to share there knowledge?

Thank you!!

http_unifi.py:

Code: Select all

# Tracker for Ubiquiti Unifi Controller

import Domoticz
import requests
import json
from trackers.tracker_base import tracker


class http_unifi(tracker):
    def __init__(self, tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval):
        # Update default port
        if tracker_port == 22:
            tracker_port = 443

        super().__init__(tracker_ip, tracker_port, tracker_user, tracker_password, tracker_keyfile, poll_interval)

        self.baseurl = 'https://{}:{}'.format(self.tracker_ip, self.tracker_port)
        self.site = 'default'
        self.verify_ssl = False
        self.prepare_for_polling()

    def poll_present_tag_ids(self):
        try:
            if not self.http_session or :
                self.connect()

            response = self.http_session.get(
                "{}/proxy/network/api/s/{}/stat/sta".format(self.baseurl, self.site),
                verify=self.verify_ssl,
                data="json={}"
            )

        except Exception as e:
            Domoticz.Error(self.tracker_ip + ' Polling error: ' + str(e))
            self.close_connection()

        if (response.status_code == 401):
            self.close_connection()
            Domoticz.Error(self.tracker_ip + ' Invalid login, or login has expired')
            return

        raw_data = response.text
        Domoticz.Debug(self.tracker_ip + ' Returned: ' + raw_data)

        jData = json.loads(raw_data)
        OnlineClients_list = []

        for client in jData["data"]:
            if client["is_wired"] == False:
                OnlineClients_list.append(client['mac'])

        self.receiver_callback(OnlineClients_list)

    def connect(self):
        login_data = {
            'username': self.tracker_user,
            'password': self.tracker_password
        }

        self.http_session = requests.Session()

        response = self.http_session.post(
            "{}/api/auth/login".format(self.baseurl),
            data=json.dumps(login_data),
            verify=self.verify_ssl,
            headers={"content-type": "application/json"}
        )

        if response.status_code == 400:
            Domoticz.Error(self.tracker_ip + 'Failed to log in to Unifi API')
            return False

        Domoticz.Status(self.baseurl + ' Initialized as Ubiquiti Unifi Controller API')
        return True

    def disconnect(self):
        try:
            self.http_session.get("{}/logout".format(self.baseurl))
        finally:
            self.close_connection()

    def close_connection(self):
        try:
            self.http_session.close()
            self.http_session = None
            Domoticz.Debug(self.tracker_ip + ' HTTP session closed')
        except Exception as e:
            Domoticz.Debug(self.tracker_ip + ' Close session exception: ' + str(e))
        return

    def prepare_for_polling(self):
        if self.connect():
            self.is_ready = True

    def stop_now(self):
        self.is_ready = False
        self.disconnect()

        super().stop_now()

My problem is that it stops reporting online devices after a while.

2021-01-19 16:51:34.414 (iDetect) onHeartbeat called
2021-01-19 16:51:34.415 (iDetect) 0 devices are present (excluding ignored devices)

[killtoo]

Hi tezzlicious,

Sorry for my late response.
I have the same problem since a few weeks...

I'll do some tests with Postman to see what's going on.

[DarkAllMan]

I updated my Linux version to 20.04 and after that no devices are reporting online.
Logs show:

Code: Select all

 2021-01-29 09:14:02.476 Python Plugin System: (iDetect) onHeartbeat called
2021-01-29 09:14:02.476 Python Plugin System: (iDetect) 0 devices are present (excluding ignored devices)
2021-01-29 09:14:06.351 Python Plugin System: (iDetect) 192.168.1.1 Not (yet) ready for polling 

Any idea what I need to do to make it work again? How to debug this issue?

[EscApe]

I updated my Linux version to 20.04 and after that no devices are reporting online.

A full debug log from the moment the plugin starts might shed some more light on this, but my first guess would be that a new python version got installed with the linux update. This would require you to also install the required modules (like paramiko) for that python version. You can see what python version Domoticz is using on the Domoticz 'about' screen.
Edit: make sure the modules are installed for the right python (virtual) environment. I always install them globally to be sure, but that is not necessarily the default behavior.

[DarkAllMan]

I updated my Linux version to 20.04 and after that no devices are reporting online.

A full debug log from the moment the plugin starts might shed some more light on this, but my first guess would be that a new python version got installed with the linux update. This would require you to also install the required modules (like paramiko) for that python version. You can see what python version Domoticz is using on the Domoticz 'about' screen.
Edit: make sure the modules are installed for the right python (virtual) environment. I always install them globally to be sure, but that is not necessarily the default behavior.

I'm running Python Version: 3.8.5
Where do I find the full debug logs? Can you point me in the right direction?
Updated the required elements for this python version and it is working again! Thx....