Page 1 of 13

How to monitor your internet connection with virtual custom sensors

Posted: Saturday 22 October 2016 17:38
by safi78
Hi all,

I was having some troubles with my internet connection and couldn't figure out where they were coming from. I really like the graphing functionality of domoticz and I have implemented a script that notifies me with Telegram, so I wanted to monitor my download speed with domoticz, thought I'd share:

1. Install speedtest

Code: Select all

sudo apt-get install speedtest-cli
2. Create 3 Virtual Sensors of the type 'Custom Sensors' (for ping, download, upload-values) and remember the idx's (you need that later on)

3. Create a script (I called it st2domo.sh) with:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
pingidx=xxx
downloadidx=xxx
uploadidx=xxx

# no need to edit
speedtest-cli --simple > output.txt
ping=$(cat output.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')
download=$(cat output.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat output.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')

#output if you run it manually
echo "ping = $ping ms"
echo "download = $download Mbps"
echo "upload =  $upload Mbps"

curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$pingidx&svalue=$ping"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$downloadidx&svalue=$download"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$uploadidx&svalue=$upload"

and fill in all the details after #setup:
host = your domoticz server (e.g. localhost or 192.168.x.x)
port = your domoticz port (e.g. 8080)
username = your domoticz username
password = your domoticz password
pingidx = the idx of the virtual sensor for accepting the ping-value
downloadidx = the idx of the virtual sensor for accepting the download-value
uploadidx = the idx of the virtual sensor for accepting the upload-value

Don't forget to make it executable (I always do):

Code: Select all

chmod +x st2domo.sh
4. edit your crontab (I scheduled this @hourly):

Code: Select all

crontab -e
And put in the command:

Code: Select all

@hourly /path/to/scrip/st2domo.sh
If setup correctly, domoticz will receive your results from speedtest every hour:
Screen Shot 2016-10-22 at 19.57.29.png
Screen Shot 2016-10-22 at 19.57.29.png (124.73 KiB) Viewed 22367 times
And you can do 'useful' stuff with it ;)

Enjoy!

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 18:20
by woody4165
Hi and thanks for sharing your script.

I have not found speedtest as package, but only speedtest-cli and probably the usage is different.
Should I add some source to download this package?

Thanks

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 18:22
by safi78
my bad, you install 'speedtest-cli' but the command you use is 'speedtest' (so the script is ok). I'll update my post accordingly.

Thanks for the heads up!

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 18:36
by manjh
This is great for the R-Pi, but is there an alternative for a Windows based server?

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 18:37
by safi78
manjh wrote:This is great for the R-Pi, but is there an alternative for a Windows based server?
Sorry manjh, I do not use Windows ever.

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:10
by woody4165
@safi78
it's working like a charm, just replaced

Code: Select all

speedtest
with

Code: Select all

speedtest-cli
and it's ok.
I've also added --server xxxx where xxx is the server number near to myself so that test is more accurate.
You can find your test server with

Code: Select all

speedtest-cli --list
and you will find the number associated to the server name.

So my test is:

Code: Select all

speedtest-cli --server xxxx | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
Do you know how can I get also ping value?

the return is like this:
......[15.27 km]: 12.356 ms

Thanks!

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:32
by safi78
changing the speedtest command to speedtest --simple will just produce this output:

Code: Select all

Ping: 27.302 ms
Download: 36.26 Mbits/s
Upload: 5.03 Mbits/s
Then you can use:

Code: Select all

sed -ne 's/^Ping: \([0-9]*.[0-9]*\).*/\1/p'
results to:

Code: Select all

27.302

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:36
by woody4165
thanks again!!

what i would like to understand is if I have to run three times the speedtest to get DL,UP and Ping, or if there is a way to get all the three numbers with one speedtest.

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:38
by safi78
Sure, this was just a quick hack ;)

just run speedtest --simple > output.txt

then cat output.txt | sed (etc) ?

edit: I'll change the script to that, more efficient (I was only chasing after the Download value ;) )

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:49
by woody4165
Fantastic

so now my script is like this:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
#idx for download, upload and ping
idxdl=xxx
idxul=xxx
idxpng=xxx

# speedtest server number
serverst=xxxx


# no need to edit
speedtest-cli --simple --server $serverst > outst.txt

download=$(cat outst.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat outst.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')
png=$(cat outst.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')


#output if you run it manually
echo "download = $download Mbps"
echo "upload =  $upload Mbps"
echo "ping =  $png ms"


# Updating download, upload and ping ..
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxdl&svalue=$download" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxul&svalue=$upload" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxpng&svalue=$png" >/dev/null 2>&1
working and updating.

Perfect!

Thanks again

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 19:52
by safi78
you're welcome & thanks for your input as well :)

Re: monitor internet download speed with virtual sensor

Posted: Saturday 22 October 2016 20:20
by jumbotroll
manjh wrote:This is great for the R-Pi, but is there an alternative for a Windows based server?
On windows 10 it's working here.
Install python 3.5.2 from https://www.python.org/downloads/
Open cmd as Administrator and type: pip install speedtest-cli
Then you can type: speedtest

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 1:01
by simonrg
Nice approach, a couple of comments.

I only discovered that speedtest appears to download 30MB per test, when I used up my 50GB limit in 7 days by testing every 5 minutes :oops: (~4GB per day :lol: ).

I took a slightly different approach based on a mash up of ideas from this forum.

A simple bash script runs speedtest:

Code: Select all

#!/bin/sh
/usr/bin/speed-test --json >/var/tmp/speed-test.json 2>&1
set to run by cron every hour on the hour (min=00). The script could also run via a timer in Domoticz to keep more of the setup inside Domoticz.

A Lua script runs at 3 minutes past the hour to read the file created by speedtest, decode the json, send the value to Domoticz via commandArray and send a notification if download value is below 3.0Mbps.

Code: Select all

-- ~/domoticz/scripts/lua/script_time_speed.lua

commandArray = {}

JSON  = require('JSON')
date = os.date("*t")

function dospeed(filename, devdown, devup, devping, name, index)
-- filename - full path and filename where results from speedtest are stored
-- devdown, deveup, devping - Domoticz virtual sensor indexes
-- name - used in notification to identify isp
-- index - normal 1, increment by 5 for each additional internet connection tested
    file = io.open(filename, "r")
    json_text = file:read()
    file:close()
    speeds = JSON:decode(json_text)
    print(name..": Ping="..speeds.ping.." Download=".. speeds.download.." Upload=".. speeds.upload)
    commandArray[index]={['UpdateDevice']=devdown..'|0|'..speeds.download}
    commandArray[index+1]={['UpdateDevice']=devup..'|0|'..speeds.upload}
    commandArray[index+2]={['UpdateDevice']=devping..'|0|'..speeds.ping}
    if speeds.download <3.0 then
       commandArray[index+3]={['SendNotification']=name..' internet download speed alert#'..name..' download '..speeds.download..'Mb/s'}
    end
    if speeds.upload <1.0 then
       commandArray[index+4]={['SendNotification']=name..' internet upload speed alert#'..name..' upload '..speeds.upload..'Mb/s'}
    end
end

-- Once an hour
if (date.min == 3) then
--    os.execute('wget http://ipinfo.io/ip -qO - > /var/tmp/ip.txt')
-- 1st internet connection
    dospeed("/var/tmp/speed-test.json", 958, 959, 960, "Eclipse", 1)
-- 2nd internet connection
    dospeed("/var/tmp/speed-test-lnc.json", 961, 962, 963, "Boundless", 6)
end

return commandArray

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 7:54
by woody4165
Hi simonrg

one question is why you should test the line every five mins...

In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test

Thanks

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 10:23
by safi78
Nice example with JSON, like it.

But same here, the --json doesn't seem to be included in the debian package? What kind of speedtest are you using?

Regards,

Sander

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 12:46
by simonrg
woody4165 wrote:one question is why you should test the line every five mins...
In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test
Why every 5 minutes? Why not? Ok because it uses up my bandwith, but a realtime monitor would allow me to spot issues early and sort them out.

I have both a fibre and an over the air long range wifi-connection. While speed at one time can appear to be fast, I still to see stuttering on streaming services etc., so I was trying to see if there was anything systematic - farmer moving tractor into line of sight, wind blowing trees into line of sight etc..

So I am not sure once an hour is really useful to track long term trends, but not really useful to me for intermittant faults, as fault could already have happened 55minutes ago, and more annoylingly I could leave home after fault has occured without realising, so could have fixed it if I had known. Basically speed test is not the not the way to do it, I really need access to ISP network infrastructure as this will have the information I need on quality of connection with no unnecessary bandwidth.

For example, this morning
161023Speed.jpeg
161023Speed.jpeg (39.31 KiB) Viewed 22223 times
my speed has dropped from 10 to <2Mbps, I have no idea why, no wind no rain no obvious obstructions, but lets see what ISP says.

I am running on Raspberry Pi, I got speed-test from https://www.npmjs.com/package/speed-test.

I had forgotten my code is largely copied from viewtopic.php?t=11970#p86162 which has a lot more discussion of measuring network speed.

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 15:38
by safi78
simonrg wrote: Why every 5 minutes? Why not?
Because it will try to suck the shit out of your connection every 5 minutes and that will affect everything else you do? ;)

Even if it is just 30MB every 5 minutes you'll have major lag on everything else right.

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 17:08
by Derik
@Safi78

Try it:

Code: Select all

pi@raspberrypi ~ $ sudo apt-get install speedtest-cli
Pakketlijsten worden ingelezen... Klaar
Boom van vereisten wordt opgebouwd
De status informatie wordt gelezen... Klaar
E: Kan pakket speedtest-cli niet vinden
pi@raspberrypi ~ $
Edit:

Did some work:

Code: Select all

 sudo apt-get install python-pip
Then:

Code: Select all

pip install speedtest-cli
Then try the script :
cd pad to the script:
then:
./scriptname.sh

And working... :D


Only what is the time when i use:
@hourley

Edit 2:
I do not like to run this script exactly @ x.00 hour.
So i made the crondtab:

Code: Select all

11 0 * * * /home/pi/domoticz/scripts/speedtest.sh
11 minutes past a hour..
Hope this is going to work, and not kill my internet speed every hour...

Thanks guys.. for this script!!

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 18:43
by Derik
mm
Only i stupid question...
What is the best ping high or low..?

Re: monitor internet connection with virtual sensors

Posted: Sunday 23 October 2016 18:55
by emme
low ping means fast response.... so it's good
high ping means slow response... that' bad

Watch out the unit: normally it's ms-->milliseconds