How to monitor your internet connection with virtual custom sensors

[safi78]

Hi all,

I was having some troubles with my internet connection and couldn't figure out where they were coming from. I really like the graphing functionality of domoticz and I have implemented a script that notifies me with Telegram, so I wanted to monitor my download speed with domoticz, thought I'd share:

1. Install speedtest

Code: Select all

sudo apt-get install speedtest-cli

2. Create 3 Virtual Sensors of the type 'Custom Sensors' (for ping, download, upload-values) and remember the idx's (you need that later on)

3. Create a script (I called it st2domo.sh) with:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
pingidx=xxx
downloadidx=xxx
uploadidx=xxx

# no need to edit
speedtest-cli --simple > output.txt
ping=$(cat output.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')
download=$(cat output.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat output.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')

#output if you run it manually
echo "ping = $ping ms"
echo "download = $download Mbps"
echo "upload =  $upload Mbps"

curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$pingidx&svalue=$ping"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$downloadidx&svalue=$download"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$uploadidx&svalue=$upload"

and fill in all the details after #setup:
host = your domoticz server (e.g. localhost or 192.168.x.x)
port = your domoticz port (e.g. 8080)
username = your domoticz username
password = your domoticz password
pingidx = the idx of the virtual sensor for accepting the ping-value
downloadidx = the idx of the virtual sensor for accepting the download-value
uploadidx = the idx of the virtual sensor for accepting the upload-value

Don't forget to make it executable (I always do):

Code: Select all

chmod +x st2domo.sh

4. edit your crontab (I scheduled this @hourly):

Code: Select all

crontab -e

And put in the command:

Code: Select all

@hourly /path/to/scrip/st2domo.sh

If setup correctly, domoticz will receive your results from speedtest every hour:

Screen Shot 2016-10-22 at 19.57.29.png (124.73 KiB) Viewed 22321 times

And you can do 'useful' stuff with it

Enjoy!

[woody4165]

Hi and thanks for sharing your script.

I have not found speedtest as package, but only speedtest-cli and probably the usage is different.
Should I add some source to download this package?

Thanks

[safi78]

my bad, you install 'speedtest-cli' but the command you use is 'speedtest' (so the script is ok). I'll update my post accordingly.

Thanks for the heads up!

[manjh]

This is great for the R-Pi, but is there an alternative for a Windows based server?

[safi78]

This is great for the R-Pi, but is there an alternative for a Windows based server?

Sorry manjh, I do not use Windows ever.

[woody4165]

@safi78
it's working like a charm, just replaced

Code: Select all

speedtest

with

Code: Select all

speedtest-cli

and it's ok.
I've also added --server xxxx where xxx is the server number near to myself so that test is more accurate.
You can find your test server with

Code: Select all

speedtest-cli --list

and you will find the number associated to the server name.

So my test is:

Code: Select all

speedtest-cli --server xxxx | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')

Do you know how can I get also ping value?

the return is like this:
......[15.27 km]: 12.356 ms

Thanks!

[safi78]

changing the speedtest command to speedtest --simple will just produce this output:

Code: Select all

Ping: 27.302 ms
Download: 36.26 Mbits/s
Upload: 5.03 Mbits/s

Then you can use:

Code: Select all

sed -ne 's/^Ping: \([0-9]*.[0-9]*\).*/\1/p'

results to:

Code: Select all

27.302

[woody4165]

thanks again!!

what i would like to understand is if I have to run three times the speedtest to get DL,UP and Ping, or if there is a way to get all the three numbers with one speedtest.

[safi78]

Sure, this was just a quick hack

just run speedtest --simple > output.txt

then cat output.txt | sed (etc) ?

edit: I'll change the script to that, more efficient (I was only chasing after the Download value )

[woody4165]

Fantastic

so now my script is like this:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
#idx for download, upload and ping
idxdl=xxx
idxul=xxx
idxpng=xxx

# speedtest server number
serverst=xxxx


# no need to edit
speedtest-cli --simple --server $serverst > outst.txt

download=$(cat outst.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat outst.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')
png=$(cat outst.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')


#output if you run it manually
echo "download = $download Mbps"
echo "upload =  $upload Mbps"
echo "ping =  $png ms"


# Updating download, upload and ping ..
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxdl&svalue=$download" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxul&svalue=$upload" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxpng&svalue=$png" >/dev/null 2>&1

working and updating.

Perfect!

Thanks again

[safi78]

you're welcome & thanks for your input as well

[jumbotroll]

This is great for the R-Pi, but is there an alternative for a Windows based server?

On windows 10 it's working here.
Install python 3.5.2 from https://www.python.org/downloads/
Open cmd as Administrator and type: pip install speedtest-cli
Then you can type: speedtest

[simonrg]

Nice approach, a couple of comments.

I only discovered that speedtest appears to download 30MB per test, when I used up my 50GB limit in 7 days by testing every 5 minutes (~4GB per day ).

I took a slightly different approach based on a mash up of ideas from this forum.

A simple bash script runs speedtest:

Code: Select all

#!/bin/sh
/usr/bin/speed-test --json >/var/tmp/speed-test.json 2>&1

set to run by cron every hour on the hour (min=00). The script could also run via a timer in Domoticz to keep more of the setup inside Domoticz.

A Lua script runs at 3 minutes past the hour to read the file created by speedtest, decode the json, send the value to Domoticz via commandArray and send a notification if download value is below 3.0Mbps.

Code: Select all

-- ~/domoticz/scripts/lua/script_time_speed.lua

commandArray = {}

JSON  = require('JSON')
date = os.date("*t")

function dospeed(filename, devdown, devup, devping, name, index)
-- filename - full path and filename where results from speedtest are stored
-- devdown, deveup, devping - Domoticz virtual sensor indexes
-- name - used in notification to identify isp
-- index - normal 1, increment by 5 for each additional internet connection tested
    file = io.open(filename, "r")
    json_text = file:read()
    file:close()
    speeds = JSON:decode(json_text)
    print(name..": Ping="..speeds.ping.." Download=".. speeds.download.." Upload=".. speeds.upload)
    commandArray[index]={['UpdateDevice']=devdown..'|0|'..speeds.download}
    commandArray[index+1]={['UpdateDevice']=devup..'|0|'..speeds.upload}
    commandArray[index+2]={['UpdateDevice']=devping..'|0|'..speeds.ping}
    if speeds.download <3.0 then
       commandArray[index+3]={['SendNotification']=name..' internet download speed alert#'..name..' download '..speeds.download..'Mb/s'}
    end
    if speeds.upload <1.0 then
       commandArray[index+4]={['SendNotification']=name..' internet upload speed alert#'..name..' upload '..speeds.upload..'Mb/s'}
    end
end

-- Once an hour
if (date.min == 3) then
--    os.execute('wget http://ipinfo.io/ip -qO - > /var/tmp/ip.txt')
-- 1st internet connection
    dospeed("/var/tmp/speed-test.json", 958, 959, 960, "Eclipse", 1)
-- 2nd internet connection
    dospeed("/var/tmp/speed-test-lnc.json", 961, 962, 963, "Boundless", 6)
end

return commandArray

[woody4165]

Hi simonrg

one question is why you should test the line every five mins...

In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test

Thanks

[safi78]

Nice example with JSON, like it.

But same here, the --json doesn't seem to be included in the debian package? What kind of speedtest are you using?

Regards,

Sander

[simonrg]

one question is why you should test the line every five mins...
In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test

Why every 5 minutes? Why not? Ok because it uses up my bandwith, but a realtime monitor would allow me to spot issues early and sort them out.

I have both a fibre and an over the air long range wifi-connection. While speed at one time can appear to be fast, I still to see stuttering on streaming services etc., so I was trying to see if there was anything systematic - farmer moving tractor into line of sight, wind blowing trees into line of sight etc..

So I am not sure once an hour is really useful to track long term trends, but not really useful to me for intermittant faults, as fault could already have happened 55minutes ago, and more annoylingly I could leave home after fault has occured without realising, so could have fixed it if I had known. Basically speed test is not the not the way to do it, I really need access to ISP network infrastructure as this will have the information I need on quality of connection with no unnecessary bandwidth.

For example, this morning

161023Speed.jpeg (39.31 KiB) Viewed 22177 times

my speed has dropped from 10 to <2Mbps, I have no idea why, no wind no rain no obvious obstructions, but lets see what ISP says.

I am running on Raspberry Pi, I got speed-test from https://www.npmjs.com/package/speed-test.

I had forgotten my code is largely copied from viewtopic.php?t=11970#p86162 which has a lot more discussion of measuring network speed.

[safi78]

Why every 5 minutes? Why not?

Because it will try to suck the shit out of your connection every 5 minutes and that will affect everything else you do?

Even if it is just 30MB every 5 minutes you'll have major lag on everything else right.

[Derik]

@Safi78

Try it:

Code: Select all

pi@raspberrypi ~ $ sudo apt-get install speedtest-cli
Pakketlijsten worden ingelezen... Klaar
Boom van vereisten wordt opgebouwd
De status informatie wordt gelezen... Klaar
E: Kan pakket speedtest-cli niet vinden
pi@raspberrypi ~ $

Edit:

Did some work:

Code: Select all

 sudo apt-get install python-pip

Then:

Code: Select all

pip install speedtest-cli

Then try the script :
cd pad to the script:
then:
./scriptname.sh

And working...


Only what is the time when i use:
@hourley

Edit 2:
I do not like to run this script exactly @ x.00 hour.
So i made the crondtab:

Code: Select all

11 0 * * * /home/pi/domoticz/scripts/speedtest.sh

11 minutes past a hour..
Hope this is going to work, and not kill my internet speed every hour...

Thanks guys.. for this script!!

[Derik]

mm
Only i stupid question...
What is the best ping high or low..?

[emme]

low ping means fast response.... so it's good
high ping means slow response... that' bad

Watch out the unit: normally it's ms-->milliseconds