pfSense Internet Usage and Bandwidth

All kinds of 'OS' scripts

Moderator: leecollings

ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

pfSense Internet Usage and Bandwidth

Post by ben53252642 »

Hey Folks,

This is a bash script I wrote for getting the last minutes bandwidth information from pfsense and loading it into Domoticz.

Steps:
1) Go to this url on your pfsense box: https://pfsenseip/services_snmp.php
2) Tick the box to enable snmp and set the "Read Community String" to: "public" then scroll down and bind it to the LAN interface and press save.
3) Install requirements on your Domoticz server: apt-get install snmp bc
3) Fill in everything in the script except for the "waninterfacename"
3) Run the script like this: "./update.sh list" and you will get an output looking something like this:

vtnet0
vtnet1
enc0
lo0
pflog0
pfsync0
pppoe0
ovpns1

In this case my wan interface is called pppoe0 so what is what I would enter in "waninterfacename" in the script (go ahead and fill the script with the name of your wan interface).

update.sh

Code: Select all

#!/bin/bash

# Pfsense Configuration
pfsenseip="192.168.0.1"
snmpid="public"
snmpversion="2c"
waninterfacename="pppoe0"
measuretime="60" # If you want avg mbps per minute, this would be 60

# Domoticz Configuration
domoticzserver="192.168.0.5"
domoticzport="443"
domoticzuser="username"
domoticzpass="password"

# Identify the WAN interface
interfacelist=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.2)
wansnmpid=$(echo "$interfacelist" | grep "$waninterfacename" | awk '{ print $1 }' | grep -o '.\{1\}$')

# Get list of interfaces and display them in terminal
if [[ $1 == 'list' ]]; then
echo "$interfacelist" | awk '{ print $4 }' | sed 's/\"//g'
exit
fi

# Start work loop
while true; do

# Notify terminal
echo && echo "Interface: $waninterfacename"
echo "Calculating bandwidth per $measuretime seconds in mbps... (wait $measuretime seconds)"

# Get data
waninkb=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.10."$wansnmpid" | awk '{ print $4 }')
wanoutkb=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.16."$wansnmpid" | awk '{ print $4 }')

# Wait time period
sleep "$measuretime"

# Get data a second time
waninkb2=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.10."$wansnmpid" | awk '{ print $4 }')
wanoutkb2=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.16."$wansnmpid" | awk '{ print $4 }')

# Do math for Mbps
inbits=$(echo $((("$waninkb2"-"$waninkb") / 60 * 8)) | bc -l)
inmegabits=$(echo "scale=2; $inbits / 1000000" | bc -l)
outbits=$(echo $((("$wanoutkb2"-"$wanoutkb") / 60 * 8)) | bc -l)
outmegabits=$(echo "scale=2; $outbits / 1000000" | bc -l)

# Check data viability
if (( $(echo "$inbits > 2" |bc -l) )) && (( $(echo "$outbits > 2" |bc -l) )); then

# Notify Terminal 2
echo "In: $inmegabits Mbps"
echo "Out: $outmegabits Mbps"

# Send data to Domoticz
curl --max-time 60 -s -k "https://{$domoticzuser}:{$domoticzpass}@{$domoticzserver}:{$domoticzport}/json.htm?type=command&param=udevice&idx=503&svalue=${inmegabits}"
curl --max-time 60 -s -k "https://{$domoticzuser}:{$domoticzpass}@{$domoticzserver}:{$domoticzport}/json.htm?type=command&param=udevice&idx=502&svalue=${outmegabits}"

# Report if there was a problem
else
echo "There was a problem getting the data"
fi

done
This is the startup.sh script I use for running it in the background:

Code: Select all

# Start pfsense monitor
/usr/bin/screen -S pfsense-monitor -d -m /root/pfsense/update.sh
screenshot112532.JPG
screenshot112532.JPG (77.33 KiB) Viewed 7690 times
terminaloutput1534.JPG
terminaloutput1534.JPG (41.67 KiB) Viewed 7686 times
Last edited by ben53252642 on Thursday 26 April 2018 23:50, edited 5 times in total.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

Great script but wouldn't be good to set the IDX as variables too?

Abit howto setup the dummy devices too.

I added it to the crontab so I don't need to start the screen if I reboot my server.
Removed the while loop and sleep

Added custom icons
Attachments
domoticz_custom_icon_upload.zip
(4.36 KiB) Downloaded 217 times
domoticz_custom_icon_download.zip
(4.38 KiB) Downloaded 199 times
Last edited by Varazir on Sunday 12 March 2017 16:54, edited 1 time in total.
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

and this icons

They look like this
pfsense.JPG
pfsense.JPG (51.15 KiB) Viewed 10341 times
Attachments
domoticz_custom_icon_upspeed.zip
(2.72 KiB) Downloaded 176 times
domoticz_custom_icon_downspeed.zip
(2.84 KiB) Downloaded 180 times
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

Awesome! Love the icons! :D

If anyone finds a simpler way to get the data out of pfsense please let me know.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
void
Posts: 26
Joined: Wednesday 04 November 2015 10:26
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by void »

Anyone tried with the 2.4 branch? Some things got a major update, such as the login screen.
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

void wrote: Monday 06 November 2017 11:03 Anyone tried with the 2.4 branch? Some things got a major update, such as the login screen.
Can confirm that existing script works fine with 2.4.* branch.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
void
Posts: 26
Joined: Wednesday 04 November 2015 10:26
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by void »

Did you consider creating a Wiki article for it at http://www.domoticz.com/wiki/Main_Page ?
Might be interessting for more people, and the forum search function kind of sucks... :)
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

void wrote: Wednesday 31 January 2018 9:21 Did you consider creating a Wiki article for it at http://www.domoticz.com/wiki/Main_Page ?
Might be interessting for more people, and the forum search function kind of sucks... :)
I tried to add something to the wiki recently but was unable to create an account:

viewtopic.php?f=43&p=167358#p167358

I think for now this is ok in the Forum, also people can post if they have problems and it's easy to reply.

When I can access the Wiki perhaps I'll make a page directing people here, Ultimately it would be better if there was a native Domoticz plugin. Until then we have this bash script.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
MnM001
Posts: 4
Joined: Wednesday 31 January 2018 3:42
Target OS: Linux
Domoticz version:
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by MnM001 »

Hi guys looking for some help here.

Where I run the script I get:

./test.sh: line 19: /usr/bin/curl: Argument list too long
error : Unknown IO error

The script does log into my pfsense as I can see it in my logs.
DarkStat and Totals configured correctly.

I am a beginner with scripting so I will appreciate if someone can help me resolve this error.

Thanks
domoticzcom1234
Posts: 19
Joined: Saturday 25 June 2016 17:21
Target OS: Raspberry Pi / ODroid
Domoticz version: Stable
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by domoticzcom1234 »

This can be caused by many things. Answering your question therefore is not very easy.
This implementation can be very complicated and is time consuming to get it working.

The only thing I can do is give you a basic guidance as listed below:
- Did you create an account on Pfsense
- Did you gave the correct permissions to access the right modules
- login as the new user and password and run this in your browser: https://IP-ADDRESS-OF-YOUR-PFSENSE/vnst ... h_json.php, are you getting the right response? No?, your permissions are set wrong.
- Are you using the right PFsense network interface and is it in the same order listed in the PFsense URL in the script (try this in the browser first as listed above)
- did you installed jq, bc AND libxml2-utils
- are you using the latest version of pfsense
- do you have the right permissions in linux

And the best one to check:
- run the script with the -x such as "sh -x scriptname.sh" so you can check what goes wrong



Good luck and this is the only thing I can do.
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

I'm in the process of completely re-writing this to work from SNMP and not require either bandwidth totals or darkstat on Pfsense.

I've completed the bandwidth part, it's currently installed on my Domoticz box updating nicely every minute.

I'm now working on the monthly bandwidth calculations. Should be posted in the next couple of days, stay tuned.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

ben53252642 wrote: Tuesday 10 April 2018 23:07 I'm in the process of completely re-writing this to work from SNMP and not require either bandwidth totals or darkstat on Pfsense.

I've completed the bandwidth part, it's currently installed on my Domoticz box updating nicely every minute.

I'm now working on the monthly bandwidth calculations. Should be posted in the next couple of days, stay tuned.
Sweet
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

I've updated the first post with the new code which I believe is significantly better than the previous version.

Note that for now it's only the last minutes bandwidth (not monthly downloads), I'm still working on a better way to do that.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

ben53252642 wrote: Thursday 26 April 2018 23:45 I've updated the first post with the new code which I believe is significantly better than the previous version.

Note that for now it's only the last minutes bandwidth (not monthly downloads), I'm still working on a better way to do that.
will not Domoticz keep track of the monthly data traffic ? ?
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

ben53252642 wrote: Saturday 25 February 2017 1:45 Hey Folks,
Maybe create a variable for the device ID in domoticz ?
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

getting error

Code: Select all

++ awk '{ print $4 }'
+ wanoutkb2=869273840
update.sh: line 47: ("2653134149"-"2652444641") / 60 * 8: syntax error: operand expected (error token is ""2653134149"-"2652444641") / 60 * 8")
++ bc -l
+ inbits=
++ bc -l
++ echo 'scale=2;  / 1000000'
(standard_in) 1: syntax error
+ inmegabits=
update.sh: line 49: ("869273840"-"868626335") / 60 * 8: syntax error: operand expected (error token is ""869273840"-"868626335") / 60 * 8")
++ bc -l
+ outbits=
++ echo 'scale=2;  / 1000000'
++ bc -l
(standard_in) 1: syntax error
+ outmegabits=
++ bc -l
++ echo ' > 2'
(standard_in) 1: syntax error
+ ((    ))
+ echo 'There was a problem getting the data'
There was a problem getting the data
+ true
+ echo
I hade to edit the script a bit

Code: Select all

# Do math for Mbps
inbits=$(echo $((($waninkb2-$waninkb) / 60 * 8)) | bc -l)
inmegabits=$(echo "scale=2; $inbits / 1000000" | bc -l)
outbits=$(echo $((($wanoutkb2-$wanoutkb) / 60 * 8)) | bc -l)
outmegabits=$(echo "scale=2; $outbits / 1000000" | bc -l)
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
ya got it to work by changing the snmp script.
Added a variable fore the domoticz devices ID as well.
Then why not using crontab to loop the script ?

Still running the old script as it's showing the monthly download as well.
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
User avatar
Varazir
Posts: 360
Joined: Friday 20 February 2015 22:23
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by Varazir »

ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
I'm trying to get the old script to work. I think I messed it up. do you have a copy of the old ?
The new only shows the traffic I want to see the amount of data as well.
Raspberry PI 2 with RaZberry Controller 2016 ZWave+ and CC2531(zigbee)
Several IKEA devices/z-wave devices
ben53252642
Posts: 543
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: pfSense Internet Usage and Bandwidth

Post by ben53252642 »

Varazir wrote: Tuesday 11 December 2018 11:12
ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
I'm trying to get the old script to work. I think I messed it up. do you have a copy of the old ?
The new only shows the traffic I want to see the amount of data as well.
Sent as PM.

New users should be using the snmp version, it is much simpler and in my opinion more reliable.

Cheers
Unless otherwise stated, all my code is released under GPL 3 license: https://www.gnu.org/licenses/gpl-3.0.en.html
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests