Page 1 of 2

pfSense Internet Usage and Bandwidth

Posted: Saturday 25 February 2017 1:45
by ben53252642
Hey Folks,

This is a bash script I wrote for getting the last minutes bandwidth information from pfsense and loading it into Domoticz.

Steps:
1) Go to this url on your pfsense box: https://pfsenseip/services_snmp.php
2) Tick the box to enable snmp and set the "Read Community String" to: "public" then scroll down and bind it to the LAN interface and press save.
3) Install requirements on your Domoticz server: apt-get install snmp bc
3) Fill in everything in the script except for the "waninterfacename"
3) Run the script like this: "./update.sh list" and you will get an output looking something like this:

vtnet0
vtnet1
enc0
lo0
pflog0
pfsync0
pppoe0
ovpns1

In this case my wan interface is called pppoe0 so what is what I would enter in "waninterfacename" in the script (go ahead and fill the script with the name of your wan interface).

update.sh

Code: Select all

#!/bin/bash

# Pfsense Configuration
pfsenseip="192.168.0.1"
snmpid="public"
snmpversion="2c"
waninterfacename="pppoe0"
measuretime="60" # If you want avg mbps per minute, this would be 60

# Domoticz Configuration
domoticzserver="192.168.0.5"
domoticzport="443"
domoticzuser="username"
domoticzpass="password"

# Identify the WAN interface
interfacelist=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.2)
wansnmpid=$(echo "$interfacelist" | grep "$waninterfacename" | awk '{ print $1 }' | grep -o '.\{1\}$')

# Get list of interfaces and display them in terminal
if [[ $1 == 'list' ]]; then
echo "$interfacelist" | awk '{ print $4 }' | sed 's/\"//g'
exit
fi

# Start work loop
while true; do

# Notify terminal
echo && echo "Interface: $waninterfacename"
echo "Calculating bandwidth per $measuretime seconds in mbps... (wait $measuretime seconds)"

# Get data
waninkb=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.10."$wansnmpid" | awk '{ print $4 }')
wanoutkb=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.16."$wansnmpid" | awk '{ print $4 }')

# Wait time period
sleep "$measuretime"

# Get data a second time
waninkb2=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.10."$wansnmpid" | awk '{ print $4 }')
wanoutkb2=$(snmpwalk -c "$snmpid" -v "$snmpversion" "$pfsenseip" .1.3.6.1.2.1.2.2.1.16."$wansnmpid" | awk '{ print $4 }')

# Do math for Mbps
inbits=$(echo $((("$waninkb2"-"$waninkb") / 60 * 8)) | bc -l)
inmegabits=$(echo "scale=2; $inbits / 1000000" | bc -l)
outbits=$(echo $((("$wanoutkb2"-"$wanoutkb") / 60 * 8)) | bc -l)
outmegabits=$(echo "scale=2; $outbits / 1000000" | bc -l)

# Check data viability
if (( $(echo "$inbits > 2" |bc -l) )) && (( $(echo "$outbits > 2" |bc -l) )); then

# Notify Terminal 2
echo "In: $inmegabits Mbps"
echo "Out: $outmegabits Mbps"

# Send data to Domoticz
curl --max-time 60 -s -k "https://{$domoticzuser}:{$domoticzpass}@{$domoticzserver}:{$domoticzport}/json.htm?type=command&param=udevice&idx=503&svalue=${inmegabits}"
curl --max-time 60 -s -k "https://{$domoticzuser}:{$domoticzpass}@{$domoticzserver}:{$domoticzport}/json.htm?type=command&param=udevice&idx=502&svalue=${outmegabits}"

# Report if there was a problem
else
echo "There was a problem getting the data"
fi

done
This is the startup.sh script I use for running it in the background:

Code: Select all

# Start pfsense monitor
/usr/bin/screen -S pfsense-monitor -d -m /root/pfsense/update.sh
screenshot112532.JPG
screenshot112532.JPG (77.33 KiB) Viewed 8371 times
terminaloutput1534.JPG
terminaloutput1534.JPG (41.67 KiB) Viewed 8367 times

Re: pfSense Internet Usage and Bandwidth

Posted: Sunday 12 March 2017 10:33
by Varazir
Great script but wouldn't be good to set the IDX as variables too?

Abit howto setup the dummy devices too.

I added it to the crontab so I don't need to start the screen if I reboot my server.
Removed the while loop and sleep

Added custom icons

Re: pfSense Internet Usage and Bandwidth

Posted: Sunday 12 March 2017 16:52
by Varazir
and this icons

They look like this
pfsense.JPG
pfsense.JPG (51.15 KiB) Viewed 11022 times

Re: pfSense Internet Usage and Bandwidth

Posted: Wednesday 29 March 2017 13:47
by ben53252642
Awesome! Love the icons! :D

If anyone finds a simpler way to get the data out of pfsense please let me know.

Re: pfSense Internet Usage and Bandwidth

Posted: Monday 06 November 2017 11:03
by void
Anyone tried with the 2.4 branch? Some things got a major update, such as the login screen.

Re: pfSense Internet Usage and Bandwidth

Posted: Monday 13 November 2017 9:13
by ben53252642
void wrote: Monday 06 November 2017 11:03 Anyone tried with the 2.4 branch? Some things got a major update, such as the login screen.
Can confirm that existing script works fine with 2.4.* branch.

Re: pfSense Internet Usage and Bandwidth

Posted: Wednesday 31 January 2018 9:21
by void
Did you consider creating a Wiki article for it at http://www.domoticz.com/wiki/Main_Page ?
Might be interessting for more people, and the forum search function kind of sucks... :)

Re: pfSense Internet Usage and Bandwidth

Posted: Thursday 01 February 2018 14:55
by ben53252642
void wrote: Wednesday 31 January 2018 9:21 Did you consider creating a Wiki article for it at http://www.domoticz.com/wiki/Main_Page ?
Might be interessting for more people, and the forum search function kind of sucks... :)
I tried to add something to the wiki recently but was unable to create an account:

viewtopic.php?f=43&p=167358#p167358

I think for now this is ok in the Forum, also people can post if they have problems and it's easy to reply.

When I can access the Wiki perhaps I'll make a page directing people here, Ultimately it would be better if there was a native Domoticz plugin. Until then we have this bash script.

Re: pfSense Internet Usage and Bandwidth

Posted: Thursday 15 March 2018 1:27
by MnM001
Hi guys looking for some help here.

Where I run the script I get:

./test.sh: line 19: /usr/bin/curl: Argument list too long
error : Unknown IO error

The script does log into my pfsense as I can see it in my logs.
DarkStat and Totals configured correctly.

I am a beginner with scripting so I will appreciate if someone can help me resolve this error.

Thanks

Re: pfSense Internet Usage and Bandwidth

Posted: Saturday 24 March 2018 20:04
by domoticzcom1234
This can be caused by many things. Answering your question therefore is not very easy.
This implementation can be very complicated and is time consuming to get it working.

The only thing I can do is give you a basic guidance as listed below:
- Did you create an account on Pfsense
- Did you gave the correct permissions to access the right modules
- login as the new user and password and run this in your browser: https://IP-ADDRESS-OF-YOUR-PFSENSE/vnst ... h_json.php, are you getting the right response? No?, your permissions are set wrong.
- Are you using the right PFsense network interface and is it in the same order listed in the PFsense URL in the script (try this in the browser first as listed above)
- did you installed jq, bc AND libxml2-utils
- are you using the latest version of pfsense
- do you have the right permissions in linux

And the best one to check:
- run the script with the -x such as "sh -x scriptname.sh" so you can check what goes wrong



Good luck and this is the only thing I can do.

Re: pfSense Internet Usage and Bandwidth

Posted: Tuesday 10 April 2018 23:07
by ben53252642
I'm in the process of completely re-writing this to work from SNMP and not require either bandwidth totals or darkstat on Pfsense.

I've completed the bandwidth part, it's currently installed on my Domoticz box updating nicely every minute.

I'm now working on the monthly bandwidth calculations. Should be posted in the next couple of days, stay tuned.

Re: pfSense Internet Usage and Bandwidth

Posted: Saturday 21 April 2018 10:31
by Varazir
ben53252642 wrote: Tuesday 10 April 2018 23:07 I'm in the process of completely re-writing this to work from SNMP and not require either bandwidth totals or darkstat on Pfsense.

I've completed the bandwidth part, it's currently installed on my Domoticz box updating nicely every minute.

I'm now working on the monthly bandwidth calculations. Should be posted in the next couple of days, stay tuned.
Sweet

Re: pfSense Internet Usage and Bandwidth

Posted: Thursday 26 April 2018 23:45
by ben53252642
I've updated the first post with the new code which I believe is significantly better than the previous version.

Note that for now it's only the last minutes bandwidth (not monthly downloads), I'm still working on a better way to do that.

Re: pfSense Internet Usage and Bandwidth

Posted: Wednesday 02 May 2018 21:55
by Varazir
ben53252642 wrote: Thursday 26 April 2018 23:45 I've updated the first post with the new code which I believe is significantly better than the previous version.

Note that for now it's only the last minutes bandwidth (not monthly downloads), I'm still working on a better way to do that.
will not Domoticz keep track of the monthly data traffic ? ?

Re: pfSense Internet Usage and Bandwidth

Posted: Wednesday 07 November 2018 18:17
by Varazir
ben53252642 wrote: Saturday 25 February 2017 1:45 Hey Folks,
Maybe create a variable for the device ID in domoticz ?

Re: pfSense Internet Usage and Bandwidth

Posted: Wednesday 07 November 2018 18:29
by Varazir
getting error

Code: Select all

++ awk '{ print $4 }'
+ wanoutkb2=869273840
update.sh: line 47: ("2653134149"-"2652444641") / 60 * 8: syntax error: operand expected (error token is ""2653134149"-"2652444641") / 60 * 8")
++ bc -l
+ inbits=
++ bc -l
++ echo 'scale=2;  / 1000000'
(standard_in) 1: syntax error
+ inmegabits=
update.sh: line 49: ("869273840"-"868626335") / 60 * 8: syntax error: operand expected (error token is ""869273840"-"868626335") / 60 * 8")
++ bc -l
+ outbits=
++ echo 'scale=2;  / 1000000'
++ bc -l
(standard_in) 1: syntax error
+ outmegabits=
++ bc -l
++ echo ' > 2'
(standard_in) 1: syntax error
+ ((    ))
+ echo 'There was a problem getting the data'
There was a problem getting the data
+ true
+ echo
I hade to edit the script a bit

Code: Select all

# Do math for Mbps
inbits=$(echo $((($waninkb2-$waninkb) / 60 * 8)) | bc -l)
inmegabits=$(echo "scale=2; $inbits / 1000000" | bc -l)
outbits=$(echo $((($wanoutkb2-$wanoutkb) / 60 * 8)) | bc -l)
outmegabits=$(echo "scale=2; $outbits / 1000000" | bc -l)

Re: pfSense Internet Usage and Bandwidth

Posted: Thursday 08 November 2018 19:08
by ben53252642
Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.

Re: pfSense Internet Usage and Bandwidth

Posted: Sunday 11 November 2018 14:06
by Varazir
ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
ya got it to work by changing the snmp script.
Added a variable fore the domoticz devices ID as well.
Then why not using crontab to loop the script ?

Still running the old script as it's showing the monthly download as well.

Re: pfSense Internet Usage and Bandwidth

Posted: Tuesday 11 December 2018 11:12
by Varazir
ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
I'm trying to get the old script to work. I think I messed it up. do you have a copy of the old ?
The new only shows the traffic I want to see the amount of data as well.

Re: pfSense Internet Usage and Bandwidth

Posted: Tuesday 11 December 2018 13:04
by ben53252642
Varazir wrote: Tuesday 11 December 2018 11:12
ben53252642 wrote: Thursday 08 November 2018 19:08 Varazir did you get it working?

I think the snmp version is a lot better, I've not had to do anything since I've put it in place on my system, it's just kept working as expected.
I'm trying to get the old script to work. I think I messed it up. do you have a copy of the old ?
The new only shows the traffic I want to see the amount of data as well.
Sent as PM.

New users should be using the snmp version, it is much simpler and in my opinion more reliable.

Cheers