Generate custom charts server side?

[nnylund]

Hi

I wonder if there is a way to generate the same custom charts available via the domoticz webGUI but on the server side? I wish to publish png's or jpg's on a static web page.
It seems domoticz use an online service (http://www.highcharts.com/) to generate the charts. Could the PhantomJS script be used to render charts on the server that is running domoticz?
http://www.highcharts.com/docs/export-m ... serverside

BR/Nylund

[nnylund]

I solved it by updating a rrd database using a lua script and generating graphs using rrdtool.

However it would be nice to be able to generate the domoticz charts server side using highcharts.
Does anyone know if it is possible?

[nnylund]

This is how I update the rrd database using lua if someone is interested in doing the same.

Code: Select all

function Round(num, idp)
   return tonumber(string.format("%." ..(idp or 0).. "f", num))
end

commandArray = {}

sUtetemp = Round(otherdevices_temperature['Ute'], 2)
sAcktanktopp = Round(otherdevices_temperature['Acktank topp'], 2)
sPannaturtemp = Round(otherdevices_temperature['Panndrift'], 2)
sRadkretsframled = Round(otherdevices_temperature['Framledning'], 2)
sSolreturtemp = Round(otherdevices_temperature['Från solfångare vid tank'], 2)

os.execute ('rrdtool update /mnt/templog/temp/templog.rrd N:'..sUtetemp..':'..sSolreturtemp..':'..sPannaturtemp..':'..sAcktanktopp..':'..sRadkretsframled)

return commandArray

[Toulon7559]

Interesting start to get graphs on a website.
Next step(s) also available?
= online making graphs in exportable format, and periodic upload of the graph using lua-script

Myself already some time looking for a similar 'trajectory' (see http://www.domoticz.com/forum/viewtopic ... 537#p97833 and http://www.domoticz.com/forum/viewtopic ... 23&t=14204 ), but not yet finished.
With Python-scripts able to upload the required data, but at the website-side still missing the functional link between that data and the website-graph.

[Egregius]

Store the values in mysql and use php.

[Toulon7559]

@Egregius

You mean following, or something different?
1) copy/upload SQL-database to server
2) run graphing PHP-script on server, taking data from the database

Somewhere examples to learn from?

[Toulon7559]

Example rddtool-implementation for a meteostation applying a pi raspberry:
https://www.ct.nl/softlink/1705042/
Functional chain:
1) use Adafruit-scripts for reading sensors
2) create rrd-database
3) update rrd-database
4) graphic output from rdd-database by rrdtool
5) copy/upload of the resulting png-graph to another folder

At the website the public summary is very short, but a scan of the article is available.

Looking in this thread nnylund has provided a lua-script to replace 1) and 3) of the above functional chain, making a direct link to Domoticz.

[Toulon7559]

Required steps related to nnylund's first input and the listing in my previous message:

1) Readout is performed by Domoticz
2) After preparation of the sh-file, the creation of the initial rrd-database can be once-only, manual action at the command line of Raspberry.
The example sh-file from ct2017#5 needs an adaptation to contain the information (configuration-dependent) from various sections of my/your Domoticz
3) Lua-script to be tuned to align with the layout of 2) to get the inputs in fitting format.
4) The graphic output is the png-file as requested by nnylund. The layout requires some fiddling with the example sh-file from ct2017#5 for alignment with the data in the rrd-database at one side, and with desired presentation at other side. Has some limitation/demands, because the graph has only 1 Y-axis/scale for all parameters to be shown.
5) Copy & upload of the graphic file can be separate by a dedicated, separate script or software.

Proof-of-concept achieved for a setup with 3 thermometers:
no cronjobs like in ct2017#5, but a lua-script directly calling 2 sh-files comparable to those described in ct2017#5. One sh-file updates the rrd, and the second produces the png-file. The call from the lua-script for both sh-files uses os.execute as handler.
Advantage: all control in 1 hand/lua-script.

RRD-graph for 3 temperatures

pitemp1_Domoticz.png (32.84 KiB) Viewed 3979 times

Next step is to 'mimick' the setup as described in ct2017#5 for a weatherstation, but now with temperature, humidity, pressure, rain and wind from TFA_Nexus and BMP180: RDD-update runs, but RDDTool-charting needs more stable solution, because (in my configuration) not running smooth & continuous, becoming visible as gaps in the graph-lines.
Need for rescaling of some values due to single Y-axis for all.
In the below graph the 'problem' of pressure-scale has been solved by subtraction of 1000, resulting in practical display of values ranging between +30 to -20, which is more or less in the range of possible temperatures.
Wind-info to be multiplied by 10 to become visible relative to the values for temperature and humidity.

RRD-graph for meteo

pimeteo2A_Domoticz.png (47.54 KiB) Viewed 3945 times

Status:
Nice, simple solution for collection and graphing of data. This setup can also be realised using Python or PHP, in which RDDTool can be embedded.
While the graphic output is simple to program, for combination of different types of values in one graph the scaling has demands to get good visibility of the values. In my own setup some issues with stable running of both sh-files: sometimes no update of the rrd, sometimes no graphing, or both.
Conclusions:
Not yet THE simple & robust solution for 'tailored' graphing from Domoticz.
And still a solution at client-side, not at server-side. Server-side to be preferred, because then one common solution possible for multiple clients.

[Toulon7559]

After further experiments (see e.g. this link), now the set of lua-script with related sh-files with RRDTool-application is 'Up & Running' at a Raspberry with Domoticz, related to a setup with Personal Weather Station type TFA_Nexus and a sensor type BMP180 hooked to the Raspberry.

Below are the 3 applicable software-files:

1) the sh-file generating the rrd-database-file
2) the sh-file generating the graph from the rrd-database-file
3) the lua-script extracting the data from Domoticz and controlling execution of the 2 sh-files, ultimately producing the graph.

For YOUR application obviously you have to adapt all file-names etc. according to YOUR configuration!

First you have to define what you want in the graph: that determines the minimum content and layout for the rdd-database-file.
The below setup is for the 'usual' 6 data-elements from a PWS, being temperature, humidty, pressure, wind speed and wind gust.
The numbers in the top-line and in the bottom-line define the layout of the RRA, and as consequence the layout of the graph.
Example-name (used in the lua-script!) of this file is create_pimeteo2_rrd.sh
Must be executable, therefore set properties to 755
Run this rrd-creation-file from the command line of the Raspberry.
That produces the basic rrd-file with the name in the second scriptline in directory /home/pi/

Code: Select all

#!/bin/bash
/usr/bin/rrdtool create /home/pi/pimeteo2.rrd --start N  --step 300 \
 DS:Temp1:GAUGE:600:U:U \
 DS:Humid1:GAUGE:600:U:U \
 DS:Baro1:GAUGE:600:U:U \
 DS:RainD:GAUGE:600:U:U \
 DS:WindS:GAUGE:600:U:U \
 DS:WindG:GAUGE:600:U:U \
 RRA:AVERAGE:0.5:1:3000

Second you have to make the sh-file for generation of the graph from the rrd-file.
Example-name (used in the lua-script!) is create_pimeteo2A_graph.sh
Also this file must be executable, therefore set properties to 755
In this script I solved the 'characteristic' of RDDTool of having only 1 Y-axis by
- deduction of 1000 from the pressure-value, resulting of effective scale-values between -30 and +30
- multiplication of wind-values by 10, putting them upto approx 70 for windspeed of 7m/s

Code: Select all

#!/bin/bash
rrdtool graph /home/pi/pimeteo2.png \
  DEF:temp=/home/pi/pimeteo2.rrd:Temp1:AVERAGE \
  DEF:humid=/home/pi/pimeteo2.rrd:Humid1:AVERAGE \
  DEF:press=/home/pi/pimeteo2.rrd:Baro1:AVERAGE \
  DEF:raind=/home/pi/pimeteo2.rrd:RainD:AVERAGE \
  DEF:winds=/home/pi/pimeteo2.rrd:WindS:AVERAGE \
  DEF:windg=/home/pi/pimeteo2.rrd:WindG:AVERAGE \
  CDEF:press2=press,1000,- \
  CDEF:winds2=winds,10,* \
  CDEF:windg2=windg,10,* \
  LINE1:temp#ff0000:Temperature \
  LINE1:humid#0000ff:Humidity \
  LINE1:press2#5fd00b:Pressure \
  LINE1:raind#00ffff:Rain_today \
  LINE1:winds2#000000:Windspeed \
  LINE1:windg2#871f78:Windgust \
  COMMENT:"\t\t\t\t\t\t\l" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:temp:LAST:"Temperature   Latest\: %2.1lf" \
  GPRINT:temp:MAX:"  Max.\: %2.1lf" \
  GPRINT:temp:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:humid:LAST:"Humidity      Latest\: %2.1lf" \
  GPRINT:humid:MAX:"  Max.\: %2.1lf" \
  GPRINT:humid:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:press:LAST:"Pressure      Latest\: %2.1lf" \
  GPRINT:press:MAX:"  Max.\: %2.1lf" \
  GPRINT:press:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:raind:LAST:"Rain_today    Latest\: %2.1lf" \
  GPRINT:raind:MAX:"  Max.\: %2.1lf" \
  GPRINT:raind:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:winds:LAST:"Windspeed     Latest\: %2.1lf" \
  GPRINT:winds:MAX:"  Max.\: %2.1lf" \
  GPRINT:winds:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  GPRINT:windg:LAST:"Windgust      Latest\: %2.1lf" \
  GPRINT:windg:MAX:"  Max.\: %2.1lf" \
  GPRINT:windg:MIN:"  Min.\: %2.1lf" \
  COMMENT:"\t\t\t\t\t\t\l" \
  --width 700 --height 400 \
  --title="Temperature, Humidity, Pressure, Rain & Wind for last 24 hour" \
  --vertical-label="Temp.(C)/Hum.(%)/Baro(hPa,-1000)/Rain(mm)/Wind(m/s,*10)" \
  --watermark "`date`"
# Copy picture to directory of local website
# cp /home/pi/pimeteo2.png /var/www/html

Finally the lua-script as required to link everything, which I called script_time_meteo2_fill_rrd_database_graph.lua, has following steps:
1) define the inputs
2) define the interval for graphing
3) extract, process & correct values, as required
4) put the values in a list with understandable and compatible labels as inputs for the rdd-file
5) update the rrd-file
6) perform graphing at the defined interval

Products of the lua-script are an updated rrd-file and a related png-file in directory /home/pi/
If you want to apply that png-file elsewhere, you need to copy or upload that file.
The sh-file for graphic_output ends with a copy-instruction, but unfortunately (in my setup) it reports an error, to be solved by you ....
For upload I apply a python-script or sync-software which 'grasps' the png-file.

As usual for my lua-scripts with line-numbers, print-lines and comments, which you might delete.

Code: Select all

------------------------------------------------------------------------------
-- Version 01C 20171111
--
-- Domoticz lua script to put values from Domoticz to RRD_Database. 
-- Extracts the contents based on the unique names of items, passes them
-- to the designated RRD_Database and generates a graph from selected data.
-- This version for collected, basic Meteo-info from TFA_Nexus & BMP180.
-- Relative to the setup described in ct2017#5, this script
-- within the lua-script updates the database and generates the graph,
-- by calling sh-files through os.execute 
------------------------------------------------------------------------------
-- Source-info from ct2017,#5 + Domoticz-forum, adapted by Toulon7559 (c)2017
------------------------------------------------------------------------------
print ('Start of meteo2 rrd-script 1C')
-- Line 15, Definition of Inputs
Outside_Temp_Hum = 'TFA1_Temp_RV_Hoog'
Barometer = 'BMP180_Temp_Baro'
RainMeter = 'TFA_Nexus_Neerslag'
WindMeter = 'TFA_Nexus_Wind'
-- Line 19, Settings
-- Intervals
Interval1 = 15; Offset1 = 1 -- Time in minutes to run create_rrd_graph

-- Line 23, Local Functions

local function Round(num, idp)
   return tonumber(string.format("%." ..(idp or 0).. "f", num))
end

-- Line 29, Data Extraction from Domoticz'svalues for TFA_Nexus
sTempTFA, sRVTFA = otherdevices_svalues[Outside_Temp_Hum]:match("([^;]+);([^;]+)")
sTempTFA = tonumber(sTempTFA);
sRVTFA = tonumber(sRVTFA);
-- Line 33 Data Extraction from Domoticz'svalues for BMP180
sBMP180T, sBMP180B = otherdevices_svalues[Barometer]:match("([^;]+);([^;]+)")
sBMP180T = tonumber(sBMP180T); -- Temperature at BMP180-sensor
sBMP180B = tonumber(sBMP180B); -- Pressure at BMP180-sensor
print ('BMP180_druk_raw = '.. sBMP180B)
-- Line 38, Checkline Outside_Temp_Hum
if Outside_Temp_Hum ~= '' then
   TempOut1 = sTempTFA
   HumOut1 = sRVTFA
end
-- Line 43, Checkline Barometer & Correction & Offset
a = 1 -- relative correction for air pressure in hPa, default is 1, range from 0.9 to 1.1 for max. 10% offset
b = 0 -- absolute correction for air pressure in hPa, default is 0, practical range from -10 to +10
if Barometer ~= '' then
    sBMP180B = a*sBMP180B + b
    print ('BMP180_corrected = '.. sBMP180B)
    PressOut1 = sBMP180B
    print ('BMP180_offset = '.. sBMP180B)
end
-- Line 52, Checkline for Rain-info
if RainMeter ~= '' then
   Rain_Day = otherdevices_rain[RainMeter]
   Rain_hour = otherdevices_rain_lasthour[RainMeter]
end
-- Line 57, Checkline for Wind-info
if WindMeter ~= '' then
   Winddir = otherdevices_winddir[WindMeter]
   Windspeedmps = otherdevices_windspeed[WindMeter] 
   Windgustmps = otherdevices_windgust[WindMeter] 
end

commandArray = {}

-- Line 66, Compilation & Print Temp&Hum&Baro
Temp1 = Round(TempOut1, 2); print ('Temp = '..Temp1) -- Outside temperature
Humid1 = Round(HumOut1, 2); print ('Humid = '..Humid1) -- Outside humidity
Baro1 = Round(PressOut1, 2); print ('Baro = '..Baro1) -- Barometric pressure
RainD = Round(Rain_Day, 2); print ('RainD = '..RainD) -- Rain today
-- RainH = Round(Rain_Hour, 2); print ('RainH = '..RainH) -- Rain last hour
-- WindD = Round(Winddir, 2); print ('WindD = '..WindD) -- Wind Direction
WindS = Round(Windspeedmps, 2); print ('WindS = '..WindS) -- Wind speed
WindG = Round(Windgustmps, 2); print ('WindG = '..WindG) -- Wind gust

-- Line 76, Export of data = update of rrd database(s) and generation of graph
-- Line 77, Extract current date as date.year, date.month, date.day, date.hour, date.min, date.sec
date = os.date("*t")
-- Line 79, Update of rrd database
-- RRD has internal setting for expecting update once per 5 minutes
   os.execute ('rrdtool update /home/pi/pimeteo2.rrd N:'..Temp1..':'..Humid1..':'..Baro1..':'..RainD..':'..WindS..':'..WindG)
   print ('RRD, Database updated!')
-- end
-- Line 84, Generation of graph
-- if (date.min % Interval1 == Offset1) then
   os.execute ('sudo /home/pi/create_pimeteo2A_graph.sh')
-- next line is required to set rw-permissions at all levels for open&copy&shift of the output-file 
--   os.execute("chmod a+rw /home/pi/pimeteo2.png")
   print ('RRD, Graph generated!')
-- end
print ('End of meteo2 rrd-script 1C')
return commandArray

[Toulon7559]

One significant side-effect noted for RRDTools in combination with Domoticz, Python etc.:
any gap in update of the RRD-database is mercilessly exposed in the graph.
Obviously is correct, but is a problematic aspect e.g. for continuous registration of data from Domoticz.

Although your data-source may have all data seamlessly & continuously available, a simple script will only take the latest data for the next update.
Example: if you update Domoticz, it takes some time to reboot, and that gap in operation of the scripts will become visible.
To avoid that gap in the read-out of the data-source and subsequent update of the RRD-database, the script must be much more clever, taking missing data from the data-source.
That defies the intention to keep things simple .......