Rolling 24-hour graph using Google Charts and dzVents

Moderator: leecollings

Post Reply
MikeF
Posts: 350
Joined: Sunday 19 April 2015 0:36
Target OS: Raspberry Pi / ODroid
Domoticz version: V2022.2
Location: UK
Contact:

Rolling 24-hour graph using Google Charts and dzVents

Post by MikeF »

For some time I had been looking for a way to produce a rolling, 24-hour composite graph showing multiple temperatures from Domoticz. Sure, you can create a Custom Graph from within the Temperatures tab in Domoticz, but this does not appear to be persistent, i.e., you need to create it every time you want to see it.

I then came across Google Charts, which has enabled me to produce such a graph in a separate webpage in my browser. Here's a sample:
Image.

Data file

To produce this, I created a csv file with data from Domoticz every 10 minutes. I used a python script which adds the latest data and drops off the oldest data every 10 minutes (cron), thus maintaining 144 rows (24 x 6 per hour) plus header. For my requirement, the file looks like this:

Code: Select all

now, inside, target, outside, lounge, greenhouse, heating
10:30,19.3,20.0,5.2,18.6,6.0,0
10:40,19.6,20.0,5.3,18.8,6.3,0
10:50,19.7,20.0,5.4,18.8,6.7,0
11:00,19.8,19.0,5.5,18.9,7.0,0
etc. (oldest data first)
Script to create data file

However, the Python script seemed very clunky, and I asked (in this forum) if there was a better way of doing this in dzVents. @waaren kindly responded, and has created a dzVents script here:
https://www.domoticz.com/forum/viewtopi ... 72&t=30993
This approach is more elegant, and more tightly integrated with Domoticz.
You can have as many or as few data series as you want (but not too many - the graph will get very busy!) - just amend the devicenames / methods in the script (dataTable[1], etc.).

Script to create graphs

The script to create the graphs, in the form of an html file with Javascript code and Google Charts commands is here:
Spoiler: show

Code: Select all

<html>

<head>
<meta charset="utf-8">

<title>Temperatures from Domoticz</title>

<script src="http://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-csv/1.0.8/jquery.csv.min.js"></script>
<script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>

<script type="text/javascript"> // load the visualisation API

google.charts.load('current', { packages: ['corechart','gauge'] });
google.charts.setOnLoadCallback(drawTemps);
//google.charts.setOnLoadCallback(drawPower);

function drawTemps() {
   $.get("temps_daily.csv", function(csvString) {
      // transform the CSV string into a 2-dimensional array
      var arrayData = $.csv.toArrays(csvString, {onParseValue: $.csv.hooks.castToScalar});
	  
      // this new DataTable object holds all the data
      var data = new google.visualization.arrayToDataTable(arrayData);
      
      // get min, max values for series 3 - change this for a different series (counting from 0)
      var range = data.getColumnRange(3);
      
      var options = {
        explorer: {},
        title: 'Temperatures for last 24 hours',
        titleTextStyle: {color: 'grey', fontSize: 16},
        hAxis: {title: 'Time', titleTextStyle: {bold: false, italic: false, color: 'grey'}, slantedTextAngle: 90, textStyle: {fontSize: 12}},
        vAxis: {title: 'Degrees (C)', titleTextStyle: {bold: false, italic: false, color: 'grey'}},
        seriesType: 'line',
        interpolateNulls: true,
      	};
      
      var chart = new google.visualization.LineChart(document.getElementById('temps_div'));
      chart.draw(data, options);
      
      var columns = [];
      var series = {};
      for (var i = 0; i < data.getNumberOfColumns(); i++) {
        columns.push(i);
        if (i > 0) {
            series[i - 1] = {};
        }
      };
      
      google.visualization.events.addListener(chart, 'select', function () {
        var sel = chart.getSelection();
        // if selection length is 0, we deselected an element
        if (sel.length > 0) {
            // if row is undefined, we clicked on the legend
            if (sel[0].row === null) {
                var col = sel[0].column;
                if (columns[col] == col) {
                    // hide the data series
                    columns[col] = {
                        label: data.getColumnLabel(col),
                        type: data.getColumnType(col),
                        calc: function () {
                            return null
                        }
                    };
                    
                    // grey out the legend entry
                    series[col - 1].color = '#CCCCCC';
                }
                else {
                    // show the data series
                    columns[col] = col;
                    series[col - 1].color = null;
                }
                var view = new google.visualization.DataView(data);
                view.setColumns(columns);
                chart.draw(view, options);
            }
        }
      });

      var gauge_min_data = google.visualization.arrayToDataTable([
        ['Label', 'Value'],
        ['min', range.min]
      ]);
      
      var formatter = new google.visualization.NumberFormat(
    	{suffix: 'C', pattern: '##.#'}
	  );
	  formatter.format(gauge_min_data,1);
      
      var gauge_min_options = {
        width: 120, height: 120,
        min: -5, max: 35,
        redFrom: -5, redTo: 5,
        greenFrom: 15, greenTo: 25,
        yellowFrom: 25, yellowTo: 35,
        minorTicks: 5
      };
      
      var chart1 = new google.visualization.Gauge(document.getElementById('gauge_min_div'));
      chart1.draw(gauge_min_data, gauge_min_options);
      
      var gauge_max_data = google.visualization.arrayToDataTable([
        ['Label', 'Value'],
        ['max', range.max]
      ]);
      
	  formatter.format(gauge_max_data,1);
	  
      var gauge_max_options = {
        width: 120, height: 120,
        min: -5, max: 35,
        redFrom: -5, redTo: 5,
        greenFrom: 15, greenTo: 25,
        yellowFrom: 25, yellowTo: 35,
        minorTicks: 5
      };
      
      var chart2 = new google.visualization.Gauge(document.getElementById('gauge_max_div'));
      chart2.draw(gauge_max_data, gauge_max_options);
      
    },'text');
      
}


function GetClock(){
	var d=new Date();
	var nmonth=d.getMonth(),ndate=d.getDate(),nyear=d.getFullYear();
	var nhour=d.getHours(),nmin=d.getMinutes(),ap;
	
	if(nhour==0){ap=" AM";nhour=12;}
	else if(nhour<12){ap=" AM";}
	else if(nhour==12){ap=" PM";}
	else if(nhour>12){ap=" PM";nhour-=12;}

	if(nmin<=9) nmin="0"+nmin;

	document.getElementById('clockbox').innerHTML=""+ndate+"/"+(nmonth+1)+"/"+nyear+" "+nhour+":"+nmin+ap+"";
}

window.onload=function(){
	GetClock();
	setInterval(GetClock,1000);
	setInterval(drawTemps, 60000);
//	setInterval(drawPower, 60000);
}

</script>

<style>
	table, th, td {
    	border: 2px solid #ccc;
    	border-collapse: collapse;
    	font-family: verdana;
    	font-size: 16px
	}
	th, td {
    	padding: 5px;
    	text-align: center;    
	}
	td {
		font-size: 12px
	}
	.graph {
  		width: 1000;
  		height: 400;
	}
	.meter {
  		padding: 15px;
  		width: 130;
  		height: 150;
  		font-size: 14px;
	}
</style>

</head>

<body>

<table>
  	<tr>
		<th>Temperatures from Domoticz</td>
		<td><div id="clockbox" style="font-size: 100%"></div></td>
  	</tr>
  	<tr>
		<td rowspan = "2"><div id="temps_div" class="graph"></div></td>
		<td>Outside<div id="gauge_min_div" class="meter"></div></td>
  	</tr>
  	<tr>
  		<td><div id="gauge_max_div" class="meter"></div></td>
  	</tr>

</table>

</body>

</html>
In terms of customisation, you only need to change line 19 for the name of your csv file (and see below, re webserver). However, there are also gauges in the chart, which show the minimum and maximum values for data series 3 - you can change this at line 27.
You can hover over any series, and Google Charts will display values in tooltips.
There is a GetClock function within the script, which updates the webpage every 10 minutes (as data is only updated at the same rate).

Webserver

To display the graphs, I set up a webserver using nginx - see instructions for Raspberry Pi here:
https://www.raspberrypi.org/documentati ... r/nginx.md
I created a separate www folder, and saved the above chart script as index.html inside it. I also changed the location / name of the csv file in the dzVents script so that it creates the csv file within this folder. As per the nginx instructions, I changed the location of the webpage in /etc/nginx/sites-available/default.

So, in my setup I have created a folder /home/pi/devices/www, which contains:

Code: Select all

index.html
temps_daily.csv
The output line in the dzVents script is changed to:

Code: Select all

         local output = '/home/pi/devices/temps_daily.csv' -- location/name of your csv text file
and the webpage location in /etc/nginx/sites-available/default is changed to:

Code: Select all

        root /home/pi/devices/www;
I apologise if this seems very long-winded - I will try and produce a TL;DR; version!

Anyway, if you find this useful or if you have any queries, please let me know.
Nautilus
Posts: 722
Joined: Friday 02 October 2015 12:12
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Finland
Contact:

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by Nautilus »

Hi, I was looking for something like this so this is very intersting. I was thinking to create a custom page of this to Domoticz (in ~domoticz/www/templates/) but it looks like it is quite tricky to get it load the .js - I guess because the header etc. where scripts are typically loaded are in the main index.html and the custom page only gets injected in the middle of this. So I am getting “Uncaught ReferenceError: google is not defined” indicating it is not loading the .js. Do you know (or anyone else) if there is some workaround for this?

Edit. I think it is easiest if I just put it in iframe. This at least seems to work :)
michah
Posts: 4
Joined: Saturday 02 May 2020 15:54
Target OS: Raspberry Pi / ODroid
Domoticz version: dev-fork
Location: The Netherlands
Contact:

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by michah »

Hi MikeF and Nautilus,

I am currently reconstructing the inner workings that produce the charts in Domoticz and adding autorefresh (or the 'rolling' feature as you call it) while I'm at it. It's still in the beta version though. Maybe this can be of interest to you.

The current charts are created using Highcharts. I'm not familiar with Google Charts. Does Google Charts have a specific feature that you need?

Regards, Micha
Nautilus
Posts: 722
Joined: Friday 02 October 2015 12:12
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Finland
Contact:

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by Nautilus »

michah wrote: Friday 23 October 2020 13:18 Hi MikeF and Nautilus,

I am currently reconstructing the inner workings that produce the charts in Domoticz and adding autorefresh (or the 'rolling' feature as you call it) while I'm at it. It's still in the beta version though. Maybe this can be of interest to you.

The current charts are created using Highcharts. I'm not familiar with Google Charts. Does Google Charts have a specific feature that you need?

Regards, Micha
Sounds excellent! :)

I like highcharts as well, I don't think there is any particular feature that would be needed from google charts. What I would like to see is some customization options for the chart, best would be to have something that can be done on installation level for default colors etc. and on chart level in case you need to override them. And it would also be great if one could request chart as an image through the API. I'd like to automate a daily summary to Telegram of how the temperatures in my cabin have changed. Currently I'm picking high, low and average as text from the past 24h, but it would be even better to be able to attach an image of a chart to accompany those.. :)
michah
Posts: 4
Joined: Saturday 02 May 2020 15:54
Target OS: Raspberry Pi / ODroid
Domoticz version: dev-fork
Location: The Netherlands
Contact:

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by michah »

Nautilus wrote: Friday 23 October 2020 14:24
I like highcharts as well, I don't think there is any particular feature that would be needed from google charts. What I would like to see is some customization options for the chart, best would be to have something that can be done on installation level for default colors etc. and on chart level in case you need to override them. And it would also be great if one could request chart as an image through the API. I'd like to automate a daily summary to Telegram of how the temperatures in my cabin have changed. Currently I'm picking high, low and average as text from the past 24h, but it would be even better to be able to attach an image of a chart to accompany those.. :)
That's a nice idea! Something like a cascading highcharts configuration, the way CSS works. That would be a nice addition. It would require some time to build. Perhaps a one or two-level highcharts configuration template would suffice. I'll think about it.

About the chart images, I just found this highcharts-export-server. That would be useful. You could take a look at that!
Nautilus
Posts: 722
Joined: Friday 02 October 2015 12:12
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: Finland
Contact:

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by Nautilus »

Yes I can certainly understand it would require some time to build. But maybe at some point...:)

I think the export server is already used in Domoticz as you can export the chart to e.g. png/jpeg/pdf. But this export is not in any way exposed to the Domoticz API and I'm not sure how I would be able to use it in scripts. I just see that it sends s POST request to export.highcharts.com and I guess the payload is the data that is needed for the image (i.e. the full html). This would be very difficult to parse together e.g. from the standard API call "json.htm?idx=IDX&range=day&sensor=temp&type=graph". But yes, with some skills I guess this would already be doable with the current functionalities available :)
User avatar
frank666
Posts: 17
Joined: Monday 07 December 2020 10:11
Target OS: Linux
Domoticz version: 2024.7
Location: San Marino

Re: Rolling 24-hour graph using Google Charts and dzVents

Post by frank666 »

following some posts and your indications, I was able in part to create a chart (similar to that of domoticz made with highcharts) online.
I did so
  • inserted in domoticz the script dzvents written by @waaren
    Spoiler: show

    Code: Select all

    --[[
        script to create a CSV file with following layout.
        
        now,inside,target,outside,lounge,greenhouse,heating  [header row]
        22:10,21.2,7.0,12.7,21.3,11.8,0   [oldest data: 24 hours minus 10 mins. ago]
        22:20,21.1,7.0,12.7,21.3,11.8,0   [next data: 10 mins. later]
        ...
        22:00,21.1,7.0,9.4,22.0,8.0,0     [latest data]
    
            History:
            20200108: Start coding
            20200110: Start testing on life situation (thanks to MikeF)
            20200122: First public release
    
    ]]--
    
    return
    {
        on = 
        { 
            timer = 
            {
                -- already quite busy at round minutes so shift a couple of minutes. Choose the one you prefer by uncommenting the line
                'at *:03', 'at *:18', 'at *:23', 'at *:33', 'at *:43', 'at *:53', 
                -- 'every 10 minutes', 
            }, 
        }, 
            
        data = 
        {  
            rows = 
            { 
                history = true, maxItems = 300, maxHours = 24, -- max is whatever is reached first (what will be written to csv is a subset of this) 
            }
        }, 
        
        logging =   
        {
            level = domoticz.LOG_DEBUG, -- set to LOG_ERROR when tested and OK
            marker = 'createCSV', 
        }, 
           
        execute = function(dz)
            
            local dataTable = {}
    
            -----------------------------------------------------------------------------------------------------------  
            -- Enter header line of your csv text file (or nil if you don't need one ) 
            -- Enter devicenames / methods in the required order. This will determine what attribute-values will be written to the csvFile 
            -- Enter location/name of your csv text file 
            -- Enter time-span to be written to csv file
            -----------------------------------------------------------------------------------------------------------  
            local header =  'time,temperature' 
            
            dataTable[1] = dz.time.rawTime:sub(1, 4) .. '0' -- consistent times even if script is not triggered precisely every x0 minutes
            dataTable[2] = dz.utils.round(dz.devices('bmp280').temperature, 1) 
            local output = '/tmp/temperature.csv' -- location/name of your csv text file
            local span   = '23:59:00'  -- time-span to be written to csv file
            -----------------------------------------------------------------------------------------------------------  
            -- No changes required below these lines
            -----------------------------------------------------------------------------------------------------------  
            
            local function makePersistentData()    
                local row = {}
                for index, value in ipairs(dataTable) do
                    row[index] = value
                end          
                return row 
            end
    
            local function writeCSV()
                local relevantRows = dz.data.rows.subsetSince(span)
                local csvFile = assert(io.open(output, "w"))
                if header then csvFile:write(header ..'\n') end
               
                for index = #relevantRows, 1, -1 do 
                    dz.log(table.concat(relevantRows[index].data, ','), dz.LOG_DEBUG) 
                    csvFile:write(table.concat(relevantRows[index].data, ',') ..'\n')  
                end
    
                csvFile:close()
                dz.log((#relevantRows) .. ' data-rows written.', dz.LOG_DEBUG) 
            end
     
            -- main
            dz.data.rows.add(makePersistentData())  -- store in persistent historical data
            writeCSV()
        end
    }
  • exported to the site with cron ----> sftp the daily file temperature.csv
  • created using the free online editor http://editor.highcharts.com/full.html exported to the site and I get this chart.
    Schermata del 2020-12-14 09.08.00.png
    Schermata del 2020-12-14 09.08.00.png (63.55 KiB) Viewed 1423 times
  • Emulating the daily chart of domoticz I have these problems :
    • in the tooltips and in the title of the graph, how to insert the date
    • in case of sensor lock, the last temperature is shown several times in the graph until the sensor is unlocked (how to insert an interruption)
    • how to insert the zoom of a part of the graph
    • on the horizontal axis how to insert only the hourly values and at midnight the day.
    • you could also insert the barometer values in a single graph and 2 splines
Domoticz running on Docker,Orange Pi Zero Plus
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests