Permanent storage of high resolution data Topic is solved

Topics (not sure which fora)
when not sure where to post, post here and mods will move it to right forum.

Moderators: leecollings, remb0

Post Reply
arraWX
Posts: 4
Joined: Wednesday 15 June 2016 15:27
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Permanent storage of high resolution data

Post by arraWX »

Hi

How can one permanently store all sensor readings reported by a sensor (e.g. temperature)?

A few days ago I installed domoticz on my windows pc. I use it with a temperature sensor and a gateway (both mysensors.org).

Next, the plan is to buy my first raspberry pi and run domoticz from there. And add more sensors. And add some actuators later on.

However, I would really like to permanently store all sensor readings reported by most of the sensors. The reason is that I would like to do some detailed data analysis using e.g. gnu octave or excel.
  • Can I store the data on the raspberry pi?
  • Is it suitable for that?
  • If not - what other options do I have?
  • Should I store in a text file or in a database?
I am a beginner so a simple solution is preferred.
georgesattali
Posts: 84
Joined: Saturday 05 March 2016 16:40
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: France
Contact:

Re: Permanent storage of high resolution data

Post by georgesattali »

Hello,
his topic maybe of some help : viewtopic.php?f=28&t=11631#p83856

Regards
GD
arraWX
Posts: 4
Joined: Wednesday 15 June 2016 15:27
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by arraWX »

Thank you georgesattali for pointing me to that topic. I will take a closer look at the lua script you posted there.
Last edited by arraWX on Saturday 25 June 2016 11:17, edited 1 time in total.
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

I recently created a small php script that drops all measurements into a seperate mysql database.
This way I can store ALL measurements and not only the average each 5 minutes.
I did this because in several weeks I will place cavity wall insulation so hopefully I can see some difference...

Code in my pass2php script:

Code: Select all

function logwrite($device,$value,$table) {
    file_get_contents('http://127.0.0.1/secure/logwrite.php?device='.$device.'&value='.$value.'&table='.$table);
}
function living_temp(){$this->logwrite(__FUNCTION__,$this->a,'temp');}
 
logwrite.php:

Code: Select all

<?php
if(isset($_REQUEST['device'])&&isset($_REQUEST['value'])&&isset($_REQUEST['table'])) logwrite($_REQUEST['device'],$_REQUEST['value'],$_REQUEST['table']);
function logwrite($device,$value,$table) {
    $time=microtime(true);$dFormat="Y-m-d H:i:s";$mSecs=$time-floor($time);$mSecs=substr(number_format($mSecs,3),1);
    $stamp = sprintf("%s%s",date($dFormat),$mSecs);
    $db = new mysqli('localhost', 'user', 'pass', 'database');
    if($db->connect_errno > 0){ die('Unable to connect to database [' . $db->connect_error . ']');}
    $query = "INSERT INTO `$table` (`stamp`,`device`,`value`) VALUES ('$stamp','$device','$value');";
    if(!$result = $db->query($query)) { die('There was an error running the query ['.$query .' - ' . $db->error . ']');}
    $db->close();
}
 
The graphs can be viewed with another php page and looks like this:
Image
arraWX
Posts: 4
Joined: Wednesday 15 June 2016 15:27
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by arraWX »

Thank you Egregius!
  • Where did you install mysql?
  • I plan to use the Domoticz RaspberryPi SD Image. Will it be possible to install and run mysql on the same raspberry pi?
  • Is the pass2php script written in lua?
  • Should that script be stored in the domoticz lua directory?
  • Where should the logwrite.php script be stored?
  • How to run it?
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

I'm running on a Synology DS1513+ wich has plenty of cpu and memory for stuff like that.
A RPI is also capable of running mysql server. The main thing is writing to database and reading isn't for tons of users.
Because the script is called by a file_get_contents it doens't mather where the page and database are located. Could even be on a cloud server.
For the pass2php part, see http://www.domoticz.com/forum/viewtopic ... 23&t=12343
The logwrite.php must be on a PHP enabled webserver wich also has the mysql server installation.
PjotrE
Posts: 9
Joined: Saturday 04 February 2017 21:32
Target OS: Linux
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by PjotrE »

Hi Egregius,

I have read this post several times for some time, and have also skimmed through your site at https://github.com/Egregius/LUA-Pass2PH ... stallation.

Finally I have decided to try to save my own measurements. I am very interested in using your proposed script to send all measurements for selected sensors to a mysql-database. It seems to be the best way to store values with correct time stamps, since all your work is focused on efficient programming.

Unfortunately I believe it would take me a lot of time to implement your functionality described at https://github.com/Egregius/LUA-Pass2PH ... stallation into my domoticz installation.

My hope is that using a standard Dotocicz installation on a standard Ubuntu server 16.04 (where for example php7 already is installed) I only need to place a modified script_device_pass2php.lua in /home/UserName/domoticz/scripts/lua, perhaps pointing to a pass2php.php in for example /home/UserName/domoticz/scripts...
With mysql installed on the same ubuntu server, I might also place the logwrite.php in the same place.

After that I am not sure what to do :? .
Unfortunately I am not really sure how to modify your proposed scripts either :oops: .

Is this even possible, or do you think it is better to include all your separate scripts, placing them in the apache web-site on the computer (/var/www/html/) as described in https://github.com/Egregius/LUA-Pass2PH ... stallation ?
If so I assume I should remove all files in /secure/pass2php/.. and only add a file for the device I want to store the measurements from.


Sorry for asking so many different questions in one post, but I really could use some help :) . If You read this and try to help I will be very grateful. Please don't be offended if my response seems slow, I just have many other things to fix at home :(
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

It all depends on what you exactly want to store. Do you want to store temps in realtime in a table with idx and temp? Or like me one record every minute with all temps in one record?
It may be even possible to not use pass2php at all, just a json request of a roomplan.
Loop thru the results in php and get the data you want.
Once the php variables are set you can easy store them in MySQL.
ropske
Posts: 483
Joined: Tuesday 12 August 2014 5:37
Target OS: Raspberry Pi / ODroid
Domoticz version: V3_8394
Location: Rumbeke,Belgium
Contact:

Re: Permanent storage of high resolution data

Post by ropske »

Egregius wrote: Tuesday 19 September 2017 6:48 It all depends on what you exactly want to store. Do you want to store temps in realtime in a table with idx and temp? Or like me one record every minute with all temps in one record?
It may be even possible to not use pass2php at all, just a json request of a roomplan.
Loop thru the results in php and get the data you want.
Once the php variables are set you can easy store them in MySQL.
Can you show us your php page with the graphical layout? :D
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

What graphical layout? Of my temperatures?
PjotrE
Posts: 9
Joined: Saturday 04 February 2017 21:32
Target OS: Linux
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by PjotrE »

I believe it would be good to have real time data sent to a separate database (mysql). My current settings for the 1-wire network is to check every 5 seconds. And the smaller time-frame the better to be able to calculate energy from the power-readings from my counters (10 000 imps/kWh).
As for the wireless temperatures from RFXtrx433 I can settle for longer periods of measurements, stored with time-stamps when received.

Since I plan to have more temperature and humidity sensors from the 1-wire network, it would be good to later just add them to what is passed on the the mysql-database. It feels right to store everything with a time stamp, that way I can have different frequency on different measurements and still be able to correlate them later on if needed.
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

It's not that hard to use a part of pass2php for this:

script_device_pass2php.lua (goes in the lua scripts folder):

Code: Select all

for d,s in pairs(devicechanged)
do
os.execute('curl -X POST -d "d='..d.."&s="..s..'" http://127.0.0.1/secure/pass2php.php &')
end
commandArray={}
return commandArray
The lua script posts the new status in realtime to the pass2php page. There they can be catched with $_REQUEST['d'] for the devicename and $_REQUEST['s'] for the status.

in /var/www/html/secure/pass2php.php just the minimal to catch those:

Code: Select all

<?php
if($_SERVER['REQUEST_METHOD']=='POST'){
	$device=$_POST['d'];
	$status=$_POST['s'];
	//Do something with the vars...
}
?>
For example post to a MySQL database the timestamp, devicename and the status:

Code: Select all

<?php
if($_SERVER['REQUEST_METHOD']=='POST'){
	$device=$_POST['d'];
	$status=$_POST['s'];
	$db=new mysqli('server','user','password','database');
	if($db->connect_errno>0){die('Unable to connect to database ['.$db->connect_error.']');}
	$query="INSERT IGNORE INTO `statusses` (`stamp`,`device`,`status`) VALUES (CURRENT_TIMESTAMP,'$device','$status');";
	if(!$result=$db->query($query)){die('There was an error running the query ['.$query.'-'.$db->error.']');}
}
?>
Problem here is that every status change would be stored in the database and so the database would grow fast. If you still want to add data in realtime but only for a few devices you add an if statement:

Code: Select all

<?php
if($_SERVER['REQUEST_METHOD']=='POST'){
	$device=$_POST['d'];
	$status=$_POST['s'];
	if(in_array($device,array('devicename1','devicename2','devicename3','devicename4','devicename5'))){
		$db=new mysqli('server','user','password','database');
		if($db->connect_errno>0){die('Unable to connect to database ['.$db->connect_error.']');}
		$query="INSERT IGNORE INTO `statusses` (`stamp`,`device`,`status`) VALUES (CURRENT_TIMESTAMP,'$device','$status');";
		if(!$result=$db->query($query)){die('There was an error running the query ['.$query.'-'.$db->error.']');}
	}
}
?>
What I did is posting the data to temporary files and then grab them together to put them in one record. This is then very easy to grab the data and create a grap with several temperatures. /var/log/cache is a tmpfs folder

Code: Select all

<?php
if($_SERVER['REQUEST_METHOD']=='POST'){
	$device=$_POST['d'];
	$status=$_POST['s'];
	if(in_array($device,array('buiten_temp','living_temp','badkamer_temp','kamer_temp','tobi_temp','alex_temp','zolder_temp'))){
		setstatus($device,$status);
	}
}
function setstatus($name,$value){file_put_contents('/var/log/cache/s'.$name.'.cache',$value);}
?>
And then combine the several statusses each minute with a cron job:

Code: Select all

<?php
$buiten_temp=status('buiten_temp');
$stamp=sprintf("%s",date("Y-m-d H:i"));
$living=status('living_temp');
$badkamer=status('badkamer_temp');
$kamer=status('kamer_temp');
$tobi=status('tobi_temp');
$alex=status('alex_temp');
$zolder=status('zolder_temp');
$query="INSERT IGNORE INTO `temp` (`stamp`,`buiten`,`living`,`badkamer`,`kamer`,`tobi`,`alex`,`zolder`) VALUES ('$stamp','$buiten_temp','$living','$badkamer','$kamer','$tobi','$alex','$zolder');";
$db=new mysqli('server','user','password','database');
if($db->connect_errno>0)die('Unable to connect to database ['.$db->connect_error.']');
if(!$result=$db->query($query))die('There was an error running the query ['.$query.'-'.$db->error.']');
?>
PjotrE
Posts: 9
Joined: Saturday 04 February 2017 21:32
Target OS: Linux
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by PjotrE »

Thanks very much for the time and effort to explain this for me Egregius!

I will then try to use your pass2php-concept and get-started files to get this up and running. First though I have to recompile Domoticz since I have a problem with values multiplied with 1000 when reading values from 1-wire counters.
I suspect it will take some time before I can bother you with additional questions :D

Thanks again, Best Regards
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

Just divide them by 1000 in php ;)
PjotrE
Posts: 9
Joined: Saturday 04 February 2017 21:32
Target OS: Linux
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by PjotrE »

Hi again,
You have no idea how many times I have started to formulate a question here, and while trying to explain what I have done, realizing an error of mine and then being able to continue :lol:

First regarding the 10^3 problem: Somewhere in Domoticz the 1-wire counter value is multiplied by 1000, and since my electrical meter sends 10000 imps/kWh it floods the variable to the limit. But I did a quick and dirty solution involving removing the battery and cable to the counter, thereby resetting it to zero. So that problem is now postponed..

Anyway, I have done lots of strange bad things on my ubuntu 16 server => creating errors, but finally managed to get this pass2php.php working to send temperature and humidity to a database and table I created in mysql. This is what it looks like at the moment:
<?php
ini_set('display_errors', 1);

if($_SERVER['REQUEST_METHOD']=='POST'){
$device=$_POST['d'];
$status=$_POST['s'];
if(in_array($device,array('Ute_Temperature','Ute_Humidity'))){
$db=new mysqli('localhost','nisse','thepassword','databas');
if($db->connect_errno>0){die('Unable to connect to database ['.$db->connect_error.']');}
$query="INSERT IGNORE INTO `sensors` (`tid`,`namn`,`status`) VALUES (CURRENT_TIMESTAMP,'$device','$status');";
if(!$result=$db->query($query)){die('There was an error running the query ['.$query.'-'.$db->error.']');}
}
}
?>


Using a seriously crazy way I found out that a sensor I have named Ute, has two different names in php: 'Ute_Temperature' and 'Ute_Humidity'. It is a double temperature/humidity wireless sensor.

It would be very helpful if there was some way of finding out all the names of the other sensors in this array called $device.
Do you perhaps have any simple way of finding that out? :?:
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Permanent storage of high resolution data

Post by Egregius »

Look in my pass2php code for the lg() function. That can write the device and stqtus to a logfile.

$device isn't an array, just a variable containing the name of the changed device.
PjotrE
Posts: 9
Joined: Saturday 04 February 2017 21:32
Target OS: Linux
Domoticz version:
Contact:

Re: Permanent storage of high resolution data

Post by PjotrE »

Thanks!
Modified the function lg I found in functions.php, and It seems to work just perfect.
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest