Page 1 of 3

Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 06 January 2021 0:23
by waltervl
I wanted to share this with you, but I have to tell you that I am no Grafana user, I just tested it for feasibility/usability.
Warning: I have no clue what the impact on performance is when Grafana is reading the Domoticz DB on regular time!!

Example dashboards made with Grafana
Selection_021.png
Selection_021.png (188.2 KiB) Viewed 10611 times

I found this Grafana plugin for SQLite https://github.com/fr-ser/grafana-sqlite-datasource
It makes the Domoticz SQLite database directly accessible in Grafana. So no need to use the Data Push to influxDB https://www.domoticz.com/wiki/Influxdb
You have to be knowledgeable of SQL to get usable data out of the Domoticz DB into Grafana.

Usage:
  • Install Grafana
    Install in Grafana the SQLite-datasource plugin (see instructions in plugin github)
The only required configuration is the path to the SQLite database (local path on the Grafana Server):
  • Add an SQLite datasource in Grafana.
    Set the path to the domoticz SQLite database (=domoticz.db in installation folder)
    Save the datasource and use it.
The Domoticz DB has tables where the logging of the day is stored for all the sensors. (Logging size/days depends on Dz Logging Short Log Sensors settings) Those tables have a Date Time field in it so that can be used but they have to be modified in Grafana.
I see the following usable tables (but it could be more):
  • Fan
    Meter
    Multimeter
    Percentage
    Rain
    SceneLog
    SetpointTimers
    Temperature
    UV
    Wind
Use following SQL Query in Grafana to get the data (in this example the temperatures of device 213:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temperature, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM Temperature WHERE DeviceRowID=213
)
SELECT datetime, Temperature FROM converted ORDER BY datetime ASC
convention:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT <value to chart>, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM <Domoticz Table> WHERE DeviceRowID=<Domoticz sensor IDX>
)
SELECT datetime, <value to chart> FROM converted ORDER BY datetime ASC
Set datetime as "time formatted column"

Quick Example in Grafana. There are more instructions on the internet to create Domoticz Grafana Dashboards.
Screenshot from 2021-01-06 00-01-08.png
Screenshot from 2021-01-06 00-01-08.png (75.71 KiB) Viewed 10710 times
Domoticz Temperature table (screenshot made with application sqlitebrowser)
Screenshot from 2021-01-05 23-51-50.png
Screenshot from 2021-01-05 23-51-50.png (85.54 KiB) Viewed 10710 times

Long term data

You can also use the long term data (per day) as present in the xxxxxx_Calender tables (xxxxx being the data source like temperature, multimeter etc)
So to display the avarage temperature of sensor IDX=213 you can follow the same method but as the date notation is different in this SQLite table, you have to convert it differently.

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- rename temp value to temp_idx to allow multiple queries (and so graph lines) labels
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temp_Avg AS Temp_213, Date  ||  'T00:00:00Z' AS datetime FROM Temperature_Calendar WHERE DeviceRowID=213
)
SELECT datetime, Temp_213 FROM converted ORDER BY datetime  ASC
Edit 28/1: I found an easy way to have multiple sensors show in one graph (Grafana time-series).
Example: To show temperature devices 11, 97 and 213 in one graph I used the following query.
There is an extra conversion needed as metric (in domoticz IDX) has to be a string and not a number. So I added Temp_ as a prefix. You are offcourse free to use your own prefixes....

Code: Select all

WITH converted AS (
   SELECT Temperature ,  REPLACE(Date, ' ','T') || 'Z' AS time, 'Temp_' || DeviceRowID AS metric FROM Temperature  Where  DeviceRowID in (11, 97, 213)
)
SELECT time, metric, Temperature FROM converted ORDER BY time
Have Fun!

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Wednesday 06 January 2021 11:52
by McMelloW
Looks very interesting?

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Thursday 07 January 2021 1:50
by plugge
Thank you for sharing!
waltervl wrote: Wednesday 06 January 2021 0:23 Warning: I have no clue what the impact on performance is whenGrafana is reading the Domoticz DB on regular time!!
I think a solution could be to use a (renamed) daily copy of the db from the backup directory?

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Thursday 07 January 2021 8:52
by waltervl
plugge wrote: Thursday 07 January 2021 1:50 Thank you for sharing!
I think a solution could be to use a (renamed) daily copy of the db from the backup directory?
Then you miss the data of today, which I think is why you want a dashboard...
If you would like to use Grafana to visualize historical data you can use the other tables. Domoticz db also containes the daily summary data from which you can do more research on yearly/monthly base.

As Grafana is only reading the database the impact is only there with limited CPU and Disk drive (SD card) resources. You will have to minimize the polling of the database, not every second but once per 5 min for example.

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Wednesday 20 January 2021 21:01
by waltervl
Edited the opening post for also graphing the long term sensor data and a method to distinguish sensor data from different devices.

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Monday 25 January 2021 12:14
by delius
functions perfectly. my setup is a dedicated ubuntu domoticz with a generic docker ubuntu to which I have now added Grafana.

Added Samba to the domoticz server for sharing the folder containing the domoticz.db. Then mounted this folder on the docker server and presented it to the Grafana docker-containter.

Temperature sensors are now shown in graphs, now look at other sensor types.

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 26 January 2021 15:41
by Stephanicz
Great post. It helped me a lot, because I can't get influxdb to work on my Pi. With the help of your code and my limited sql skills I get data from the Domoticz table into Grafana. I used a thermometer as an example and get the reading in a gauge or even a table. I can't get it to work with a graph. What am I doing wrong?

I used the code below Where 35 is my thermometer:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temperature, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM Temperature WHERE DeviceRowID=35
)
SELECT datetime, Temperature FROM converted ORDER BY datetime ASC
I get the error: Unable to Graph data switch to table view.

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 26 January 2021 17:01
by waltervl
Stephanicz wrote: Tuesday 26 January 2021 15:41 Great post. It helped me a lot, because I can't get influxdb to work on my Pi. With the help of your code and my limited sql skills I get data from the Domoticz table into Grafana. I used a thermometer as an example and get the reading in a gauge or even a table. I can't get it to work with a graph. What am I doing wrong?

I get the error: Unable to Graph data switch to table view.
It looks like Grafana is not accepting the time.
Did you set datetime as a time formatted column? See the screenshot in the opening post. You have to add it.

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Posted: Thursday 28 January 2021 23:29
by waltervl
delius wrote: Monday 25 January 2021 12:14 Temperature sensors are now shown in graphs, now look at other sensor types.
I found an easy way to have multiple sensors show in one graph (Grafana time-series).
Example: To show temperature devices 11, 97 and 213 in one graph I used the following query.
There is an extra conversion needed as metric (in domoticz IDX) has to be a string and not a number. So I added Temp_ as a prefix. You are offcourse free to use your own prefixes....

Code: Select all

WITH converted AS (
   SELECT Temperature ,  REPLACE(Date, ' ','T') || 'Z' AS time, 'Temp_' || DeviceRowID AS metric FROM Temperature  Where  DeviceRowID in (11, 97, 213)
)
SELECT time, metric, Temperature FROM converted ORDER BY time
Selection_023.png
Selection_023.png (82.35 KiB) Viewed 10512 times

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 09 March 2021 4:27
by dallas
Hi! For 2 weeks now I have been playing with the transfer of data from Domotics ... I am not a programmer, but I have already read a lot, although I do not understand everything. I don't know the syntax, everything is already merging into one heap during this time ...
I have Windows Server 16. In Domotics, I seem to have configured everything - SQL, HTTP. I installed and configured Grafana, but InfluxDB failed. Therefore, as you wrote, I installed SQLite, but I cannot make them friends! I changed all the data, read the information, substituted it, as I thought it was correct - it does not work! Tell me where is my error? And how can you put more than 20 sensors on one graph? Thanks for the answer!
Spoiler: show
GrafanaSQL.jpg
GrafanaSQL.jpg (25.22 KiB) Viewed 10218 times
DomoSQL.jpg
DomoSQL.jpg (48.94 KiB) Viewed 10218 times

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 09 March 2021 7:36
by waltervl
dallas wrote: Tuesday 09 March 2021 4:27Tell me where is my error? And how can you put more than 20 sensors on one graph? Thanks for the answer!
I am sorry, this is the wrong topic for you. For this is without influxdb. You could ask your question in this topic viewtopic.php?f=38

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 09 March 2021 8:37
by dallas
This topic is just for me - because I cannot and do not want to install Influx. I did everything according to your instructions. Tell me - what settings need to be made so that Domotics sends data directly to Grafana? In the photo I showed my Domotics settings. Grafana is almost set up ... I made a folder with daily backups in variable environments. What could be next? Thank you!
P.S. I thought to install Docker, but it seems to be not installed on systems below Windows 10 ...

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 09 March 2021 13:33
by waltervl
Domoticz is sending nothing to Grafana.
Grafana with SQLite plugin is pulling the values directly out of the Domoticz database wich is SQLite. No need to install SQLite itself. Just link Grafana with the datasource to the domoticz database (eg domoticz.db) that is placed in the installation folder. But I would strongly advise you to use a copy of this domoticz.db for test purposes.
But I do not know if this Grafana SQLite plugin is windows compatible. Were you able to install Grafana and install the plugin on Grafana?

Edit: I see your picture now and see the Grafana plugin. The path in plugin should be to the Domoticz.db file (could be a backup somewhere on your server)

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Tuesday 09 March 2021 22:59
by dallas
Good evening Waltervl!
Now I did something like this ... Below is a photo.

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 0:19
by waltervl
Grafana screenshot: You have to give the path including the database filename. So for example C:\x\x\xx\backups\daily\domoticz.db (check the filename)

Domoticz screenshot: You do not need the data push to influxdb. You can delete this. Not needed.

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 1:31
by dallas
Thanks for the tip! I thought about it, then tried adding the file extension and it worked! After that I looked at the forum and saw that you wrote the same! It works!!!
Only, as I understand it, Grafana works with the database, but does not collect new data from Domoticz? That's what Influx is for, right? Or can you somehow customize Grafana?
Thanks for your answers and help !!!

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 2:00
by waltervl
Great! If you use the main domoticz.db file as datasource and not a backup file you will always get the up to date data for your graphs.

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 4:22
by dallas
One more question - I just noticed that the data in the graphs does not accumulate, but decreases to the size of exactly 7 days, as in the program settings ... In the photo there are 2 hours difference between the computer and Grafana. Not sure how you can accumulate metrics? Or do I still need to install Influx?

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 8:09
by waltervl
Yes that is correct, it will show what is in the database. In domoticz after the keep log setting (for you 7 days) it will move the day summary to another table xxxxx_Calendar. If you want to see this data also you have to add it to your graph.

If you want to see the detailed history (per 5 minutes like the first 7 days) than you have to install influx db and use the data push to influxdb and make a graph on the influxdb data.

The difference in time could be the time difference in UTC for your location? What time is stored in the database?

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Posted: Wednesday 10 March 2021 13:25
by McMelloW
@Waltervl
Quick Example in Grafana. There are more instructions on the internet to create Domoticz Grafana Dashboards.
Can you give me the link on this. Are there any SQLite examples, almost all use the IfluxDB connection?