Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Client tools or tools that can connect with Domoticz. Tools for Windows, iOS, Android, Linux etc.

Moderator: leecollings

User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

I wanted to share this with you, but I have to tell you that I am no Grafana user, I just tested it for feasibility/usability.
Warning: I have no clue what the impact on performance is when Grafana is reading the Domoticz DB on regular time!!

Example dashboards made with Grafana
Selection_021.png
Selection_021.png (188.2 KiB) Viewed 10613 times

I found this Grafana plugin for SQLite https://github.com/fr-ser/grafana-sqlite-datasource
It makes the Domoticz SQLite database directly accessible in Grafana. So no need to use the Data Push to influxDB https://www.domoticz.com/wiki/Influxdb
You have to be knowledgeable of SQL to get usable data out of the Domoticz DB into Grafana.

Usage:
  • Install Grafana
    Install in Grafana the SQLite-datasource plugin (see instructions in plugin github)
The only required configuration is the path to the SQLite database (local path on the Grafana Server):
  • Add an SQLite datasource in Grafana.
    Set the path to the domoticz SQLite database (=domoticz.db in installation folder)
    Save the datasource and use it.
The Domoticz DB has tables where the logging of the day is stored for all the sensors. (Logging size/days depends on Dz Logging Short Log Sensors settings) Those tables have a Date Time field in it so that can be used but they have to be modified in Grafana.
I see the following usable tables (but it could be more):
  • Fan
    Meter
    Multimeter
    Percentage
    Rain
    SceneLog
    SetpointTimers
    Temperature
    UV
    Wind
Use following SQL Query in Grafana to get the data (in this example the temperatures of device 213:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temperature, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM Temperature WHERE DeviceRowID=213
)
SELECT datetime, Temperature FROM converted ORDER BY datetime ASC
convention:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT <value to chart>, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM <Domoticz Table> WHERE DeviceRowID=<Domoticz sensor IDX>
)
SELECT datetime, <value to chart> FROM converted ORDER BY datetime ASC
Set datetime as "time formatted column"

Quick Example in Grafana. There are more instructions on the internet to create Domoticz Grafana Dashboards.
Screenshot from 2021-01-06 00-01-08.png
Screenshot from 2021-01-06 00-01-08.png (75.71 KiB) Viewed 10712 times
Domoticz Temperature table (screenshot made with application sqlitebrowser)
Screenshot from 2021-01-05 23-51-50.png
Screenshot from 2021-01-05 23-51-50.png (85.54 KiB) Viewed 10712 times

Long term data

You can also use the long term data (per day) as present in the xxxxxx_Calender tables (xxxxx being the data source like temperature, multimeter etc)
So to display the avarage temperature of sensor IDX=213 you can follow the same method but as the date notation is different in this SQLite table, you have to convert it differently.

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- rename temp value to temp_idx to allow multiple queries (and so graph lines) labels
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temp_Avg AS Temp_213, Date  ||  'T00:00:00Z' AS datetime FROM Temperature_Calendar WHERE DeviceRowID=213
)
SELECT datetime, Temp_213 FROM converted ORDER BY datetime  ASC
Edit 28/1: I found an easy way to have multiple sensors show in one graph (Grafana time-series).
Example: To show temperature devices 11, 97 and 213 in one graph I used the following query.
There is an extra conversion needed as metric (in domoticz IDX) has to be a string and not a number. So I added Temp_ as a prefix. You are offcourse free to use your own prefixes....

Code: Select all

WITH converted AS (
   SELECT Temperature ,  REPLACE(Date, ' ','T') || 'Z' AS time, 'Temp_' || DeviceRowID AS metric FROM Temperature  Where  DeviceRowID in (11, 97, 213)
)
SELECT time, metric, Temperature FROM converted ORDER BY time
Have Fun!
Last edited by waltervl on Monday 09 August 2021 14:32, edited 9 times in total.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
User avatar
McMelloW
Posts: 427
Joined: Monday 20 November 2017 17:01
Target OS: Raspberry Pi / ODroid
Domoticz version: V2024.1
Location: Harderwijk, NL
Contact:

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by McMelloW »

Looks very interesting?
Greetings McMelloW
plugge

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by plugge »

Thank you for sharing!
waltervl wrote: Wednesday 06 January 2021 0:23 Warning: I have no clue what the impact on performance is whenGrafana is reading the Domoticz DB on regular time!!
I think a solution could be to use a (renamed) daily copy of the db from the backup directory?
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by waltervl »

plugge wrote: Thursday 07 January 2021 1:50 Thank you for sharing!
I think a solution could be to use a (renamed) daily copy of the db from the backup directory?
Then you miss the data of today, which I think is why you want a dashboard...
If you would like to use Grafana to visualize historical data you can use the other tables. Domoticz db also containes the daily summary data from which you can do more research on yearly/monthly base.

As Grafana is only reading the database the impact is only there with limited CPU and Disk drive (SD card) resources. You will have to minimize the polling of the database, not every second but once per 5 min for example.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by waltervl »

Edited the opening post for also graphing the long term sensor data and a method to distinguish sensor data from different devices.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
delius
Posts: 11
Joined: Monday 02 January 2017 16:44
Target OS: Linux
Domoticz version: beta
Location: Netherlands
Contact:

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by delius »

functions perfectly. my setup is a dedicated ubuntu domoticz with a generic docker ubuntu to which I have now added Grafana.

Added Samba to the domoticz server for sharing the folder containing the domoticz.db. Then mounted this folder on the docker server and presented it to the Grafana docker-containter.

Temperature sensors are now shown in graphs, now look at other sensor types.
Stephanicz
Posts: 1
Joined: Sunday 08 November 2020 20:09
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by Stephanicz »

Great post. It helped me a lot, because I can't get influxdb to work on my Pi. With the help of your code and my limited sql skills I get data from the Domoticz table into Grafana. I used a thermometer as an example and get the reading in a gauge or even a table. I can't get it to work with a graph. What am I doing wrong?

I used the code below Where 35 is my thermometer:

Code: Select all

WITH converted AS (
	--- DeviceRowID value is Domoticz IDX sensor reference
	--- Conversion of Domoticz Date notation to Grafana Date notation
          SELECT Temperature, REPLACE(Date, ' ','T') || 'Z' AS datetime FROM Temperature WHERE DeviceRowID=35
)
SELECT datetime, Temperature FROM converted ORDER BY datetime ASC
I get the error: Unable to Graph data switch to table view.
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

Stephanicz wrote: Tuesday 26 January 2021 15:41 Great post. It helped me a lot, because I can't get influxdb to work on my Pi. With the help of your code and my limited sql skills I get data from the Domoticz table into Grafana. I used a thermometer as an example and get the reading in a gauge or even a table. I can't get it to work with a graph. What am I doing wrong?

I get the error: Unable to Graph data switch to table view.
It looks like Grafana is not accepting the time.
Did you set datetime as a time formatted column? See the screenshot in the opening post. You have to add it.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Dashboard without InfluxDB datapush

Post by waltervl »

delius wrote: Monday 25 January 2021 12:14 Temperature sensors are now shown in graphs, now look at other sensor types.
I found an easy way to have multiple sensors show in one graph (Grafana time-series).
Example: To show temperature devices 11, 97 and 213 in one graph I used the following query.
There is an extra conversion needed as metric (in domoticz IDX) has to be a string and not a number. So I added Temp_ as a prefix. You are offcourse free to use your own prefixes....

Code: Select all

WITH converted AS (
   SELECT Temperature ,  REPLACE(Date, ' ','T') || 'Z' AS time, 'Temp_' || DeviceRowID AS metric FROM Temperature  Where  DeviceRowID in (11, 97, 213)
)
SELECT time, metric, Temperature FROM converted ORDER BY time
Selection_023.png
Selection_023.png (82.35 KiB) Viewed 10514 times
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
dallas
Posts: 11
Joined: Monday 04 January 2021 0:34
Target OS: Windows
Domoticz version: 14127
Location: Ukraine
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by dallas »

Hi! For 2 weeks now I have been playing with the transfer of data from Domotics ... I am not a programmer, but I have already read a lot, although I do not understand everything. I don't know the syntax, everything is already merging into one heap during this time ...
I have Windows Server 16. In Domotics, I seem to have configured everything - SQL, HTTP. I installed and configured Grafana, but InfluxDB failed. Therefore, as you wrote, I installed SQLite, but I cannot make them friends! I changed all the data, read the information, substituted it, as I thought it was correct - it does not work! Tell me where is my error? And how can you put more than 20 sensors on one graph? Thanks for the answer!
Spoiler: show
GrafanaSQL.jpg
GrafanaSQL.jpg (25.22 KiB) Viewed 10220 times
DomoSQL.jpg
DomoSQL.jpg (48.94 KiB) Viewed 10220 times
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

dallas wrote: Tuesday 09 March 2021 4:27Tell me where is my error? And how can you put more than 20 sensors on one graph? Thanks for the answer!
I am sorry, this is the wrong topic for you. For this is without influxdb. You could ask your question in this topic viewtopic.php?f=38
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
dallas
Posts: 11
Joined: Monday 04 January 2021 0:34
Target OS: Windows
Domoticz version: 14127
Location: Ukraine
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by dallas »

This topic is just for me - because I cannot and do not want to install Influx. I did everything according to your instructions. Tell me - what settings need to be made so that Domotics sends data directly to Grafana? In the photo I showed my Domotics settings. Grafana is almost set up ... I made a folder with daily backups in variable environments. What could be next? Thank you!
P.S. I thought to install Docker, but it seems to be not installed on systems below Windows 10 ...
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

Domoticz is sending nothing to Grafana.
Grafana with SQLite plugin is pulling the values directly out of the Domoticz database wich is SQLite. No need to install SQLite itself. Just link Grafana with the datasource to the domoticz database (eg domoticz.db) that is placed in the installation folder. But I would strongly advise you to use a copy of this domoticz.db for test purposes.
But I do not know if this Grafana SQLite plugin is windows compatible. Were you able to install Grafana and install the plugin on Grafana?

Edit: I see your picture now and see the Grafana plugin. The path in plugin should be to the Domoticz.db file (could be a backup somewhere on your server)
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
dallas
Posts: 11
Joined: Monday 04 January 2021 0:34
Target OS: Windows
Domoticz version: 14127
Location: Ukraine
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by dallas »

Good evening Waltervl!
Now I did something like this ... Below is a photo.
Attachments
GrafanaSQL2.jpg
GrafanaSQL2.jpg (259.01 KiB) Viewed 10175 times
DomoSQL2.jpg
DomoSQL2.jpg (318.26 KiB) Viewed 10175 times
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

Grafana screenshot: You have to give the path including the database filename. So for example C:\x\x\xx\backups\daily\domoticz.db (check the filename)

Domoticz screenshot: You do not need the data push to influxdb. You can delete this. Not needed.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
dallas
Posts: 11
Joined: Monday 04 January 2021 0:34
Target OS: Windows
Domoticz version: 14127
Location: Ukraine
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by dallas »

Thanks for the tip! I thought about it, then tried adding the file extension and it worked! After that I looked at the forum and saw that you wrote the same! It works!!!
Only, as I understand it, Grafana works with the database, but does not collect new data from Domoticz? That's what Influx is for, right? Or can you somehow customize Grafana?
Thanks for your answers and help !!!
Attachments
GrafanaSQL4.jpg
GrafanaSQL4.jpg (124.63 KiB) Viewed 10165 times
GrafanaSQL3.jpg
GrafanaSQL3.jpg (294.54 KiB) Viewed 10165 times
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

Great! If you use the main domoticz.db file as datasource and not a backup file you will always get the up to date data for your graphs.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
dallas
Posts: 11
Joined: Monday 04 January 2021 0:34
Target OS: Windows
Domoticz version: 14127
Location: Ukraine
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by dallas »

One more question - I just noticed that the data in the graphs does not accumulate, but decreases to the size of exactly 7 days, as in the program settings ... In the photo there are 2 hours difference between the computer and Grafana. Not sure how you can accumulate metrics? Or do I still need to install Influx?
Attachments
Снимок.JPG
Снимок.JPG (51.68 KiB) Viewed 10152 times
User avatar
waltervl
Posts: 5149
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by waltervl »

Yes that is correct, it will show what is in the database. In domoticz after the keep log setting (for you 7 days) it will move the day summary to another table xxxxx_Calendar. If you want to see this data also you have to add it to your graph.

If you want to see the detailed history (per 5 minutes like the first 7 days) than you have to install influx db and use the data push to influxdb and make a graph on the influxdb data.

The difference in time could be the time difference in UTC for your location? What time is stored in the database?
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
User avatar
McMelloW
Posts: 427
Joined: Monday 20 November 2017 17:01
Target OS: Raspberry Pi / ODroid
Domoticz version: V2024.1
Location: Harderwijk, NL
Contact:

Re: Grafana SQLite plugin: Graphs Dashboard without InfluxDB datapush

Post by McMelloW »

@Waltervl
Quick Example in Grafana. There are more instructions on the internet to create Domoticz Grafana Dashboards.
Can you give me the link on this. Are there any SQLite examples, almost all use the IfluxDB connection?
Greetings McMelloW
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest