Page 1 of 1
Possible problem with sending data to Influx
Posted: Sunday 07 February 2021 20:32
by MeAlbert
Hi
I have a script running to push some value's. The script was made by @waaren and looks like this.
Code: Select all
return
{
on =
{
timer =
{
'every minute' ,
},
},
logging =
{
level = domoticz.LOG_ERROR, -- change to domoticz.LOG_DEBUG when not working as expected
marker = 'forcePushToInfluxdb with filter Example',
},
execute = function(dz)
local myInfluxDevices =
{
64, 73, 29, 33, 76, 77, 79, 86, 16, 71
}
local function forcePushToInfluxdb(dv)
dz.log('Forcing push to influxDb of ' .. dv.name .. ' : ' ..dv.sValue, dz.LOG_FORCE)
local url = dz.settings['Domoticz url'] .. '/json.htm?type=command¶m=udevice&idx=' .. dv.idx ..
'&nvalue=' .. ( dv.nValue or 0 ) ..
'&svalue=' .. ( dv.sValue or '' )
dz.openURL(url)
end
dz.devices().filter(myInfluxDevices).forEach(function(dv)
forcePushToInfluxdb(dv)
end)
end
}
I have for some time that my pi is running very slow or stopping. Now I was able to catch the error lines before this happens and it looks like that this script is the bad one.
Code: Select all
2021-02-07 20:11:54.768 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:31.426 Error: MQTT: disconnected, restarting (rc=14)
2021-02-07 20:12:39.128 Error: dzVents: Error: (3.0.2) forcePushToInfluxdb with filter Example: ------ Finished Influx push force after >10 seconds. (using 1.341 seconds CPU time !)
2021-02-07 20:12:40.034 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:51.884 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:51.946 Error: BasePush: Could not determine data push value
2021-02-07 20:12:51.951 Error: BasePush: Could not determine data push value
2021-02-07 20:12:53.115 Error: BasePush: Could not determine data push value
2021-02-07 20:12:53.116 Error: BasePush: Could not determine data push value
What is the matter here and could someone suggest a cure?
Re: Possible problem with sending data to Influx
Posted: Sunday 07 February 2021 21:03
by waltervl
Is there a reason you did not use the standard push to influxdb function?
https://www.domoticz.com/wiki/Influxdb
Re: Possible problem with sending data to Influx
Posted: Sunday 07 February 2021 21:45
by waaren
MeAlbert wrote: ↑Sunday 07 February 2021 20:32
I have a script running to push some value's.
What is the matter here and could someone suggest a cure?
I guess forcing 10 devices like this every minute with the openURL method is a bit too much for your system in this version. In recent builds the openURL is coded as none blocking but even then this method should not be used for this many devices. It is meant as exception only. If you really need all these devices to be pushed to influx using this method I suggest you change the triggering of the script in such a way that it is triggered on a device update
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 0:35
by MeAlbert
Ok so I will change the minute trigger to a device trigger. If I face problems afterwards I will come back here.
Thanks for the quick reply.
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 22:34
by MeAlbert
@waaren
I brought back the amount of devices to 2. That should not be a problem I would think.
The other devices are Solar Panel devices. They are updated when device value changes. I use this code.
Code: Select all
return
{
on =
{
devices =
{
16, 29, 33, 76, 77, 86, 71
},
},
logging =
{
level = domoticz.LOG_DEBUG,
marker = 'Push to Influx',
},
data =
{
lastExecutionTimes = { initial = {} },
},
execute = function(dz, item)
dz.log('Device ' .. item.name .. ' was changed ' .. item.rawData[1], dz.LOG_DEBUG)
local function pushToInfluxdb(idx , nValue, rawData )
local sValue = table.concat(rawData,';')
local nValue = nValue or 0
local url = dz.settings['Domoticz url'] .. '/json.htm?type=command¶m=udevice&idx=' .. idx ..
'&nvalue=' .. nValue ..
'&svalue=' .. sValue
dz.openURL(url)
end
if dz.data.lastExecutionTimes[item.name] == nil or ( dz.data.lastExecutionTimes[item.name] < ( dz.time.dDate - 1 ) ) then
pushToInfluxdb( item.idx, 0, item.rawData )
dz.data.lastExecutionTimes[item.name] = dz.time.dDate
else
dz.log('Too soon after last update; we don\'t want an endless loop', dz.LOG_DEBUG)
end
end
}
I have 2 different solar panel systems. The GoodWe one I use a plugin. The other one I collect the date by a script that runs from 10 minutes before sunset until 10 minutes after sunset. So at nighttime no data.
I still get error messages.
Code: Select all
2021-02-08 21:45:41.655 Error: BasePush: Could not determine data push value
2021-02-08 21:49:35.089 Error: BasePush: Could not determine data push value
2021-02-08 21:49:35.123 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.453 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.453 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.487 Error: BasePush: Could not determine data push value
If I change the trigger to this:
Code: Select all
on = {
timer = { 'at daytime' },
devices = {16, 29, 33, 76, 77, 86, 71}
},
I think I get rid of these errors. Is this assumption correct and is the suggested code correct?
Edit
See the 10 sec error again.
2021-02-08 22:39:02.982 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 22:51
by waltervl
Again, Why not use the standard influxdb push?
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 23:05
by MeAlbert
I was told that for dummy devices adding those devices in de influx push is not enough. Then nothing happens.
So I used the push on trigger script to get the data to influx.
If you tell me now that only adding it to the data push part is enough I can and will skip these scripts.
I use this version:
Version: 2020.2
Build Hash: b63341bc0
Compile Date: 2020-04-26 13:47:55
dzVents Version: 3.0.2
Python Version: 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 23:13
by waltervl
I am not aware of such an prerequisite of the data push.
I do not use influxdb so I cannot test it but you can yourself.
Are you pushing data to influxdb only from Domoticz to publish it into Grafana?
Because now there is a plugin for Grafana that can use SQLite (the database of Domoticz) directly. No need to send data to influxdb for that reason anymore:
viewtopic.php?f=21&t=35101
Re: Possible problem with sending data to Influx
Posted: Monday 08 February 2021 23:48
by MeAlbert
OK, thanks I will look into that tomorrow. Maybe this is the better way to do.
Thanks for your help.
Re: Possible problem with sending data to Influx
Posted: Tuesday 09 February 2021 0:52
by waaren
waltervl wrote: ↑Monday 08 February 2021 23:13
I am not aware of such an prerequisite of the data push.
Check this
post which is also referring to a wiki page you updated recently yourself?
I have not tested this recently and since I wrote this roughly 2100 new commits were merged into domoticz, so something might have changed in this area changing this behavior....
Re: Possible problem with sending data to Influx
Posted: Tuesday 09 February 2021 8:10
by waltervl
I saw that later but because the httplinker wiki page is about sending to http and not influxdb I ignored it until confirmation. That wiki page I only edited to add the note for a direct influxdb function as the example for Http link was to send data to influxdb. The short influxdb wiki page I created does not have this warning nor could I find a warning about this.
So perhaps someone can test this and I can add the warning too for influxdb wiki.
Re: Possible problem with sending data to Influx
Posted: Thursday 11 March 2021 21:20
by EddyG
As far as I know there is still no push implemented in Domoticz for Influxdb 2.0, which is now the default.
If you want to use the script from @waaren you can use this OpenURL
Code: Select all
dz.openURL({
url = "http://<ipaddress>:8086/api/v2/write?org=<Organisation>&bucket=<Bucketname>&precision=s" ,
method = "POST",
callback= "Influxdb",
headers = {
['Authorization'] = 'Token ' .. TOKEN,
['Content-Type'] = 'text/plain'
},
postData = pushdata,
callback = 'Influxdb'
TOKEN is the token from influxdb
The variable 'pushdata' can be made like this
Code: Select all
if item.deviceType == "General" and item.deviceSubType == "kWh" then
pushdata = item.name::gsub(' ', '-') -- ' watt=' .. tostring(item.usage)
end
if item.deviceType == "Temp + Humidity" and item.deviceSubType == "WTGR800" then
pushdata = item.name:gsub(' ', '-') .. ' temp=' .. tostring(dz.utils.round(item.temperature,1)) .. '\n' ..
item.name:gsub(' ', '-') .. ' humi=' .. tostring(dz.utils.round(item.humidity,0))
end
On my Pi 4B the average time that my script runs is 2ms, so no real impact on performance.
Edit: changed snippet for space in item.name tot item.name:gsub(' ', '-') because space is not accepted by Influxdb, and I used an OLD round function.
Re: Possible problem with sending data to Influx
Posted: Thursday 11 March 2021 21:56
by waltervl
EddyG wrote: ↑Thursday 11 March 2021 21:20
As far as I know there is still no push implemented in Domoticz for Influxdb 2.0, which is now the default.
If you want to use the script from @waaren you can use this OpenURL
Is this something I should add to the wiki page as a warning?
I see no github issues regarding incompatibility with influxdb 2.0.
Re: Possible problem with sending data to Influx
Posted: Friday 12 March 2021 9:41
by EddyG
The influxdb 1.8 etc. versions are based on user/password login as is in the Influxdb Data Push in Domoticz (my version 2020.2 build 13058)
Influxdb 2.0 etc. is based on API key pushing data to influxdb. I did not succeed in using user/password to push data.
It might be that I am doing something wrong, but I did not succeed, so I use dzVents with the above snippets, and that works.
Re: Possible problem with sending data to Influx
Posted: Friday 12 March 2021 10:44
by waltervl
I have added a warning to the wiki.
https://www.domoticz.com/wiki/Influxdb
Perhaps someone will pick it up in the source code to change.
Re: Possible problem with sending data to Influx
Posted: Monday 10 May 2021 12:08
by waltervl
Influxdb 2.0 support is now available in latest 2021.1 beta see commit:
https://github.com/domoticz/domoticz/co ... 842b9ceba3
Re: Possible problem with sending data to Influx
Posted: Monday 02 August 2021 15:52
by Derik
Perhaps some one have a aexample query form the bucket?
I do not get an error in domoticz, log, so i think i upload to the bucket.
Only i cannot set the bucket the good way to get data visiable..

- ScreenShot135.png (47.22 KiB) Viewed 2323 times

- ScreenShot134.png (46.03 KiB) Viewed 2323 times

- ScreenShot136.png (26.23 KiB) Viewed 2323 times
Re: Possible problem with sending data to Influx
Posted: Monday 02 August 2021 16:00
by Derik
And when i look into flux there a idx and a lot of data

- ScreenShot138.png (192.84 KiB) Viewed 2321 times