Possible problem with sending data to Influx

[MeAlbert]

Hi
I have a script running to push some value's. The script was made by @waaren and looks like this.

Code: Select all

return
{
    on =
    {
        timer =
        {
            'every minute' ,
        },
    },

    logging =
    {
        level = domoticz.LOG_ERROR, -- change to domoticz.LOG_DEBUG when not working as expected
        marker = 'forcePushToInfluxdb with filter Example',
    },

    execute = function(dz)

        local myInfluxDevices =
        {
            64, 73, 29, 33, 76, 77, 79, 86, 16, 71
        }

        local function forcePushToInfluxdb(dv)
            dz.log('Forcing push to influxDb of ' .. dv.name .. ' : '  ..dv.sValue, dz.LOG_FORCE)

            local url = dz.settings['Domoticz url'] ..  '/json.htm?type=command&param=udevice&idx=' .. dv.idx  ..
                    '&nvalue=' .. ( dv.nValue or 0  ) ..
                    '&svalue=' .. ( dv.sValue or '' )
            dz.openURL(url)
        end

        dz.devices().filter(myInfluxDevices).forEach(function(dv)
            forcePushToInfluxdb(dv)
        end)
    end
}

I have for some time that my pi is running very slow or stopping. Now I was able to catch the error lines before this happens and it looks like that this script is the bad one.

Code: Select all

2021-02-07 20:11:54.768 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:31.426 Error: MQTT: disconnected, restarting (rc=14)
2021-02-07 20:12:39.128 Error: dzVents: Error: (3.0.2) forcePushToInfluxdb with filter Example: ------ Finished Influx push force after >10 seconds. (using 1.341 seconds CPU time !)
2021-02-07 20:12:40.034 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:51.884 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
2021-02-07 20:12:51.946 Error: BasePush: Could not determine data push value
2021-02-07 20:12:51.951 Error: BasePush: Could not determine data push value
2021-02-07 20:12:53.115 Error: BasePush: Could not determine data push value
2021-02-07 20:12:53.116 Error: BasePush: Could not determine data push value

What is the matter here and could someone suggest a cure?

[waltervl]

Is there a reason you did not use the standard push to influxdb function?
https://www.domoticz.com/wiki/Influxdb

[waaren]

I have a script running to push some value's.
What is the matter here and could someone suggest a cure?

I guess forcing 10 devices like this every minute with the openURL method is a bit too much for your system in this version. In recent builds the openURL is coded as none blocking but even then this method should not be used for this many devices. It is meant as exception only. If you really need all these devices to be pushed to influx using this method I suggest you change the triggering of the script in such a way that it is triggered on a device update

[MeAlbert]

Ok so I will change the minute trigger to a device trigger. If I face problems afterwards I will come back here.
Thanks for the quick reply.

[MeAlbert]

@waaren
I brought back the amount of devices to 2. That should not be a problem I would think.
The other devices are Solar Panel devices. They are updated when device value changes. I use this code.

Code: Select all

return
{
    on =
    {
        devices =
        {
            16, 29, 33, 76, 77, 86, 71
        },
    },
   
    logging =
    {
        level = domoticz.LOG_DEBUG,
        marker = 'Push to Influx',
    },
 
    data =
    {
        lastExecutionTimes = { initial = {} },
    },
   
    execute = function(dz, item)

        dz.log('Device ' .. item.name .. ' was changed ' .. item.rawData[1], dz.LOG_DEBUG)
       
        local function pushToInfluxdb(idx , nValue, rawData )
            local sValue = table.concat(rawData,';')
            local nValue = nValue or 0
           
            local url = dz.settings['Domoticz url'] ..  '/json.htm?type=command&param=udevice&idx=' .. idx  ..
                        '&nvalue=' .. nValue ..
                        '&svalue=' .. sValue
            dz.openURL(url)
            
        end

        if dz.data.lastExecutionTimes[item.name] == nil or ( dz.data.lastExecutionTimes[item.name] < ( dz.time.dDate - 1 ) ) then
            pushToInfluxdb( item.idx, 0, item.rawData )
            dz.data.lastExecutionTimes[item.name] = dz.time.dDate
        else
            dz.log('Too soon after last update; we don\'t want an endless loop', dz.LOG_DEBUG)
        end
    end
}

I have 2 different solar panel systems. The GoodWe one I use a plugin. The other one I collect the date by a script that runs from 10 minutes before sunset until 10 minutes after sunset. So at nighttime no data.
I still get error messages.

Code: Select all

2021-02-08 21:45:41.655 Error: BasePush: Could not determine data push value
2021-02-08 21:49:35.089 Error: BasePush: Could not determine data push value
2021-02-08 21:49:35.123 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.453 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.453 Error: BasePush: Could not determine data push value
2021-02-08 21:57:38.487 Error: BasePush: Could not determine data push value

If I change the trigger to this:

Code: Select all

on = {
            timer = { 'at daytime' },
            devices = {16, 29, 33, 76, 77, 86, 71}
    },

I think I get rid of these errors. Is this assumption correct and is the suggested code correct?

Edit
See the 10 sec error again.
2021-02-08 22:39:02.982 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

[waltervl]

Again, Why not use the standard influxdb push?

[MeAlbert]

I was told that for dummy devices adding those devices in de influx push is not enough. Then nothing happens.
So I used the push on trigger script to get the data to influx.
If you tell me now that only adding it to the data push part is enough I can and will skip these scripts.

I use this version:
Version: 2020.2
Build Hash: b63341bc0
Compile Date: 2020-04-26 13:47:55
dzVents Version: 3.0.2
Python Version: 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]

[waltervl]

I am not aware of such an prerequisite of the data push.
I do not use influxdb so I cannot test it but you can yourself.

Are you pushing data to influxdb only from Domoticz to publish it into Grafana?
Because now there is a plugin for Grafana that can use SQLite (the database of Domoticz) directly. No need to send data to influxdb for that reason anymore:
viewtopic.php?f=21&t=35101

[MeAlbert]

OK, thanks I will look into that tomorrow. Maybe this is the better way to do.
Thanks for your help.

[waaren]

I am not aware of such an prerequisite of the data push.

Check this post which is also referring to a wiki page you updated recently yourself?

I have not tested this recently and since I wrote this roughly 2100 new commits were merged into domoticz, so something might have changed in this area changing this behavior....

[waltervl]

I saw that later but because the httplinker wiki page is about sending to http and not influxdb I ignored it until confirmation. That wiki page I only edited to add the note for a direct influxdb function as the example for Http link was to send data to influxdb. The short influxdb wiki page I created does not have this warning nor could I find a warning about this.
So perhaps someone can test this and I can add the warning too for influxdb wiki.

[EddyG]

As far as I know there is still no push implemented in Domoticz for Influxdb 2.0, which is now the default.
If you want to use the script from @waaren you can use this OpenURL

Code: Select all

                dz.openURL({
                    url = "http://<ipaddress>:8086/api/v2/write?org=<Organisation>&bucket=<Bucketname>&precision=s" ,
                    method = "POST",
                    callback= "Influxdb",
                    headers = {
                        ['Authorization'] = 'Token ' .. TOKEN,
                        ['Content-Type'] = 'text/plain'
                    },
                    postData = pushdata,
                    callback = 'Influxdb'

TOKEN is the token from influxdb
The variable 'pushdata' can be made like this

Code: Select all

                if item.deviceType == "General" and item.deviceSubType == "kWh" then
                    pushdata = item.name::gsub(' ', '-') -- ' watt=' .. tostring(item.usage)
                end
                if item.deviceType == "Temp + Humidity" and item.deviceSubType == "WTGR800" then
                    pushdata = item.name:gsub(' ', '-') .. ' temp=' .. tostring(dz.utils.round(item.temperature,1)) .. '\n' ..
                        item.name:gsub(' ', '-') .. ' humi=' .. tostring(dz.utils.round(item.humidity,0))
                end

On my Pi 4B the average time that my script runs is 2ms, so no real impact on performance.

Edit: changed snippet for space in item.name tot item.name:gsub(' ', '-') because space is not accepted by Influxdb, and I used an OLD round function.

[waltervl]

As far as I know there is still no push implemented in Domoticz for Influxdb 2.0, which is now the default.
If you want to use the script from @waaren you can use this OpenURL

Is this something I should add to the wiki page as a warning?
I see no github issues regarding incompatibility with influxdb 2.0.

[EddyG]

The influxdb 1.8 etc. versions are based on user/password login as is in the Influxdb Data Push in Domoticz (my version 2020.2 build 13058)
Influxdb 2.0 etc. is based on API key pushing data to influxdb. I did not succeed in using user/password to push data.
It might be that I am doing something wrong, but I did not succeed, so I use dzVents with the above snippets, and that works.

[waltervl]

I have added a warning to the wiki. https://www.domoticz.com/wiki/Influxdb
Perhaps someone will pick it up in the source code to change.

[waltervl]

Influxdb 2.0 support is now available in latest 2021.1 beta see commit:
https://github.com/domoticz/domoticz/co ... 842b9ceba3

[Derik]

Perhaps some one have a aexample query form the bucket?
I do not get an error in domoticz, log, so i think i upload to the bucket.

Only i cannot set the bucket the good way to get data visiable..

ScreenShot135.png (47.22 KiB) Viewed 2319 times

ScreenShot134.png (46.03 KiB) Viewed 2319 times

ScreenShot136.png (26.23 KiB) Viewed 2319 times

[Derik]

And when i look into flux there a idx and a lot of data

ScreenShot138.png (192.84 KiB) Viewed 2317 times