Error parsing json to LUA table: (invalid json string)

[mancide]

platform: raspberry pi
OS: Raspbian GNU/Linux 11 (bullseye)
Domoticz:
Version: 2023.2
Build Hash: 19efd03
Compile Date: 2023-07-21 17:23:44
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Mar 12 2021, 04:06:34) [GCC 10.2.1 20210110]

Hi,

I have a webhook that posts meter readings to a server on the internet (energyid.eu).

On the callback I always get the following error twice in the logs:

Error: dzVents: Error: (3.1.8) Gasmeter and watermeter EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"

Why do I get this error?
I know the response is not JSON. Hence the commented out code.
I do want to check the HTTPResponse statuscode.

2023-11-09.webhook.txt

(2.6 KiB) Downloaded 75 times

2023-11-09.log.txt

(3.44 KiB) Downloaded 58 times

[waltervl]

In the log you see line

Code: Select all

2023-11-09 10:50:04.296 Status: dzVents: Info: Gasmeter and watermeter EnergyId webhook: {["trigger"]="webhookcallback", ["ok"]=true, ["isHTTPResponse"]=true, ["_contentType"]="application/json; charset=utf-8", ["callback"]="webhookcallback", ["isScene"]=false, ["statusText"]="OK", ["headers"]={["X-Powered-By"]="ASP.NET", ["Vary"]="Accept-Encoding", ["Cache-Control"]="no-store,no-cache", ["Content-Type"]="application/json; charset=utf-8", ["Pragma"]="no-cache", ["Server"]="Microsoft-IIS/10.0", ["Date"]="Thu, 09 Nov 2023 09:50:02 GMT", ["X-Content-Type-Options"]="nosniff", ["Transfer-Encoding"]="chunked", ["Strict-Transport-Security"]="max-age=31536000; includeSubDomains", ["X-Frame-Options"]="SAMEORIGIN", ["Content-Encoding"]="gzip"}, ["isXML"]=false, ["isDevice"]=false, ["isSystem"]=false, ["dump"]=function, ["baseType"]="httpResponse", ["statusCode"]=200, ["isCustomEvent"]=false, ["isHardware"]=false, ["isGroup"]=false, ["isSecurity"]=false, ["isVariable"]=false, ["isJSON"]=false, ["protocol"]="HTTP/1.1", ["isTimer"]=false, ["data"]=""OK"", ["isShellCommandResponse"]=false}

Part of this line is

Code: Select all

 ["data"]=""OK""

This seems an incorrect response (double quotes)

I miss the part were you define the webhookcallback (probably with openURL)

[waltervl]

BTW the error seems to come from line

Code: Select all

domoticz.log(item, domoticz.LOG_INFO)

As that probably wants to convert the response in something readable (json) and gives the error.

[mancide]

this is where I send the data:

Code: Select all

        EnergyIdPush = function(domoticz, device, key, secret, remoteName, metric, unit, callback_ID)
	        local timestamp     = os.date("!%Y-%m-%dT%TZ")
	        local value         = device.counter
	        local metricKind    = "total"
	        local log
	        local datastring

            log = string.format('updating EnergyId.eu %s counter to %.2f %s', remoteName, value, unit)
        
            domoticz.log(log, domoticz.LOG_INFO)
            datastring = string.format('{'..
                                                            '\"remoteId\":\"%s\",'..
                                                            '\"remoteName\":\"%s\",'..
                                                            '\"metric\":\"%s\",'..
                                                            '\"metricKind\":\"%s\",'..
                                                            '\"unit\":\"%s\",'..
                                                            '\"data\":[[\"%s\",%.2f]]'..
                                                         '}',device.id, remoteName, metric, metricKind, unit, timestamp, value )    
            domoticz.openURL( {
                                url = string.format('https://hooks.energyid.eu/services/WebhookIn/%s/%s', key, secret),  --fill in your own key and secret
                                method = 'POST',
                                --callback = 'webhookcallback',
                                callback = callback_ID,
                                headers = {['Content-type']	= 'application/json; charset=utf-8'},
                                postData = datastring
                              } )
        end,

[mancide]

even with this, I get the Error:

Code: Select all

            if(item.ok)
            then
            else
                domoticz.log('There was an error', domoticz.LOG_ERROR)
                
                domoticz.log(string.format('%s ==> %s',item.statusCode, item.statusText), domoticz.LOG_ERROR)
            end

[willemd]

Try doing a dumptable of item instead of using domoticz.log.
Once you know that the exact structure is of the response, you can take the next steps and select the values you need from it.

[mancide]

Hi,

I did a dumpTable(item):
2023-12-15 12:09:01.174 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new: HTTPResponse: "callback_gas"
2023-12-15 12:09:01.208 Status: dzVents: Info: EnergyId webhook: *** start ***
2023-12-15 12:09:01.208 Status: dzVents: > isSecurity: false
2023-12-15 12:09:01.208 Status: dzVents: > isDevice: false
2023-12-15 12:09:01.208 Status: dzVents: > isTimer: false
2023-12-15 12:09:01.208 Status: dzVents: > headers:
2023-12-15 12:09:01.208 Status: dzVents: > Pragma: no-cache
2023-12-15 12:09:01.208 Status: dzVents: > Content-Encoding: gzip
2023-12-15 12:09:01.208 Status: dzVents: > Content-Type: application/json; charset=utf-8
2023-12-15 12:09:01.208 Status: dzVents: > X-Content-Type-Options: nosniff
2023-12-15 12:09:01.208 Status: dzVents: > Date: Fri, 15 Dec 2023 11:09:00 GMT
2023-12-15 12:09:01.208 Status: dzVents: > X-Powered-By: ASP.NET
2023-12-15 12:09:01.208 Status: dzVents: > Vary: Accept-Encoding
2023-12-15 12:09:01.208 Status: dzVents: > Transfer-Encoding: chunked
2023-12-15 12:09:01.208 Status: dzVents: > Strict-Transport-Security: max-age=31536000; includeSubDomains
2023-12-15 12:09:01.208 Status: dzVents: > Cache-Control: no-store,no-cache
2023-12-15 12:09:01.208 Status: dzVents: > X-Frame-Options: SAMEORIGIN
2023-12-15 12:09:01.208 Status: dzVents: > Server: Microsoft-IIS/10.0
2023-12-15 12:09:01.208 Status: dzVents: > isXML: false
2023-12-15 12:09:01.208 Status: dzVents: > data: "OK"
2023-12-15 12:09:01.208 Status: dzVents: > isSystem: false
2023-12-15 12:09:01.208 Status: dzVents: > isShellCommandResponse: false
2023-12-15 12:09:01.208 Status: dzVents: > ok: true
2023-12-15 12:09:01.208 Status: dzVents: > isScene: false
2023-12-15 12:09:01.208 Status: dzVents: > dump()
2023-12-15 12:09:01.208 Status: dzVents: > trigger: callback_gas
2023-12-15 12:09:01.208 Status: dzVents: > isHTTPResponse: true
2023-12-15 12:09:01.208 Status: dzVents: > isGroup: false
2023-12-15 12:09:01.208 Status: dzVents: > statusText: OK
2023-12-15 12:09:01.208 Status: dzVents: > baseType: httpResponse
2023-12-15 12:09:01.208 Status: dzVents: > protocol: HTTP/1.1
2023-12-15 12:09:01.208 Status: dzVents: > _contentType: application/json; charset=utf-8
2023-12-15 12:09:01.208 Status: dzVents: > isCustomEvent: false
2023-12-15 12:09:01.209 Status: dzVents: > isHardware: false
2023-12-15 12:09:01.209 Status: dzVents: > isVariable: false
2023-12-15 12:09:01.209 Status: dzVents: > isJSON: false
2023-12-15 12:09:01.209 Status: dzVents: > callback: callback_gas
2023-12-15 12:09:01.209 Status: dzVents: > statusCode: 200
2023-12-15 12:09:01.209 Status: dzVents: Info: EnergyId webhook: HTTPResponse:
2023-12-15 12:09:01.218 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-15 12:09:01.208 Error: dzVents: Error: (3.1.8) EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"

Is the problem that the HTTPResponse contains "OK" as data?
Any action on 'item' in response to a HTTPResponse results in a "Error parsing json to LUA table: (invalid json string) "OK" " error

[waltervl]

Di you remove or comment out the below line from your script?

Code: Select all

domoticz.log(item, domoticz.LOG_INFO)

[mancide]

yes,

at the moment, with this code I still have the problem...

Code: Select all

	execute = function(domoticz, item)
        domoticz.log('*** start ***', domoticz.LOG_INFO)
        domoticz.utils.dumpTable(item)
	    if(item.isTimer)
	    then
            domoticz.log('Triggered by timer', domoticz.LOG_INFO)
            
	        local key           = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'        --energyid key
	        local secret        = 'xxxxxxxxxxxx'                                --energyid secret
	        --local log

	        local device
	        local remoteName
	        local unit
	        local metric
	        --local readingType

	        
            --water meter
	        device        = domoticz.devices('watermeter: value')
	        remoteName    = 'watermeter RPI'
	        unit          = 'm³'
	        metric        = 'drinkingWaterImport'

            domoticz.helpers.EnergyIdPush(domoticz, device, key, secret, remoteName, metric, unit,'callback_water')
            
            
            --gas meter
	        device        = domoticz.devices('gasmeter: value')
	        remoteName    = 'gasmeter RPI'
	        unit          = 'm³'
	        metric        = 'naturalGasImport'

            domoticz.helpers.EnergyIdPush(domoticz, device, key, secret, remoteName, metric, unit,'callback_gas')


        elseif(item.isHTTPResponse)
        then
--        else
            --domoticz.log('NOT Triggered by timer', domoticz.LOG_INFO)
            
            domoticz.log('HTTPResponse:', domoticz.LOG_INFO)
--            domoticz.log(item, domoticz.LOG_INFO)

            --domoticz.log(string.format('%s ==> %s',item.statusCode, item.statusText), domoticz.LOG_INFO)
--            domoticz.log(string.format('%s',item.data), domoticz.LOG_INFO)
            --domoticz.utils.dumpTable(item)
            --if(item.ok)
            if(true)
            then
--                if (item.isJSON) 
--                then
--                    local response = item
--                    
--                    domoticz.log('response:', domoticz.LOG_INFO)
--                    domoticz.log(response, domoticz.LOG_INFO)
--                    domoticz.log(string.format('HTTPResponse statusCode=%s',response.statusCode), domoticz.LOG_INFO)
--                    
--                    local json = domoticz.utils.fromJSON(item.data)
--                    
--                    domoticz.log('response in JSON:', domoticz.LOG_INFO)
--                    domoticz.log('=================', domoticz.LOG_INFO)
--                    domoticz.log(json, domoticz.LOG_INFO)
--
--                    domoticz.log(string.format('response:%s',response.json.some.value), domoticz.LOG_INFO)
--                else
--                    domoticz.log('response is not in JSON', domoticz.LOG_INFO)
--                end
            else
                domoticz.log('There was an error', domoticz.LOG_ERROR)
                --domoticz.log(item, domoticz.LOG_ERROR)
                
                --domoticz.log(string.format('%s ==> %s',item.statusCode, item.statusText), domoticz.LOG_ERROR)
--                domoticz.log(string.format('%s',item.data), domoticz.LOG_ERROR)
            end
--        else
--            domoticz.log('response item:', domoticz.LOG_INFO)
--            domoticz.log(item, domoticz.LOG_INFO)
        end
    
        --domoticz.log('*** end ***', domoticz.LOG_INFO)
	end

[waltervl]

And where in the script do you get the error? What line in the script triggers this error?

[mancide]

I reduced my script to this:

Code: Select all

return {
	on = {
	    timer = { 'every 10 minutes' },
	    --timer = { 'every 2 minutes' },
        httpResponses = { 'webhookcallback', 'callback_gas', 'callback_water'  }
	},
	logging = {
		level = domoticz.LOG_INFO,
		marker = 'EnergyId webhook',
	},
	execute = function(domoticz, item)
	    if(item.isTimer)
	    then
            domoticz.log('Triggered by timer', domoticz.LOG_INFO)
            
	        local key           = ''        --energyid key
	        local secret        = ''                                --energyid secret
	        --local log

	        local device
	        local remoteName
	        local unit
	        local metric
	        --local readingType

	        
            --water meter
	        device        = domoticz.devices('watermeter: value')
	        remoteName    = 'watermeter RPI'
	        unit          = 'm³'
	        metric        = 'drinkingWaterImport'

            domoticz.helpers.EnergyIdPush(domoticz, device, key, secret, remoteName, metric, unit,'callback_water')
            
            
            --gas meter
	        device        = domoticz.devices('gasmeter: value')
	        remoteName    = 'gasmeter RPI'
	        unit          = 'm³'
	        metric        = 'naturalGasImport'

            domoticz.helpers.EnergyIdPush(domoticz, device, key, secret, remoteName, metric, unit,'callback_gas')
        else
            domoticz.log('NOT Triggered by timer', domoticz.LOG_INFO)
        end
	end
}

And I still get the error...

2023-12-22 11:10:00.307 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new:, trigger: "every 10 minutes"
2023-12-22 11:10:00.347 Status: dzVents: Info: EnergyId webhook: Triggered by timer
2023-12-22 11:10:00.348 Status: dzVents: Info: EnergyId webhook: updating EnergyId.eu watermeter RPI counter to 123.22 m³
2023-12-22 11:10:00.350 Status: dzVents: Info: EnergyId webhook: updating EnergyId.eu gasmeter RPI counter to 11003.90 m³
2023-12-22 11:10:00.360 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-22 11:10:02.912 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new: HTTPResponse: "callback_water"
2023-12-22 11:10:02.947 Status: dzVents: Info: EnergyId webhook: NOT Triggered by timer
2023-12-22 11:10:02.957 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-22 11:10:02.957 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new: HTTPResponse: "callback_gas"
2023-12-22 11:10:02.991 Status: dzVents: Info: EnergyId webhook: NOT Triggered by timer
2023-12-22 11:10:03.001 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-22 11:10:02.947 Error: dzVents: Error: (3.1.8) EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"
2023-12-22 11:10:02.991 Error: dzVents: Error: (3.1.8) EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"

remark that the timestamp of the error is equal to the log line in the else...

I think it is triggered somewhere in the dzvents code that is calling my script...

[willemd]

So when the script is triggered by the callback and the only thing you do is the item.isTimer check (which performs as it should) and then print a string to the logfile you get the error?

I think the problem still lies in the contents of item, exactly as the error message indicates, a string "OK" instead of OK.

What if you remove the isTimer check to avoid handling item at all? Since you don't want to launch the http request again and again, you might want to need a persistent variable to indicate the launch was already done and the script is now triggered by the callback, just for testing.

[mancide]

So when the script is triggered by the callback and the only thing you do is the item.isTimer check (which performs as it should) and then print a string to the logfile you get the error?

yes

I think the problem still lies in the contents of item, exactly as the error message indicates, a string "OK" instead of OK.

What if you remove the isTimer check to avoid handling item at all? Since you don't want to launch the http request again and again, you might want to need a persistent variable to indicate the launch was already done and the script is now triggered by the callback, just for testing.

I agree with you that the problem is probably in processing the item.

But, I never ask to convert item into JSON.
So, why do I get the error...

I need the timer to send my meter values to "energyid.eu" every 10 minutes.
I still think it is good practice to check the result. Which I can´t in this case...

[willemd]

Probably the item is converted into a LUA table so the property/field isTimer can be checked. While doing so also the incorrect string is encountered.
Maybe you can make a small script that filters out the double quotes in the response?

[mancide]

I tried this

Code: Select all

        for index, value in pairs(item) do
            --domoticz.log(string.format('item[%s]=%s', index, value), domoticz.LOG_INFO)
            if(index=='data')
            then
                domoticz.log(string.format('item[%s]=%s', index, value), domoticz.LOG_INFO)
	            item['data'] = string.gsub(item['data'], "\"", "")
            end
        end
        domoticz.log(string.format('item[data]=%s', item['data']), domoticz.LOG_INFO)

before the

Code: Select all

        if(item.isHTTPResponse)

but I still get this as a result:

2023-12-24 15:32:00.420 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new:, trigger: "every 2 minutes"
2023-12-24 15:32:00.460 Status: dzVents: Info: EnergyId webhook: item[data]=nil
2023-12-24 15:32:00.460 Status: dzVents: Info: EnergyId webhook: Triggered by timer
2023-12-24 15:32:00.462 Status: dzVents: Info: EnergyId webhook: updating EnergyId.eu watermeter RPI counter to 123.49 m³
2023-12-24 15:32:00.463 Status: dzVents: Info: EnergyId webhook: updating EnergyId.eu gasmeter RPI counter to 11013.01 m³
2023-12-24 15:32:00.474 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-24 15:32:01.256 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new: HTTPResponse: "callbackgas"
2023-12-24 15:32:01.287 Status: dzVents: Info: EnergyId webhook: item[data]="OK"
2023-12-24 15:32:01.287 Status: dzVents: Info: EnergyId webhook: item[data]=OK
2023-12-24 15:32:01.287 Status: dzVents: Info: EnergyId webhook: Triggered by HTTPResponse
2023-12-24 15:32:01.297 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-24 15:32:01.297 Status: dzVents: Info: EnergyId webhook: ------ Start internal script: EnergyId_Webhook_new: HTTPResponse: "callbackwater"
2023-12-24 15:32:01.332 Status: dzVents: Info: EnergyId webhook: item[data]="OK"
2023-12-24 15:32:01.332 Status: dzVents: Info: EnergyId webhook: item[data]=OK
2023-12-24 15:32:01.332 Status: dzVents: Info: EnergyId webhook: Triggered by HTTPResponse
2023-12-24 15:32:01.341 Status: dzVents: Info: EnergyId webhook: ------ Finished EnergyId_Webhook_new
2023-12-24 15:32:01.287 Error: dzVents: Error: (3.1.8) EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"
2023-12-24 15:32:01.332 Error: dzVents: Error: (3.1.8) EnergyId webhook: Error parsing json to LUA table: (invalid json string) "OK"

[willemd]

I think in your for loop the item is also already converted (in the background/internals). I think the correction of the "OK" string has to be done before the callback to dzvents is done, in a shell script or something like that.