Luarocks/Lunajson

[Toulon7559]

Improvement is always possible; therefore trying different approaches.
For decoding json-files with lua, I found https://somedudesays.com/2019/12/using-json-with-lua/

However when trying to adapt the examples for my use, got stuck.
Installation of Luarocks and lunajson as instructed, without any problem.
Text at url claims tested with lua incl. lua 5.3

Looked very simple, but apparently something missing/wrong in the examples.
Original example line 002 gave error report :

Code: Select all

module 'lunajson' not found

After some experimenting no error report for adapted Line 007 (either with or without 'local').
Checking in the folders /usr/local/share/lua/5.x/ nowhere found module lunajson.
Next error report is related to line 008:

Code: Select all

syntax error near 'local'

Any hint to get correct operation of Luarocks/lunajson?
Or an equivalent, 'short' method to extract info from a json-file?

Code: Select all

commandArray = {}

-- Other scriptlines

-- Lines 001 ~ 005 = Example using luarocks/lunajson for json-readout
-- lunajson = require 'lunajson' 
-- local jsonraw = '{"test":[1,2,3,4]}'
-- local jsonparse = lunajson.decode( jsonraw )
-- print( jsonparse["test"][ 1 ] .. ", " .. jsonparse["test"][ 2 ] .. ", " .. jsonparse["test"][ 3 ] .. ", " .. jsonparse["test"][ 4 ] )
-- Lines 006 ~ 012 = Testing, read remote device & decode contents of json-file [for switch]
   local lunajson = require lunajson
   local jsonraw = 'http://192.168.0.185/json.htm?type=devices&rid=337'
   local jsonparse = lunajson.decode( jsonraw )
   print( jsonparse )
   print( jsonparse["Sunrise"] )
   print( jsonparse["result"][0][Data] )
   
 -- Other scriptlines

return commandArray

[waaren]

Improvement is always possible; therefore trying different approaches.
For decoding json-files with lua, I found https://somedudesays.com/2019/12/using-json-with-lua/

Are you trying this in domoticz?
dzVents:

Spoiler: show

Code: Select all

return
{
    on =
    {
        timer = { 'every minute'},
    },

    logging =
    {
        level = domoticz.LOG_DEBUG,
        marker = 'lunajson',
    },

    execute = function(dz)

        lunajson = require 'lunajson'
        local jsonraw = '{"test":[1,2,3,4]}'
        local jsonparse = lunajson.decode( jsonraw )

        dz.log ( jsonparse["test"][ 1 ] .. ", " .. jsonparse["test"][ 2 ] .. ", " .. jsonparse["test"][ 3 ] .. ", " .. jsonparse["test"][ 4 ] ,dz.LOG_INFO)
        dz.utils.dumpTable(jsonparse)
    end
}

log

Code: Select all

2020-09-13 21:17:00.249 Status: dzVents: Info: lunajson: ------ Start internal script: Script #3:, trigger: "every minute"
2020-09-13 21:17:00.263 Status: dzVents: Info: lunajson: 1, 2, 3, 4
2020-09-13 21:17:00.263 Status: dzVents: > test:
2020-09-13 21:17:00.263 Status: dzVents: > 1: 1
2020-09-13 21:17:00.263 Status: dzVents: > 2: 2
2020-09-13 21:17:00.263 Status: dzVents: > 3: 3
2020-09-13 21:17:00.263 Status: dzVents: > 4: 4
2020-09-13 21:17:00.265 Status: dzVents: Info: lunajson: ------ Finished Script #3

classic Lua

Spoiler: show

Code: Select all

commandArray = {}

lunajson = require 'lunajson'

local jsonraw = '{"test":[1,2,3,4]}'
local jsonparse = lunajson.decode( jsonraw )

print( jsonparse["test"][ 1 ] .. ", " .. jsonparse["test"][ 2 ] .. ", " .. jsonparse["test"][ 3 ] .. ", " .. jsonparse["test"][ 4 ] )

return commandArray

log:

Code: Select all

2020-09-13 21:10:23.569 Status: LUA: 1, 2, 3, 4

or at the OS level?

• Have you installed Lua version 5.3? (check with sudo lua -v)
• What is you luarocks version? (check with sudo luarocks --version)
• Did you change line 1 of the example script to work with Lua 5.3?

Spoiler: show

Code: Select all

 #!/usr/bin/lua5.3

lunajson = require 'lunajson'

local jsonraw = '{"test":[1,2,3,4]}'
local jsonparse = lunajson.decode( jsonraw )

print( jsonparse["test"][ 1 ] .. ", " .. jsonparse["test"][ 2 ] .. ", " .. jsonparse["test"][ 3 ] .. ", " .. jsonparse["test"][ 4 ] )

output:

Code: Select all

sudo ./lunajson.lua
1, 2, 3, 4

Any specific reason why you want a different json decoder than the one already available for dzVents- and classic Lua scripts?

[Toulon7559]

@Waaren

Shown script segment has been tested as part of a lua-script running at Raspberry with
Domoticz Version: 2020.2
Build Hash: b63341bc0
Compile Date: 2020-04-26 13:47:55
dzVents Version: 3.0.2
Python Version: 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]

Checking lua & luarocks:
sudo lua -v => 5.1.5 (as coming with Domoticz 2020.2)
sudo luarocks -- version => /usr/bin/luarocks 2.4.2

As indicated in my previous message, according to text for luarocks & lunajson, those softwares tested with lua upto and incl. lua5.3:
IMHO that seems to imply that it should work as well with 'lower' versions of lua than 5.3
Because the example at the url starts with

Code: Select all

#!/usr/bin/lua5.1

, surprised that the original example in my configuration generates an error.
Installing lua5.3 not a big deal, but unexpected, and considering other mutations due to lua5.3, wondering what afterwards will be correct application of luarocks & lunajson.

Not obliged to use lua for this script function, but a detour using dzvents or python means involvement of additional script, instead of directly merged function.

[waaren]

Shown script segment has been tested as part of a lua-script running in Domoticz 2020.2
sudo lua -v => 5.1.5 as coming with Domoticz 2020.2
sudo luarocks -- version => /usr/bin/luarocks 2.4.2

Domoticz 2020.2 uses Lua 5.3 internally.
If your Lua version at the os level is different, the modules installed by Luarocks do end up in a different location and cannot be found by the require command.
Hope this helps.

Still interested to understand if there is a specific reason to use another JSON coder/decoder as the one already available?

[Toulon7559]

@waaren

Considering your explanation, seems strange that I have lua5.1.5 reported in Domoticz 2020.2, and not lua5.3
But in that perspective perhaps not a surprise that luarocks & lunajson have been installed at now unexpected/unfindable location.

Still interested to understand if there is a specific reason to use another JSON coder/decoder as the one already available?

No reason/preference at all for JSON coder/decoder:
perhaps just lack of knowledge to use the 'one already available', and then try what can be found .....

As can be seen in Lines 006~012 of the script segment in the message, the functional requirement for this script segment is rather simple:
1. read JSON from device at other Domoticz at remote IP
2. extract specific value/contents from JSON
Objective:
3. apply extracted value/contents for local parameter in subsequent lines of lua-script

[waaren]

Considering your explanation, seems strange that I have lua5.1.5 reported in Domoticz 2020.2, and not lua5.3

@Toulon7559
Where / how do you see this version number?
The domoticz package only use an embedded version of Lua and that is Lua 5.3 from V4.11439 onwards. Before that it was on Lua 5.2
If you see other versions in a script then it might show something from a module from the Lua package from the OS level (e.g. installed with sudo apt install Lua)

Below a classic Lua script that prints both the embedded version number and the os Lua level. It also give examples how to work with lunajson decoder and how to work with the JSON decoder that comes with domoticz.

Code: Select all

commandArray = {}
    local domoticzIP = '192.168.192.116' -- change to your domoticz IP
    local domoticzPort = 8084  -- change to your domoticz Port
    local deviceIDX = 133 -- change to your deviceIDX
    
    function os.caption(cmd)
        local file = io.popen(cmd)
        local output = file:read('*all')
        file:close()
        return output
    end
    
    -- print version of embedded Lua and Lua version on the os level (different things)   
    print("Lua version in domoticz: " .. _VERSION)
    print("Lua version in OS: " .. os.caption('lua -v')) 
     
    -- get JSON via curl ( same for both json decoders )
    local httpString = 'http://' .. domoticzIP ..':' .. domoticzPort .. '/json.htm?type=devices&rid='  .. deviceIDX
    local jsoncmd = 'curl ' .. '"' .. httpString .. '"' 
    local jsonraw =  os.caption(jsoncmd )
 
    print('Using lunajson --------------------------------------------------')
    local lunajson = require 'lunajson'
    local jsonparse = lunajson.decode( jsonraw )
    
    print( "Sunrise is at " ..  jsonparse["Sunrise"] )
    print( "Data:  " .. jsonparse.result[1].Data )
 
    print('Using existing JSON for Lua and dzVents --------------------------------------------------')
    local defaultJSON = require "JSON"     -- use generic JSON.lua
    local jsonparse = defaultJSON:decode( jsonraw )
   
    print( "Sunrise is at " ..  jsonparse["Sunrise"] )
    print( "Data:  " .. jsonparse.result[1].Data )
 
return commandArray

for completeness a dzVents script doing the same but calling the API async (without blocking the event system)

Code: Select all

local scriptVar = 'getDeviceInfo'

return
{
    on =
    {
        timer =
        {
            'every minute' -- just an example to trigger the request
        },

        httpResponses =
        {
            scriptVar,
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG,
        marker = scriptVar,
    },

    execute = function(dz, item)
        local deviceIDX = 526 -- chamge to your deviceIDX

        function os.caption(cmd)
            local file = io.popen(cmd)
            local output = file:read('*all')
            file:close()
            return output
        end

        if item.isTimer then
            dz.openURL
            ({
                url = dz.settings['Domoticz url'] .. '/json.htm?type=devices&rid='  .. deviceIDX,
                callback = scriptVar,
            })
            return
        end

        dz.log("Lua version in domoticz: " .. _VERSION,dz.LOG_INFO)
        dz.log("Lua version in OS: " .. os.caption('lua -v'),dz.LOG_INFO)

        if item.json then -- translation to Lua table was successfull
            dz.log( "Sunrise is at " ..  item.json.Sunrise ,dz.LOG_INFO)
            dz.log( "Data:  " .. item.json.result[1].Data ,dz.LOG_INFO)
        end
    end
}

[Toulon7559]

@waaren

My check was as you asked, performed at Putty's cli

Checking lua & luarocks:
sudo lua -v => 5.1.5
sudo luarocks -- version => /usr/bin/luarocks 2.4.2

In this message also the other info which software installed under Domoticz (as described at Domoticz/ Setup/ Over)

[waaren]

My check was as you asked, performed at Putty's cli

The version that is shown with this command is the Lua version installed on the OS. Domoticz use an embedded version. Just like it does for sqlite, mosquito and some other tools,

I asked to check this OS level version as you try to mix the embedded Lua script with OS level Lua modules. That can work perfectly well as long as they have the same major version numbers.
Embedded 5.3 and OS level 5.3.3 does work
Embedded 5.3 and OS level 5.1.5 does not work

[Toulon7559]

Meanwhile latest lua-version is lua5.4:
installing lua5.3 at OS-level therefore not really easy anymore.

Just alternative thinking:
how to 'force' a lua-script to apply a specific lua-basis?
Such construction widely used to aim a Python-script by dedicated first script-line, either at Python 2.7, or at Python3.x.
In the quoted examples they use a 'first line'

Code: Select all

#!/usr/bin/lua5.1 

, but does not seem effective in my test-script, concluding that the required lua-files probably are located elsewhere.
What 'first line' would be applicable to aim the script to applicable Domoticz-Lua5.3.x? [Or wrong way of thinking?]

[waaren]

Meanwhile latest lua-version is lua5.4:
installing lua5.3 at OS-level therefore not really easy anymore.

sudo apt install lua5.3 works on my system

Just alternative thinking:
how to 'force' a lua-script to apply a specific lua-basis?
Such construction widely used to aim a Python-script by dedicated first script-line, either at Python 2.7, or at Python3.x.
In the quoted examples they use a 'first line'

Code: Select all

#!/usr/bin/lua5.1 

#! is named "shebang" and is a Unix way of executing scripts. When you ask the OS to execute a file it'll figure out that this is not a normal executable file, and the #! at the start serves as a magic marker which instructs the OS to execute the command after the #! and wiring up that command so this file becomes an argument of that command.
So this works for scripts on the OS level but it does not work in an embedded environment like Lua in domoticz

The basis for Lua scripts in domoticz is fixed; it is the internal Lua version.
To help Lua finding external modules you have to make sure their location is in the package.path string. In your case it would be something like

Code: Select all

package.path = '/usr/local/share/lua/5.1.5/?.lua;' .. package.path  

[Toulon7559]

@waaren

Correct understanding that you direct me to look at the folder /home/pi/domoticz/scripts/lua for e.g. the files JSON.lua, xml2lua.lua and Xmlparser.lua?

[waaren]

.. folder /home/pi/domoticz/scripts/lua for e.g. the files JSON.lua, xml2lua.lua and Xmlparser.lua?

Yes the default JSON decoder / coder and XML decoder / coder for domoticz scripts are in that folder and can be used in domoticz classic Lua- or dzVents scripts without using other libraries.

See my examples in this post in the part after
"Using existing JSON for Lua and dzVents"

[Toulon7559]

Using waaren's examples, download/reading/input seems successful:
now tuning for my applications.

Output/upload is the reverse side, e.g. for ftp-upload of files.

For a lua-implementation looking at the 2 scriptlines below

Code: Select all

-- Upload JSON-/XML-file to destination computer by URL-call
curl -T JSON_Light1.json ftp://192.168.1.24:8021/upload --user myname:mypassword

Looks correct, but result from Domoticz' log:

Code: Select all

syntax error near '-'

Anybody a hint related to this error report?
Not sure whether pointing to '-' refers to -T or to --user

Also wondering how to fill-in/comply in practise for the end-string

Code: Select all

 --user myname:mypassword

example string probably best explanation.

[waaren]

For a lua-implementation looking at the 2 scriptlines below

Code: Select all

-- Upload JSON-/XML-file to destination computer by URL-call
curl -T JSON_Light1.json ftp://192.168.1.24:8021/upload --user myname:mypassword

Because you only share these two lines it is kind of a shot in the dark to advise but if you only want to execute the curl with the string var JSON_Light.json you could try

Code: Select all

-- Upload JSON-/XML-file to destination computer by URL-call
os.execute('curl --user myname:mypassword -T ' .. JSON_Light1.json .. ' ftp://192.168.1.24:8021/upload' )

[Toulon7559]

Under StackOverflow found arguments why inclusion in the curl-string of username & password is not so clever:
https://stackoverflow.com/questions/259 ... d-password

Perhaps not critical for mutual calls on local LAN (as shown in the previous messages), but seems unwise for remote calls over internet.

[Toulon7559]

@waaren

Your dzVents example-file from this message runs without problem, after 'opening' the local networks as described in the Wiki for dzVents.

Aiming at application with lua-scripts, more biased to get the 'classic version'.
The 'classic' file from that same message as test tuned to calling a local Virtual Device mimicking Buienradar THB, should be uncomplicated [?].
However, stuck at the lines calling lunajson and calling JSON,
with Domoticz' log reporting that the module 'lunajson' cannot be found, respectively module 'JSON' cannot be found.

Code: Select all

    local lunajson = require 'lunajson'

and

Code: Select all

    local defaultJSON = require "JSON"     -- use generic JSON.lua

Especially wondering about the apparent invisibility of JSON.lua, because that file JSON.lua is sitting side by side with the testscript in folder /home/pi/domoticz/scripts/lua

For future application, not only looking at extraction of data for the sensors in that message, but also looking at fields such as ServerTime, LastUpdate and HaveTimeOut. Purpose: establish whether communication is running and what last time of communication (for calculation of validity time).
Any hints? => Continued in this thread

[waaren]

Any hints?

You need to adjust the package.path

Cannot remember if I posted this Lua test code before..

Code: Select all

commandArray = {}
    local domoticzIP = '192.168.192.116' -- change to your domoticz IP
    local domoticzPort = 8084  -- change to your domoticz domoticzPort
    local deviceIDX = 133 -- change to your deviceIDX

    function os.caption(cmd)
        local file = io.popen(cmd)
        local output = file:read('*all')
        local rc = { file:close() }
        return output, rc[3]
    end

    function getos()
    -- Unix, Linux varients
        local os = os.caption("uname -o 2>/dev/null")
        return ( os or 'Windows')
    end

    print('____________________________________________________________________________')
    print('Lua Tests and comparing lunajson and existing JSON ')
    print('OS: ' .. tostring(getos()) )

    -- Lua versions
    print("Lua version in domoticz: " .. _VERSION)
    print("Lua version in OS: " .. os.caption('lua -v'))
    print('Package path before prepend: ' .. package.path)
    package.path = 'scripts/lua/?.lua;' .. package.path -- prepend to package path to allow use of mudules in scripts/lua
    print('Package path after prepend: ' .. package.path)

    -- same for both versions
    local httpString = 'http://' .. domoticzIP ..':' .. domoticzPort .. '/json.htm?type=devices&rid='  .. deviceIDX
    local jsoncmd = 'curl ' .. '"' .. httpString .. '"'
    local jsonraw, rc =  os.caption(jsoncmd )
    print (' ReturnCode: ' .. rc )

    -- Based on lunajson decoder
    local lunajson = require 'lunajson'
    local jsonparse = lunajson.decode( jsonraw )
    print('lunajson json to table (Some device information .....')

    local counter = 0
    for key,value  in pairs(jsonparse.result[1]) do
        counter = counter + 1
         print(tostring(key) .. ': ' .. tostring(value))
         if counter > 5 then break end
    end

    print( "Sunrise is at " ..  jsonparse["Sunrise"] )
    print( "Data:  " .. jsonparse.result[1].Data )

    print('existing JSON for Lua and dzVents..')
    -- Based on existing JSBON decoder
    local defaultJSON = require "JSON"     -- use generic JSON.lua
    local jsonparse = defaultJSON:decode( jsonraw )

    print( "Sunrise is at " ..  jsonparse["Sunrise"] )
    print( "Data:  " .. jsonparse.result[1].Data )
    print('____________________________________________________________________________')



return commandArray

[Toulon7559]

Many roads lead to Rome .........

Found a slightly different (older) approach to link to JSON and to dissect a selected json-file:
viewtopic.php?f=61&t=22090&hilit=darksky+script#p170296