dzVents get garbage collection dates (various)  [SOLVED]

Moderator: leecollings

Bikey
Posts: 331
Joined: Sunday 22 February 2015 12:19
Target OS: Linux
Domoticz version: 2020.x
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by Bikey »

That script actually works! Well sort off, it complains with this error, although the devices are updated:

Code: Select all

 Error: dzVents: Error: (3.0.9) collectGarbage: Error parsing json to LUA table: /opt/domoticz/scripts/dzVents/../lua/JSON.lua:1234: /opt/domoticz/scripts/dzVents/../lua/JSON.lua:1016: Lua script execution exceeds maximum number of lines
So what I think needs to be done, is that as a first step in the script below, the response should be parsed to minimize it down to only contain the "afhaaldagen" part.

Code: Select all

local function handleResponse()
            triggerObject.json = dz.utils.fromJSON(triggerObject.data)         -- dzVents does nor recognize the response as pure JSON so conversion is required
            if #triggerObject.json < 1 then
                dz.data.garbage    = triggerObject.json.data.ophaaldagen.data      -- Store this part in dz.data 
                rt = triggerObject.json.data.ophaaldagen.data                      -- and in table
            else
               errorMessage("Problem with response (no data) using data from earlier run")
               rt  = dz.data.garbage                       -- json empty. Get last valid from dz.data
               if #rt < 1 then                              -- No valid data in dz.data either
                  errorMessage("No previous data. are zipcode and housenumber ok and in afvalkalender ?")
                  return false
               end
            end
As said before I figured out that the regEx for this is:

Code: Select all

.*(\"ophaaldagen\".*),.*\"ophaaldagenNext\":.*
So only need to work out in Lua how to handle that.
User avatar
HansieNL
Posts: 960
Joined: Monday 28 September 2015 15:13
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: dzVents get garbage collection dates (various)

Post by HansieNL »

waaren wrote: Friday 12 June 2020 16:28
HansieNL wrote: Friday 12 June 2020 16:00 @waaren What is the latest dzvents script for MijnAfvalwijzer?
Below is the latest one I made. No idea if it still working (I don't use it myself) but should not be very hard to get it to work if mijnAfvalwijzer stil returns something that looks like a JSON.
Spoiler: show

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4 ]

this script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the four types of household garbage
the alert device will contain the date and type for the garbagecollecion that will arrive first

]]--

return {
        on      =   {   timer              =  { "at 00:05","at 08:00" },    -- daily run twice
                        devices            =  { "Garbage" },               -- Only for test purposes can be ignored  
                        httpResponses      =  { "getGarbage_Response" }     -- Trigger the handle Json part
                },
        logging =   {   level              =   domoticz.LOG_DEBUG,          
                        marker             =   "collectGarbage"      },

        data    =   {   garbage            =    { initial = {}      },       -- Keep a copy of last json just in case  
                        lastStartline      =    { initial = 1       },  
                        lastEndline        =    { initial = 1000    },  
                    },             

    execute = function(dz, triggerObject)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = "3085RA"            -- Your zipcode like "3085RA"
        local myHousenumber =  38                 -- Your housenumber like 38
        local myTextDevice  = "GarbageText"       -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = "GarbageAlert"      -- Name with quotes or idx without when created as virtual alert device
        --++++---------------------------- Set your values and device names above this Line --------------------------------------------

        local myYear = os.date("%Y")
        garbageTypes  = {"restafval","gft","papier","plastic"}

        local function collectGarbageDates(secondsFromNow)
            local getGarbage_url  = "http://json.mijnafvalwijzer.nl/?method=postcodecheck&postcode="  .. 
                                    myZipcode .. "&street=&huisnummer=" .. 
                                    myHousenumber .. "&toevoeging" 
            dz.openURL  ({  url = getGarbage_url ,
                            method = "GET",
                            callback = "getGarbage_Response" }).afterSec(secondsFromNow)
        end

        -- Add entry to log and notify to all subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify(message)
        end

        local function string2Epoch(dateString) -- seconds from epoch based on stringdate (used by string2Date)
            -- Assuming a date pattern like: yyyy-mm-dd
            local pattern = "(%d+)-(%d+)-(%d+)"
            local runyear, runmonth, runday= dateString:match(pattern)
            local convertedTimestamp = os.time({year = runyear, month = runmonth, day = runday})
            return convertedTimestamp
        end

        local function string2Date(str,fmt)             -- convert string from json into datevalue
            if fmt then return os.date(fmt,string2Epoch(str)) end
            return os.date(" %A %d %B, %Y",string2Epoch(str))
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function setGarbageAlertDevice(alertDevice,alertText,alertDate)
            local delta = tonumber(string2Date(alertDate,"%d")) - tonumber(os.date("%d"))  -- delta in days between today and first garbage collection date
            dz.devices(alertDevice).updateAlertSensor(alertLevel(delta),alertText)
            dz.log("\nalertLevel: " .. alertLevel(delta) .. ", alertText: " .. alertText,dz.LOG_DEBUG)
            return (delta == 0)
        end

        local function longGarbageName(str)                                        -- Use descriptive strings
            str = tostring(str)
            str = str:gsub("plastic","  Plastic verpakkingen, blik en drinkpakken ")
            str = str:gsub("gft","  Groente-, fruit- en tuin afval            ")
            str = str:gsub("papier","  Papier en kartonnen verpakkingen          ")
            str = str:gsub("restafval" ,"  Restafval                                ")
            return str
        end

       local function handleResponse()
            triggerObject.json = dz.utils.fromJSON(triggerObject.data)         -- dzVents does nor recognize the response as pure JSON so conversion is required
            if #triggerObject.json < 1 then
                dz.data.garbage    = triggerObject.json.data.ophaaldagen.data      -- Store this part in dz.data 
                rt = triggerObject.json.data.ophaaldagen.data                      -- and in table
            else
               errorMessage("Problem with response (no data) using data from earlier run")
               rt  = dz.data.garbage                       -- json empty. Get last valid from dz.data
               if #rt < 1 then                              -- No valid data in dz.data either
                  errorMessage("No previous data. are zipcode and housenumber ok and in afvalkalender ?")
                  return false
               end
            end
            
            local lastEndline = 1000
            if dz.data.lastStartline ~= 1 then
                lastEndline = dz.data.lastStartline + 10
            end
            
            local garbageLines = ""
            local typeEarliestDate
            local overallEarliestDate   = "2999-12-31"       -- Hopefully we will have a different garbage collection system by then
            local garbageToday = false
            local today = os.date("%Y-%m-%d")
            
            for i = 1,#garbageTypes do --walk the the type Table
                typeEarliestDate      = "2999-12-31"
                for j = dz.data.lastStartline,math.min(#rt,lastEndline) do                                 -- walk the response table
                    dz.log(rt[j].date .. ": " .. rt[j].type,dz.LOG_DEBUG)        
                    if  rt[j].date >= today and rt[j].date < typeEarliestDate and 
                        rt[j].type == garbageTypes[i] then              -- Keep date closest to today per type
                        typeEarliestDate =  rt[j].date
                        if  typeEarliestDate < overallEarliestDate then     -- date closest to today overall ?
                            overallEarliestDate = typeEarliestDate          -- keep date
                            overallEarliestType =  garbageTypes[i]          -- keep type
                            dz.data.lastStartline = j 
                        end
                        garbageLines = garbageLines .. string2Date(typeEarliestDate,"%a %e %b" ) .. longGarbageName(rt[j].type) .. " " .. "\n"
                        typeEarliestDate = rt[j].date  -- Keep date closest to today

                    end
                end
            end

            if myAlertDevice then   -- Update AlertDevice with nearby date / type
                garbageToday = setGarbageAlertDevice(  myAlertDevice,
                                                        longGarbageName(overallEarliestType) .. "\n" ..
                                                        string2Date(overallEarliestDate),
                                                        overallEarliestDate)
            end

            if myTextDevice then       -- Update defined virtual text device with dates / types
                dz.devices(myTextDevice).updateText(garbageLines)
                dz.log("\n" .. garbageLines,dz.LOG_DEBUG)
            end

            if dz.time.matchesRule("at 08:00-17:00") and garbageToday then
                dz.notify(longGarbageName(overallEarliestType) .. "will be collected today")
            end
        end

        -- Main
        if triggerObject.isHTTPResponse then
            if triggerObject.ok then
                handleResponse()
            else
                errorMessage("Problem with response from hvcgroep (not ok)")
                --collectGarbageDates(600)                            -- response not OK, try again after 10 minutes
            end
        else
            collectGarbageDates(1)
        end
    end
}
Thanx. AfvalWijzer does still works o.k. with your script. Only because I had no virtual alert device I got some errors, but all is working fine now.
Oops... I got 3 dates, but missing 2 next dates:
Wed 24 Jun Restafval
Wed 1 Jul Groente-, fruit- en tuin afval
Thu 16 Jul Papier en kartonnen verpakkingen

If I use this old script:
Spoiler: show

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4 ]

this script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the four types of household garbage
the alert device will contain the date and type for the garbagecollecion that will arrive first

]]--

return {
        on      =  { timer           =  { "at 00:05","at 08:00" },   -- daily run twice
    --  on      =  { timer           =  {"every 1 minutes"},         -- During test / debug (only one "on =" line can be active ) 
    --  on      =  { timer           =  {"never"},                   -- To deactivate 
                     httpResponses   =  { "getGarbage_Response" }    -- Trigger the handle Json part
                },

--  logging =   {   level     =   domoticz.LOG_DEBUG,              -- Remove the "-- at the beginning of this and next line for debugging the script
--                  marker    =   "collectGarbage"      },

    data    =   {   garbage     = {initial = {} },             -- Keep a copy of last json just in case
                },

    execute = function(dz, triggerObject)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = "nnnntt"            -- Your zipcode like "3085RA"
        local myHousenumber =  nn                 -- Your housenumber like 38
        local myTextDevice  = "GarbageText"       -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = "GarbageAlert"      -- Name with quotes or idx without when created as virtual alert device
        --++++---------------------------- Set your values and device names above this Line --------------------------------------------

        local myYear = os.date("%Y")
        garbageTypes  = {"restafval","gft","papier","plastic"}

        local function collectGarbageDates(secondsFromNow)
            local getGarbage_url  = "http://json.mijnafvalwijzer.nl/?method=postcodecheck&postcode="  .. 
                                    myZipcode .. "&street=&huisnummer=" .. 
                                    myHousenumber .. "&toevoeging" 
            dz.openURL  ({  url = getGarbage_url ,
                            method = "GET",
                            callback = "getGarbage_Response" }).afterSec(secondsFromNow)
        end

        -- Add entry to log and notify to all subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify(message)
        end

        local function string2Epoch(dateString) -- seconds from epoch based on stringdate (used by string2Date)
            -- Assuming a date pattern like: yyyy-mm-dd
            local pattern = "(%d+)-(%d+)-(%d+)"
            local runyear, runmonth, runday= dateString:match(pattern)
            local convertedTimestamp = os.time({year = runyear, month = runmonth, day = runday})
            return convertedTimestamp
        end

        local function string2Date(str,fmt)             -- convert string from json into datevalue
            if fmt then return os.date(fmt,string2Epoch(str)) end
            return os.date(" %A %d %B, %Y",string2Epoch(str))
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function setGarbageAlertDevice(alertDevice,alertText,alertDate)
            local delta = tonumber(string2Date(alertDate,"%d")) - tonumber(os.date("%d"))  -- delta in days between today and first garbage collection date
            dz.devices(alertDevice).updateAlertSensor(alertLevel(delta),alertText)
            dz.log("\nalertLevel: " .. alertLevel(delta) .. ", alertText: " .. alertText,dz.LOG_DEBUG)
            return (delta == 0)
        end

        local function longGarbageName(str)                                        -- Use descriptive strings
            str = tostring(str)
            str = str:gsub("plastic","  Plastic verpakkingen, blik en drinkpakken ")
            str = str:gsub("gft","  Groente-, fruit- en tuin afval            ")
            str = str:gsub("papier","  Papier en kartonnen verpakkingen          ")
            str = str:gsub("restafval" ,"  Restafval                                ")
            return str
        end

       local function handleResponse()
            triggerObject.json = dz.utils.fromJSON(triggerObject.data)         -- dzVents does nor recognize the response as pure JSON so conversion is required
            if #triggerObject.json < 1 then
                dz.data.garbage    = triggerObject.json.data.ophaaldagen.data      -- Store this part in dz.data 
                rt = triggerObject.json.data.ophaaldagen.data                      -- and in table
            else
               errorMessage("Problem with response (no data) using data from earlier run")
               rt  = dz.data.garbage                       -- json empty. Get last valid from dz.data
               if #rt < 1 then                              -- No valid data in dz.data either
                  errorMessage("No previous data. are zipcode and housenumber ok and in afvalkalender ?")
                  return false
               end
            end
            
            local garbageLines = ""
            local typeEarliestDate
            local overallEarliestDate   = "2999-12-31"       -- Hopefully we will have a different garbage collection system by then
            local garbageToday = false
            local today = os.date("%Y-%m-%d")
            
            for i = 1,#garbageTypes do --walk the the type Table
                typeEarliestDate      = "2999-12-31"
                for j = 1,#rt do                                 -- walk the response table
                    dz.log(rt[j].date .. ": " .. rt[j].type,dz.LOG_DEBUG)        
                    if  rt[j].date >= today and rt[j].date < typeEarliestDate and 
                        rt[j].type == garbageTypes[i] then              -- Keep date closest to today per type
                        typeEarliestDate =  rt[j].date
                        if  typeEarliestDate < overallEarliestDate then  -- date closest to today overall ?
                            overallEarliestDate = typeEarliestDate      -- keep date
                            overallEarliestType =  garbageTypes[i]            -- keep type
                        end
                        garbageLines = garbageLines .. string2Date(typeEarliestDate,"%a %e %b" ) .. longGarbageName(rt[j].type) .. " " .. "\n"
                        typeEarliestDate = rt[j].date  -- Keep date closest to today

                    end
                end
            end

            if myAlertDevice then   -- Update AlertDevice with nearby date / type
                garbageToday = setGarbageAlertDevice(  myAlertDevice,
                                                        longGarbageName(overallEarliestType) .. "\n" ..
                                                        string2Date(overallEarliestDate),
                                                        overallEarliestDate)
            end

            if myTextDevice then       -- Update defined virtual text device with dates / types
                dz.devices(myTextDevice).updateText(garbageLines)
                dz.log("\n" .. garbageLines,dz.LOG_DEBUG)
            end

            if dz.time.matchesRule("at 08:00-17:00") and garbageToday then
                dz.notify(longGarbageName(overallEarliestType) .. "will be collected today")
            end
        end

        -- Main
        if triggerObject.isHTTPResponse then
            if triggerObject.ok then
                handleResponse()
            else
                errorMessage("Problem with response (not ok)")
                collectGarbageDates(600)                            -- response not OK, try again after 10 minutes
            end
        else
            collectGarbageDates(1)
        end
    end
}
I got all dates (inclusing 2 next dates), but dates sorting is wrong :
Wed 24 Jun Restafval
Wed 17 Jun Groente-, fruit- en tuin afval
Thu 18 Jun Papier en kartonnen verpakkingen
Blah blah blah
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

HansieNL wrote: Friday 12 June 2020 18:20 Thanx. AfvalWijzer does still works o.k. with your script. Only because I had no virtual alert device I got some errors, but all is working fine now.
Oops... I got 3 dates, but missing 2 next dates:
Wed 24 Jun Restafval
Wed 1 Jul Groente-, fruit- en tuin afval
Thu 16 Jul Papier en kartonnen verpakkingen

If I use this old script:
I got all dates (inclusing 2 next dates), but dates sorting is wrong :
Wed 24 Jun Restafval
Wed 17 Jun Groente-, fruit- en tuin afval
Thu 18 Jun Papier en kartonnen verpakkingen
If you send me (via PM) a working zipcode / housenumber, if will have a look.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Bikey
Posts: 331
Joined: Sunday 22 February 2015 12:19
Target OS: Linux
Domoticz version: 2020.x
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by Bikey »

Don’t you get this error?

Code: Select all

Error: dzVents: Error: (3.0.9) ...1016: Lua script execution exceeds maximum number of lines
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

Bikey wrote: Friday 12 June 2020 20:57 Don’t you get this error?

Code: Select all

Error: dzVents: Error: (3.0.9) ...1016: Lua script execution exceeds maximum number of lines
If you send me (via PM) a working zipcode / housenumber, if will have a look.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
ronaldbro
Posts: 327
Joined: Thursday 15 November 2018 21:38
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by ronaldbro »

I just played a little with it and it looks like line 91 can be deleted.

Code: Select all

triggerObject.json = dz.utils.fromJSON(triggerObject.data)
I guess it's not needed anymore because triggerObject.json is filled already. While testing the error is gone and it works after removing this line.

@waaren, can you confirm that this is not needed anymore?
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

ronaldbro wrote: Friday 12 June 2020 22:16 I just played a little with it and it looks like line 91 can be deleted.

Code: Select all

triggerObject.json = dz.utils.fromJSON(triggerObject.data)
I guess it's not needed anymore because triggerObject.json is filled already. While testing the error is gone and it works after removing this line.

@waaren, can you confirm that this is not needed anymore?
If the return is already recognized by dzVents as a valid JSON then this line is not needed.
If I remember well, I added this forced conversion for cases where the return was not a 100% correct JSON. Then this line will try to force a conversion from the returned data into a Lua table.

This forced conversion should not cause a fatal error but a correcter code would be something like

Code: Select all

if not(triggerOject.isJSON) then dz.utils.fromJSON(triggerObject.data) end
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
ronaldbro
Posts: 327
Joined: Thursday 15 November 2018 21:38
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by ronaldbro »

Thanks waaren,

For your information these are the full errors this line gives

Code: Select all

 2020-06-12 22:09:03.935 Error: dzVents: Error: (3.0.9) collectGarbage: Error parsing json to LUA table: /home/pi/domoticz/scripts/dzVents/../lua/JSON.lua:1234: /home/pi/domoticz/scripts/dzVents/../lua/JSON.lua:1016: Lua script execution exceeds maximum number of lines
2020-06-12 22:09:03.935 Error: dzVents: Error: (3.0.9) collectGarbage: An error occurred when calling event handler Afvalwijzer
2020-06-12 22:09:03.936 Error: dzVents: Error: (3.0.9) collectGarbage: ...moticz/scripts/dzVents/generated_scripts/Afvalwijzer.lua:93: attempt to get length of a nil value (field 'json')
Where line 93 actually is line 91 in your script. I added a few debug lines...
Anyway, I guess a correct json is returned so if I understand well the line is not needed in this script.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

ronaldbro wrote: Friday 12 June 2020 22:41 For your information these are the full errors this line gives

Code: Select all

 2020-06-12 22:09:03.935 Error: dzVents: Error: (3.0.9) collectGarbage: Error parsing json to LUA table: /home/pi/domoticz/scripts/dzVents/../lua/JSON.lua:1234: /home/pi/domoticz/scripts/dzVents/../lua/JSON.lua:1016: Lua script execution exceeds maximum number of lines
2020-06-12 22:09:03.935 Error: dzVents: Error: (3.0.9) collectGarbage: An error occurred when calling event handler Afvalwijzer
2020-06-12 22:09:03.936 Error: dzVents: Error: (3.0.9) collectGarbage: ...moticz/scripts/dzVents/generated_scripts/Afvalwijzer.lua:93: attempt to get length of a nil value (field 'json')
Where line 93 actually is line 91 in your script. I added a few debug lines...
Anyway, I guess a correct json is returned so if I understand well the line is not needed in this script.
Thx for sharing!

I guess you get this error because of the very large JSON that is now converted to a table twice resulting in too many lines to be interpreted by the Lua engine of domoticz.
Using the isJSON check that I posted before this one should prevent calling the second (redundant) conversion.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Bikey
Posts: 331
Joined: Sunday 22 February 2015 12:19
Target OS: Linux
Domoticz version: 2020.x
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by Bikey »

I tried to reduce the result set by adding these lines which should reduce the result to only the "ophaaldagen" object, but that seem not to do anything, still getting the error that there are too many lines.

Code: Select all

triggerObject.data = triggerObject.data:match('.*(\"ophaaldagen\".*),.*\"ophaaldagenNext\":.*')
triggerObject.data = "{"..triggerObject.data.."}"
triggerObject.json = dz.utils.fromJSON(triggerObject.data)         -- dzVents does nor recognize the response as pure JSON so conversion is required
 
Is that because it already get's into problems by ingesting the response, even before the first line in script above?
User avatar
jvdz
Posts: 2210
Joined: Tuesday 30 December 2014 19:25
Target OS: Raspberry Pi / ODroid
Domoticz version: 4.107
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by jvdz »

Not sure how to code this in dzVents, but this is the regex I use to strip the majority of the returned data and fix the JSON syntax:

Code: Select all

webdata:match('(.-),\"mededelingen\":').."}}"
so maybe:

Code: Select all

triggerObject.data = triggerObject.data:match('(.-),\"mededelingen\":').."}}"
Jos
New Garbage collection scripts: https://github.com/jvanderzande/GarbageCalendar
Bikey
Posts: 331
Joined: Sunday 22 February 2015 12:19
Target OS: Linux
Domoticz version: 2020.x
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by Bikey »

Hi Jos,
Thanks for the tip, I actually got the idea from your script ;-)

My version is a try to improve that even a bit further by not only filtering everything out of the reply after the "ophaaldagen" object, but also everything before. If I test this regex in an online regex editor with the resultset from afvalwijzer it gets a nice result. I guess your code should work in DzVents as well, as this is basically also Lua.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

Bikey wrote: Saturday 13 June 2020 13:07 Is that because it already get's into problems by ingesting the response, even before the first line in script above?
Yes and unfortunately this is not something that can be avoided easily. The dzVents mechanism to automatically convert the data to a json is triggered but as your return is too large and completely following the JSON RFC, it fails.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

Bikey wrote: Saturday 13 June 2020 13:07 I tried to reduce the result set by adding these lines which should reduce the result to only the "ophaaldagen" object, but that seem not to do anything, still getting the error that there are too many lines.
can you try again after changing line

Code: Select all

triggerObject.json = dz.utils.fromJSON(triggerObject.data)
to

Code: Select all

            if not triggerObject.json then
                triggerObject.json = dz.utils.fromJSON(triggerObject.data:gsub('"ophaaldagen:','"ophaaldagen":'))         -- dzVents does nor recognize the response as pure JSON so conversion is required
            end
You will still see the error but script will continue and process the data.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
ronaldbro
Posts: 327
Joined: Thursday 15 November 2018 21:38
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by ronaldbro »

Hi waaren,

I don't understand why it's better to keep this line. If you remove it the error is gone and everything works.
I understand the idea or best practice to do this when getting a bad json and try to force it. But in this case it seems the site return a proper json and it gets parsed right away. So I assume the site will always return a proper json (ok, maybe this is wishful thinking...).
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

ronaldbro wrote:Hi waaren,

I don't understand why it's better to keep this line. If you remove it the error is gone and everything works.
If it works for you it's ok but I received private mails from other members (with ofcourse other zipcode / housenumbers) where the json needed fixing before it could be converted to a Lua table.




Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
ronaldbro
Posts: 327
Joined: Thursday 15 November 2018 21:38
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by ronaldbro »

That makes sense, didn't expect the same website to return a proper json for some addresses and not for other addresses.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by waaren »

Updated the script to work for full compliant addresses and for addresses that return a somewhat crippled JSON.
Also fixed the skipping of some dates and ordering of the dates.

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4 ]

this script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the four types of household garbage
the alert device will contain the date and type for the garbagecollecion that will arrive first

]]--

return {
        on      =   {   timer              =  { "at 00:05","at 08:00" },    -- daily run twice
                        devices            =  { "Garbage" },               -- Only for test purposes can be ignored  
                        httpResponses      =  { "getGarbage_Response" }     -- Trigger the handle Json part
                },
        logging =   {   level              =   domoticz.LOG_DEBUG,          
                        marker             =   "collectGarbage"      },

        data    =   {   garbage            =    { initial = {}      },       -- Keep a copy of last json just in case  
                        lastStartline      =    { initial = 1       },  
                        lastEndline        =    { initial = 1000    },  
                    },             

    execute = function(dz, triggerObject)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = "3085RA"            -- Your zipcode like "3085RA"
        local myHousenumber =  38                 -- Your housenumber like 38
        local myTextDevice  = "GarbageText"       -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = "GarbageAlert"      -- Name with quotes or idx without when created as virtual alert device
        --++++---------------------------- Set your values and device names above this Line --------------------------------------------

        local myYear = os.date("%Y")
        garbageTypes  = {"restafval","gft","papier","plastic"}

        local function collectGarbageDates(secondsFromNow)
            local getGarbage_url  = "http://json.mijnafvalwijzer.nl/?method=postcodecheck&postcode="  .. 
                                    myZipcode .. "&street=&huisnummer=" .. 
                                    myHousenumber .. "&toevoeging" 
            dz.openURL  ({  url = getGarbage_url ,
                            method = "GET",
                            callback = "getGarbage_Response" }).afterSec(secondsFromNow)
        end

        -- Add entry to log and notify to all subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify(message)
        end

        local function string2Epoch(dateString) -- seconds from epoch based on stringdate (used by string2Date)
            -- Assuming a date pattern like: yyyy-mm-dd
            local pattern = "(%d+)-(%d+)-(%d+)"
            local runyear, runmonth, runday= dateString:match(pattern)
            local convertedTimestamp = os.time({year = runyear, month = runmonth, day = runday})
            return convertedTimestamp
        end

        local function string2Date(str,fmt)             -- convert string from json into datevalue
            if fmt then return os.date(fmt,string2Epoch(str)) end
            return os.date(" %A %d %B, %Y",string2Epoch(str))
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function setGarbageAlertDevice(alertDevice,alertText,alertDate)
            local delta = tonumber(string2Date(alertDate,"%d")) - tonumber(os.date("%d"))  -- delta in days between today and first garbage collection date
            dz.devices(alertDevice).updateAlertSensor(alertLevel(delta),alertText)
            dz.log("\nalertLevel: " .. alertLevel(delta) .. ", alertText: " .. alertText,dz.LOG_DEBUG)
            return (delta == 0)
        end

        local function longGarbageName(str)                                        -- Use descriptive strings
            str = tostring(str)
            str = str:gsub("plastic","  Plastic verpakkingen, blik en drinkpakken ")
            str = str:gsub("gft","  Groente-, fruit- en tuin afval            ")
            str = str:gsub("papier","  Papier en kartonnen verpakkingen          ")
            str = str:gsub("restafval" ,"  Restafval                                ")
            return str
        end

       local function handleResponse()
             if not triggerObject.json then
                triggerObject.json = dz.utils.fromJSON(triggerObject.data:gsub('"ophaaldagen:','"ophaaldagen":'))         -- dzVents does nor recognize the response as pure JSON so conversion is required
            end
            
            if type(triggerObject.json) == 'table' then
                dz.data.garbage    = triggerObject.json.data.ophaaldagen.data      -- Store this part in dz.data 
                rt = triggerObject.json.data.ophaaldagen.data                      -- and in table
            else
               errorMessage("Problem with response (no data) using data from earlier run")
               rt  = dz.data.garbage                       -- json empty. Get last valid from dz.data
               if #rt < 1 then                              -- No valid data in dz.data either
                  errorMessage("No previous data. are zipcode and housenumber ok and in afvalkalender ?")
                  return false
               end
            end

            dz.utils.dumpTable(rt)
            
            local garbageLines = ""
            local typeEarliestDate
            local overallEarliestDate   = "2999-12-31"       -- Hopefully we will have a different garbage collection system by then
            local garbageToday = false
            local today = os.date("%Y-%m-%d")
            local garbage = {}
            
            for i = 1,#garbageTypes do --walk the the type Table
                typeEarliestDate      = "2999-12-31"
                for j = 1, math.min(#rt, #dz.data.garbage) do                                 -- walk the response table
                   if  rt[j].date >= today and rt[j].date < typeEarliestDate and  rt[j].type == garbageTypes[i] then -- Keep date closest to today per type
                        typeEarliestDate =  rt[j].date
                        if  typeEarliestDate < overallEarliestDate then     -- date closest to today overall ?
                            overallEarliestDate = typeEarliestDate          -- keep date
                            overallEarliestType =  garbageTypes[i]          -- keep type
                            dz.data.lastStartline = j 
                        end
                        garbage[string2Epoch(typeEarliestDate,"%a %e %b" )] = string2Date(typeEarliestDate,"%a %e %b" ) .. longGarbageName(rt[j].type)
                        typeEarliestDate = rt[j].date  -- Keep date closest to today
                    end
                end
            end
            
            -- Order the lines based on dates
            local keyTable = {}
            local garbageLines = ''
            for key in pairs(garbage) do 
                table.insert(keyTable, key) 
            end
            
            table.sort(keyTable)
            
            for _, key in ipairs(keyTable) do 
                garbageLines = garbageLines .. garbage[key] ..  '\n' 
            end
            
            -- Update the devices
            if myAlertDevice then   -- Update AlertDevice with nearby date / type
                garbageToday = setGarbageAlertDevice(  myAlertDevice,
                                                        longGarbageName(overallEarliestType) .. "\n" ..
                                                        string2Date(overallEarliestDate),
                                                        overallEarliestDate)
            end

            if myTextDevice then       -- Update defined virtual text device with dates / types
                dz.devices(myTextDevice).updateText(garbageLines)
                dz.log("\n" .. garbageLines,dz.LOG_DEBUG)
            end

            if dz.time.matchesRule("at 08:00-17:00") and garbageToday then
                dz.notify(longGarbageName(overallEarliestType) .. " will be collected today")
            end
        end

        -- Main
        if triggerObject.isHTTPResponse then
            if triggerObject.ok then
                handleResponse()
            else
                errorMessage("Problem with response from hvcgroep (not ok)")
                --collectGarbageDates(600)                            -- response not OK, try again after 10 minutes
            end
        else
            collectGarbageDates(1)
        end
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
hjzwiers
Posts: 187
Joined: Friday 12 January 2018 8:26
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: dzVents get garbage collection dates (various)

Post by hjzwiers »

I had not checked this issue for a while, updated the script and its working. Thmx
ronaldbro
Posts: 327
Joined: Thursday 15 November 2018 21:38
Target OS: Raspberry Pi / ODroid
Domoticz version: 2020.2
Location: Netherlands
Contact:

Re: dzVents get garbage collection dates (various)

Post by ronaldbro »

Yep, working like a charm.

Now I need three small bulbs above my containers, lol.
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests