dzVents Garbage pick-up alternating dates two pick-ups

[plugge]

Hi,
I use waaren's getGarbageDates.lua for [ dzVents >= 2.4 ]. It works, but it cannot handle two different pick-up types on the same date.
Type 1 ("restgft") is picked-up every week
Type 2 ("plastic") is picked-up every other week on the same date as Type 1.
It seems the script checks for only one pick-up Type per date and ignores additional pick-ups of another type on the same date.
Or does it?

[waaren]

I use waaren's getGarbageDates.lua for [ dzVents >= 2.4 ]. It works, but it cannot handle two different pick-up types on the same date.
Type 1 ("restgft") is picked-up every week
Type 2 ("plastic") is picked-up every other week on the same date as Type 1.
It seems the script checks for only one pick-up Type per date and ignores additional pick-ups of another type on the same date.
Or does it?

What do you expect and what is not working ? Is it the alert device, the text device and/or the notification?
If you send me the script via PM as you use it now including zipcode / housenumber, I will have a look and see if I can make it work for you?

[plugge]

Hi waaren,

I have two types of garbage collection "plastic" and "restgft" occurring on the same date, once every two weeks, but every week "restgft"
The devices:

Garbage collection pickup dates.PNG (9.65 KiB) Viewed 2108 times

Next garbage pickup date.PNG (8.04 KiB) Viewed 2108 times

Ofcourse the Alert device lists only one, but text device should list two pickups on 2020-09-03 and on 2020-09-17.
I have the impression that the current script cannot handle two pick-up type on the same date. Am I correct?

See the list below of the table "ophaaldagen" I retrieved, as an example:

Code: Select all

------------- snip snip -------
{"nameType":"pmd","type":"plastic","date":"2020-09-03"},      ----> Ok, this one is in the virtual text device  and in the Alert device 
{"nameType":"rest\/gft","type":"restgft","date":"2020-09-03"}, -----> this one is skipped, i.e., not in the virtual text device
{"nameType":"rest\/gft","type":"restgft","date":"2020-09-10"},  -----> this one is skipped, i.e., not in the virtual text device
{"nameType":"rest\/gft","type":"restgft","date":"2020-09-17"}, -----> this one is skipped, i.e., not in the virtual text device
{"nameType":"pmd","type":"plastic","date":"2020-09-17"},        -----> this one is skipped, i.e., not in the virtual text device
{"nameType":"textiel","type":"textiel","date":"2020-09-22"},      ---> Ok, this one is in the virtual text device
{"nameType":"rest\/gft","type":"restgft","date":"2020-09-24"}, -----> this one is skipped, i.e., not in the virtual text device
{"nameType":"papier","type":"papier","date":"2020-09-25"},     ---> Ok, this one is in the virtual text device
--------------snip snip -----------------------------------

[plugge]

P.S. I've also sent you a PM

[plugge]

Thanks to waaren, this dzVents script handles multiple (at least two) garbage pickups on the same date.

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4.28 ]

By waaren
this script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the various types of household garbage
the alert device will contain the date and type for the garbagecollection that will arrive first

            History:
            20180705: Start Coding
            20180725: First public release
            20180819: Changed target URL to mijnafvalwijzer
            20181108: use curl because of badly formatted json return
            20200614: sort lines before posting them to text device
            20200614: code cleanup
            20200614: Prepared for period close to end of year (ophaaldagenNext)
            20200615: Fixed notification bug
            20200615: Add choice for notification subsystem(s)
            20200830: Change method of determining next collection dates

      tbc   20201210: interpret dates in next year ??

]]--

return
{
        on =
        {
            timer =
            {
                --'every 1 minutes',
                'at 18:10',
                'at 08:10'
            },

            devices =
            {
                -- 'getGarbage',                -- Only for test purposes can be ignored
            },

            httpResponses =
            {
                'getGarbage_Response' -- Trigger to handle Json part
            },
        },

        logging =
        {
            level = domoticz.LOG_ERROR, -- set to LOG_DEBUG when something does not work as expected
            marker = 'collectGarbage',
        },

        data =
        {
            garbage =
            {
                initial = {},              -- Keep a copy of last json just in case
            },
        },

    execute = function(dz, item)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = '1111AA'          -- Your zipcode like '3085RA'
        local myHousenumber = 1                 -- Your housenumber like 38
        local myTextDevice  = 'Garbage type pickup dates'     -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = 'Next garbage pickup date'    -- Name with quotes or idx without when created as virtual alert device

        local myNotificationTable =
        {
             -- table with one or more notification systems.
             -- uncomment the notification systems that you want to be used
             -- Can be one or more of

             -- dz.NSS_FIREBASE_CLOUD_MESSAGING,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_HTTP,
             -- dz.NSS_KODI,
             -- dz.NSS_LOGITECH_MEDIASERVER,
             -- dz.NSS_NMA,
             -- dz.NSS_PROWL,
             -- dz.NSS_PUSHALOT,
             -- dz.NSS_PUSHBULLET,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_PUSHSAFER,
             dz.NSS_TELEGRAM,
        }

        local longGarbageName =
        {
          ['restgft'] =  'Kliko',            --'Restafval en GFT',
          ['plastic'] =  'Plastic',          --'Plastic, Blik en Drankkartons',
          ['papier'] =  'Papier en karton',
          ['textiel'] =  'Textiel',
          ['kerstbomen'] = 'Kerstbomen'
        }

        --++++---------------------------- No changes required below this line --------------------------------------------

        local function collectGarbageDates(secondsFromNow)
            local getGarbage_url  = 'http://json.mijnafvalwijzer.nl/?method=postcodecheck&postcode='  ..
                                    myZipcode .. '&street=&huisnummer=' ..
                                    myHousenumber .. '&toevoeging'
            dz.openURL  ({  url = getGarbage_url ,
                            callback = 'getGarbage_Response' }).afterSec(secondsFromNow)
        end

        -- Add entry to log and notify to set subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify('Garbage',message, dz.PRIORITY_HIGH, dz.SOUND_DEFAULT, "" , myNotificationTable)
        end

        local function convertDateFormat(dateString, fromPattern, toFormat)
            local fromPattern = fromPattern or '(%d+)-(%d+)-(%d+)'
            local toFormat = toFormat or '%a %d %b'
            local runyear, runmonth, runday= dateString:match(fromPattern)
            return os.date(toFormat, os.time({year = runyear, month = runmonth, day = runday}) )
        end

        local function text(lines)
            if dz.utils.deviceExists(myTextDevice) then
               dz.devices(myTextDevice).updateText( '<tt>' .. table.concat(lines, '\n') .. '</tt>\n')
            end
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function alert(lines)
            if dz.utils.deviceExists(myAlertDevice) then
                dz.devices(myAlertDevice).updateAlertSensor(alertLevel(lines.delta), lines[1])
            end

            if dz.time.matchesRule('at 08:00-17:00') and lines.delta == 0  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected today', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            elseif dz.time.matchesRule('at 18:00-22:00') and lines.delta == 1  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected tomorrow', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            end
        end

        function table.merge (t1, t2)
            for k,v in ipairs(t2) do
                table.insert(t1, v)
            end
            return t1
        end

        local function selectRecords(t)
            dateRecords = {}
            for _, garbage in ipairs(t) do
                if garbage.date >= dz.time.rawDate then
                    if dateRecords[garbage.date] then
                         table.insert(dateRecords[garbage.date], longGarbageName[garbage.type] )
                    else
                        dateRecords[garbage.date] = { longGarbageName[garbage.type] }
                    end
                end
            end
            return dateRecords
        end

        local function makeLines(t)
            local startDate = dz.time
            local futureDay = 0
            local lineCounter = 1
            local textLines = {}

            while lineCounter < 5 and futureDay < 60 do
                local traverseDate = startDate.addDays(futureDay).rawDate
                if t[traverseDate] then
                    local textLine = convertDateFormat(traverseDate) .. ': ' .. table.concat(t[traverseDate], ' & ')
                    if lineCounter == 1 then textLines['delta'] = futureDay end
                    textLines[lineCounter] = textLine
                    lineCounter = lineCounter + 1
                end
                futureDay = futureDay + 1
            end

            return textLines
        end

       local function handleResponse()

            if not(item.json) then
                item.json = dz.utils.fromJSON(item.data:gsub('"ophaaldagen:','"ophaaldagen":'))  -- Some responses are even crippled in this part
            end

            if type(item.json) == 'table' then
                rt = selectRecords(table.merge(item.json.data.ophaaldagen.data, item.json.data.ophaaldagenNext.data))
                dz.data.garbage = rt
            else
                errorMessage('Problem with response (no data). Try using data from earlier run')
                rt  = dz.data.garbage         -- json empty. Get last valid from dz.data
                if #rt < 1 then               -- No valid data in dz.data either
                    errorMessage('No previous data. Are zipcode and housenumber ok and available in afvalkalender?')
                    return false
                end
            end

            textLines = makeLines(rt)
            alert(textLines)
            text(textLines)

        end

        -- Main
        if item.isHTTPResponse then
            if item.ok then
                handleResponse()
            else
                errorMessage('Problem with response (not ok)')
                collectGarbageDates(600)                                           -- response not OK, try again after 10 minutes
            end
        else
            collectGarbageDates(1)
        end
    end
}

[waaren]

New version because http ://json.mijnafvalwijzer.nl/ - only accepts calls from paying customers.

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4.28 ]

This script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the various types of household garbage
the alert device will contain the date and type for the garbage collection that will arrive first

A notification is send in the morning when garbage collection will happen today and in the evening when garbage collection will happen tomorrow.

            History:
            20180705: Start Coding
            20180725: First public release
            20180819: Changed target URL to mijnafvalwijzer
            20181108: use curl because of badly formatted json return
            20200614: sort lines before posting them to text device
            20200614: code cleanup
            20200614: Prepared for period close to end of year (ophaaldagenNext)
            20200615: Fixed notification bug
            20200615: Add choice for notification subsystem(s)
            20200830: Change method of determining next collection dates
            20200930: Scraping HTTP return because http://json.mijnafvalwijzer.nl/ - only accepts calls from paying customers

      tbc   20201210: interpret dates in next year ??

]]--

return
{
        on =
        {
            timer =
            {
                'at 18:10',
                'at 08:10'
            },

            devices =
            {
                'getGarbage',                -- Only for test purposes can be ignored
            },

            httpResponses =
            {
                'getGarbage_Response' -- Trigger to handle HTTP response
            },
        },

        logging =
        {
            level = domoticz.LOG_ERROR, -- set to LOG_DEBUG when something does not work as expected
            marker = 'collectGarbage',
        },

        data =
        {
            garbage =
            {
                initial = {},              -- Keep a copy of yeardates just in case
            },
        },

    execute = function(dz, item)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = '1111AA'          -- Your zipcode like '3085RA'
        local myHousenumber = 1                 -- Your housenumber like 38
        local myTextDevice  = 'Garbage type pickup dates'     -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = 'Next garbage pickup date'    -- Name with quotes or idx without when created as virtual alert device

        local myNotificationTable =
        {
             -- table with one or more notification systems.
             -- uncomment the notification systems that you want to be used
             -- Can be one or more of

             -- dz.NSS_FIREBASE_CLOUD_MESSAGING,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_HTTP,
             -- dz.NSS_KODI,
             -- dz.NSS_LOGITECH_MEDIASERVER,
             -- dz.NSS_NMA,
             -- dz.NSS_PROWL,
             -- dz.NSS_PUSHALOT,
             -- dz.NSS_PUSHBULLET,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_PUSHSAFER,
             dz.NSS_TELEGRAM,
        }

        local myGarbageLabels =
        {
          ['Restafval'] =  'Kliko',             --'Restafval en GFT',
          ['Plastic'] =  'Plastic',             --'Plastic, Blik en Drank kartons',
          ['Papier'] =  'Papier en karton',
          ['Textiel'] =  'Textiel',
          ['Kerstbomen'] = 'Kerstbomen'
        }

        --++++---------------------------- No changes required below this line --------------------------------------------

        local months =
        {
            ['januari'] = 1,
            ['februari'] = 2,
            ['maart'] = 3,
            ['april'] = 4,
            ['mei'] = 5,
            ['juni'] = 6,
            ['juli'] = 7,
            ['augustus'] = 8,
            ['september'] = 9,
            ['oktober'] = 10,
            ['november'] = 11,
            ['december'] = 12,
        }

        local function collectGarbageDates(delay)
            local getGarbage_url = 'https://www.mijnafvalwijzer.nl/nl/' .. myZipcode .. '/' .. myHousenumber
            dz.openURL(
            {
                url = getGarbage_url ,
                callback = 'getGarbage_Response'
            }).afterSec(delay)
        end

        -- Add entry to log and notify to set subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify('Garbage',message, dz.PRIORITY_HIGH, dz.SOUND_DEFAULT, "" , myNotificationTable)
        end

        local function convertDateFormat(dateString, fromPattern, toFormat)
            local fromPattern = fromPattern or '(%d+)-(%d+)-(%d+)'
            local toFormat = toFormat or '%a %d %b'
            local runyear, runmonth, runday= dateString:match(fromPattern)
            return os.date(toFormat, os.time({year = runyear, month = runmonth, day = runday}) )
        end

        local function text(lines)
            if dz.utils.deviceExists(myTextDevice) then
               dz.devices(myTextDevice).updateText( '<tt>' .. table.concat(lines, '\n') .. '</tt>\n')
            end
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function alert(lines)
            if dz.utils.deviceExists(myAlertDevice) then
                dz.devices(myAlertDevice).updateAlertSensor(alertLevel(lines.delta), lines[1])
            end

            if dz.time.matchesRule('at 08:00-17:00') and lines.delta == 0  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected today', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            elseif dz.time.matchesRule('at 18:00-22:00') and lines.delta == 1  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected tomorrow', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            end
        end

        function table.merge (t1, t2)
            for k,v in ipairs(t2) do
                table.insert(t1, v)
            end
            return t1
        end

        local function selectRecords(t)
            dz.data.garbage = t -- store for future use
            for date in pairs(t) do
                if date < dz.time.rawDate then
                    t[date] = nil
                end
            end
            return t
        end

        local function makeLines(t)
            local startDate = dz.time
            local futureDay = 0
            local lineCounter = 1
            local textLines = {}

            while lineCounter < 5 and futureDay < 60 do
                local traverseDate = startDate.addDays(futureDay).rawDate
                if t[traverseDate] then
                    local textLine = convertDateFormat(traverseDate) .. ': ' .. table.concat(t[traverseDate], ' & ')
                    if lineCounter == 1 then textLines['delta'] = futureDay end
                    textLines[lineCounter] = textLine
                    lineCounter = lineCounter + 1
                end
                futureDay = futureDay + 1
            end
            return textLines
        end

        local function handleResponse(t, freshData)
            local garbage = {}
            if freshData then
                local date = ''
                for line in t:gmatch('[^\r\n]+') do  -- read response line by line
                    if line:find('<span class=%"span%-line%-break%">%a+%s%d+%s%a') then
                        local dutchDate = line:match('%b><'):sub(2,-2)
                        date = dz.time.year .. '-' .. dz.utils.leadingZeros(months[dutchDate:match('%a+%s+%d+%s+(%a+)')],2) .. '-' .. dz.utils.leadingZeros(dutchDate:match('%a+%s+(%d+)'),2)
                    elseif line:find('<span class=%"afvaldescr%">') then
                        local garbageLong = line:match('%b><'):sub(2,-2)
                        if garbage[date] then
                            table.insert(garbage[date], myGarbageLabels[garbageLong:match("(%a+)%.*")])
                        else
                            garbage[date] =  { myGarbageLabels[garbageLong:match("(%a+)%.*")] }
                        end
                    end
                end
            elseif t ~= nil and next(dz.data.garbage) then
                garbage = t
            else
                errorMessage('Problem with response and no previous data')
                return
            end

            textLines = makeLines(selectRecords(garbage))
            alert(textLines)
            text(textLines)
        end

        -- Main
        if item.isHTTPResponse then
            if item.ok then
                handleResponse(item.data, true)
            else
                errorMessage('Problem with response (not ok) Using previous data')
                handleResponse(dz.data.garbage)
                collectGarbageDates(600)                                           -- response not OK, try again after 10 minutes
            end
        else
             collectGarbageDates(1)
        end
    end
}

[plugge]

New version because http ://json.mijnafvalwijzer.nl/ - only accepts calls from paying customers.

Thanks waaren, works like a charm!
And good learning material for scraping in dzVents.

[deluccio]

Can someone help me out on this? I have used the script from "Waaren".

Because a notification looks helpfull for me as well I tried to make the script working as well. But Helaas peanut butter, i failt

Maybe i missed some steps:

1. I have created a dummy Alert "Next garbage pickup date" (Idx=56) and a dummy Text "Garbage type pickup dates" (Idx=55)

I have adjusted the code, filled in my postal code and housenumber and last but not least i changed the Device name to my Idx code.

Code: Select all

        local myTextDevice  = 55     -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = 56    -- Name with quotes or idx without when created as virtual alert device

Where it goes wrong? What step did i missed?

[plugge]

Can someone help me out on this? I have used the script from "Waaren".

Could you copy paste here what the log shows after you set logging to level = domoticz.LOG_DEBUG?
NB: also set a temporary new timer to force the script to execute, otherwise you have to wait until this evening

Code: Select all

        logging =
        {
            level = domoticz.LOG_DEBUG, -- set to LOG_DEBUG when something does not work as expected
            marker = 'collectGarbage',
        },

[deluccio]

Spoiler: show

Can someone help me out on this? I have used the script from "Waaren".

Could you copy paste here what the log shows after you set logging to level = domoticz.LOG_DEBUG?
NB: also set a temporary new timer to force the script to execute, otherwise you have to wait until this evening

Code: Select all

        logging =
        {
            level = domoticz.LOG_DEBUG, -- set to LOG_DEBUG when something does not work as expected
            marker = 'collectGarbage',
        },

Hmm noob question perhaps: but how to do this

NB: also set a temporary new timer to force the script to execute, otherwise you have to wait until this evening

The only (debug) response i get for now is:

Code: Select all

2020-10-19 13:58:05.410 Status: dzVents: Write file: /usr/local/domoticz/var/scripts/dzVents/generated_scripts/Afvalwijzer.lua

[plugge]

Hmm noob question perhaps: but how to do this

The only (debug) respons i get for now is:

Code: Select all

2020-10-19 13:58:05.410 Status: dzVents: Write file: /usr/local/domoticz/var/scripts/dzVents/generated_scripts/Afvalwijzer.lua

See the code below.

Code: Select all

timer =
            {
               'at 14:30',   -- set this on a time in the future, close to your current time, save the script and wait until the script fired.
                'at 18:10',
                'at 08:10'
            },

Don't forget to change the debug level too in the script. See my previous message

And maybe, to be sure set:
Setup -> Settings -> Other -> dzVents ->
Enabled: On (=green)
Log level = Debug (eveything)

[deluccio]

Perfect it worked now

Hehe i was to unpatient The timer setting solved the "problem"

[plugge]

Good to hear that. Thanx for the feedback.

[deluccio]

No i have got another "problem". I receive a <empty> result for GFT pickup date.

I thought the solution was easy and I just simply added a extra Garbage label "GFT" but this did not worked for me.

Code: Select all

['GFT'] = 'Groente, Fruit en Tuinafval'

Code: Select all

local myGarbageLabels =
        {
          ['Restafval'] =  'Kliko',             --'Restafval en GFT',
          ['Plastic'] =  'Plastic',             --'Plastic, Blik en Drank kartons',
          ['Papier'] =  'Papier en karton',
          ['Textiel'] =  'Textiel',
          ['Kerstbomen'] = 'Kerstbomen',
	 ['GFT'] = 'Groente, Fruit en Tuinafval'
		  
        }

It should be noted that in my region "Restafval and "GFT" are collected separately.

[waaren]

Now i have got another "problem". I receive a <empty> result for GFT pickup date.
It should be noted that in my region "Restafval and "GFT" are collected separately.

If you send me your myZipcode and myHousenumber ( via PM ) I wil have a look.

[deluccio]

Now i have got another "problem". I receive a <empty> result for GFT pickup date.
It should be noted that in my region "Restafval and "GFT" are collected separately.

If you send me your myZipcode and myHousenumber ( via PM ) I wil have a look.

Thanks for your help. You solved my problem.

I almost was right, but I used the wrong tag. Instead of GFT it should be Groente.

Code: Select all

['Groente'] = 'Groente, Fruit en Tuinafval'

[deluccio]

I hijacked the forumtopic a bit and perhaps i should create a new topic, but let's give it a try and collect all information about "afvalwijzer" on one place.

I've got a new problem... i reinstalled Domoticz (new server, fresh reinstall) and used the same script as before. The problem is that i get incorrect pickup dates:

Example:
The next pickup date (which is also mentioned on afvalwijzer.nl) is 25 Nov.

The scripts shows:
24 Nov
25 Nov

Both with the same type, and this happens to every type with double information, except for the date, which is always -1

Anyone else has the same issue ?

[waaren]

I've got a new problem... i reinstalled Domoticz (new server, fresh reinstall) and used the same script as before. The problem is that i get incorrect pickup dates:

If you send me your zipcode / housenumber and the script as you have it now and if possible the complete log output of the script in debug mode (via PM), I will have a look.

[waaren]

New version needed to enable script to deal with situations where afvalwijzer shows pickup dates for two years.

Code: Select all

--[[ getGarbageDates.lua for [ dzVents >= 2.4.28 ]

This script is only useful in those  areas of the Netherlands where the household garbage collector is connected to afvalwijzer.nl

Enter your zipcode and housenumber in the appropriate place between the lines starting with --++++
Next is to set your virtual text and or virtual alert device.

the text device will contain the most nearby collectdates for the various types of household garbage
the alert device will contain the date and type for the garbage collection that will arrive first

A notification is send in the morning when garbage collection will happen today and in the evening when garbage collection will happen tomorrow.

            History:
            20180705: Start Coding
            20180725: First public release
            20180819: Changed target URL to mijnafvalwijzer
            20181108: use curl because of badly formatted json return
            20200614: sort lines before posting them to text device
            20200614: code cleanup
            20200614: Prepared for period close to end of year (ophaaldagenNext)
            20200615: Fixed notification bug
            20200615: Add choice for notification subsystem(s)
            20200830: Change method of determining next collection dates
            20200930: Scraping HTTP return because http://json.mijnafvalwijzer.nl/ - only accepts calls from paying customers
            20201118: Use scriptVar for httpResponse, -callBack  and marker
            20201118: Don't assume this year but get year(s) from response

]]--

local scriptVar = 'getGarbage_20201118'

return
{
        on =
        {
            timer =
            {
                'at 18:10',
                'at 08:10',
            },

            devices =
            {
                'getGarbage',                -- Only for test purposes can be ignored
            },

            httpResponses =
            {
                scriptVar, -- Trigger to handle HTTP response
            },
        },

        logging =
        {
            level = domoticz.LOG_ERROR, -- set to LOG_DEBUG when something does not work as expected
            marker = scriptVar,
        },

        data =
        {
            garbage =
            {
                initial = {},              -- Keep a copy of yeardates just in case
            },
        },

    execute = function(dz, item)

        --++++--------------------- Mandatory: Set your values and device names below this Line --------------------------------------
        local myZipcode     = '1111AA'          -- Your zipcode like '3085RA'
        local myHousenumber = 1                 -- Your housenumber like 38
        local myTextDevice  = 'Garbage type pickup dates'     -- Name with quotes or idx without when created as virtual text device
        local myAlertDevice = 'Next garbage pickup date'    -- Name with quotes or idx without when created as virtual alert device

        local myNotificationTable =
        {
             -- table with one or more notification systems.
             -- uncomment the notification systems that you want to be used
             -- Can be one or more of

             -- dz.NSS_FIREBASE_CLOUD_MESSAGING,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_HTTP,
             -- dz.NSS_KODI,
             -- dz.NSS_LOGITECH_MEDIASERVER,
             -- dz.NSS_NMA,
             -- dz.NSS_PROWL,
             -- dz.NSS_PUSHALOT,
             -- dz.NSS_PUSHBULLET,
             -- dz.NSS_PUSHOVER,
             -- dz.NSS_PUSHSAFER,
             dz.NSS_TELEGRAM,
        }

        local myGarbageLabels =
        {
          ['Restafval'] =  'Kliko',             --'Restafval',
          ['Plastic'] =  'Plastic',             --'Plastic, Blik en Drank kartons',
          ['Papier'] =  'Papier en karton',
          ['Textiel'] =  'Textiel',
          ['Kerstbomen'] = 'Kerstbomen',
          ['Groente'] = 'Groente, Fruit en Tuinafval'
        }

        --++++---------------------------- No changes required below this line --------------------------------------------

        local months =
        {
            ['januari'] = 1,
            ['februari'] = 2,
            ['maart'] = 3,
            ['april'] = 4,
            ['mei'] = 5,
            ['juni'] = 6,
            ['juli'] = 7,
            ['augustus'] = 8,
            ['september'] = 9,
            ['oktober'] = 10,
            ['november'] = 11,
            ['december'] = 12,
        }

        local function collectGarbageDates(delay)
            local getGarbage_url = 'https://www.mijnafvalwijzer.nl/nl/' .. myZipcode .. '/' .. myHousenumber
            dz.openURL(
            {
                url = getGarbage_url ,
                callback = scriptVar,
            }).afterSec(delay)
        end

        -- Add entry to log and notify to set subsystems
        local function errorMessage(message)
            dz.log(message,dz.LOG_ERROR)
            dz.notify('Garbage',message, dz.PRIORITY_HIGH, dz.SOUND_DEFAULT, "" , myNotificationTable)
        end

        local function convertDateFormat(dateString, fromPattern, toFormat)
            local fromPattern = fromPattern or '(%d+)-(%d+)-(%d+)'
            local toFormat = toFormat or '%a %d %b'
            local runyear, runmonth, runday= dateString:match(fromPattern)
            return os.date(toFormat, os.time({year = runyear, month = runmonth, day = runday}) )
        end

        local function text(lines)
            if dz.utils.deviceExists(myTextDevice) then
               dz.devices(myTextDevice).updateText( '<tt>' .. table.concat(lines, '\n') .. '</tt>\n')
            end
        end

        local function alertLevel(delta)
            if delta < 2 then return dz.ALERTLEVEL_RED end
            if delta < 3 then return dz.ALERTLEVEL_YELLOW end
            if delta < 4 then return dz.ALERTLEVEL_ORANGE end
            return dz.ALERTLEVEL_GREEN
        end

        local function alert(lines)
            if dz.utils.deviceExists(myAlertDevice) then
                dz.devices(myAlertDevice).updateAlertSensor(alertLevel(lines.delta), lines[1])
            end

            if dz.time.matchesRule('at 08:00-17:00') and lines.delta == 0  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected today', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            elseif dz.time.matchesRule('at 18:00-22:00') and lines.delta == 1  then
                dz.notify('Huisafval',lines[1]:match('%: (.*)')  .. ' will be collected tomorrow', dz.PRIORITY_NORMAL,dz.SOUND_DEFAULT, "" , myNotificationTable)
            end
        end

        function table.merge (t1, t2)
            for k,v in ipairs(t2) do
                table.insert(t1, v)
            end
            return t1
        end

        local function selectRecords(t)
            dz.data.garbage = t -- store for future use
            for date in pairs(t) do
                if date < dz.time.rawDate then
                    t[date] = nil
                end
            end
            return t
        end

        local function makeLines(t)
            -- dz.utils.dumpTable(t)
            local startDate = dz.time
            local futureDay = 0
            local lineCounter = 1
            local textLines = {}

            while lineCounter < 5 and futureDay < 60 do
                local traverseDate = startDate.addDays(futureDay).rawDate
                if t[traverseDate] then
                    local textLine = convertDateFormat(traverseDate) .. ': ' .. table.concat(t[traverseDate], ' & ')
                    if lineCounter == 1 then textLines['delta'] = futureDay end
                    textLines[lineCounter] = textLine
                    lineCounter = lineCounter + 1
                end
                futureDay = futureDay + 1
            end
            return textLines
        end

        local function handleResponse(t, freshData)
            local garbage = {}
            if freshData then
                local year = '%d%d%d%d'
                local date = ''
                local garbageYear, dutchDate

                for line in t:gmatch('[^\r\n]+') do  -- read response line by line
                    if line:find('<div id=%"jaar%-' .. year .. '%" class=%"ophaaldagen%">') then
                        garbageYear = line:match(year)
                    elseif line:find('<span class=%"span%-line%-break%">%a+%s%d+%s%a') then
                        dutchDate = line:match('%b><'):sub(2,-2)
                        date = (garbageYear or dz.time.year) .. '-' ..
                               dz.utils.leadingZeros(months[dutchDate:match('%a+%s+%d+%s+(%a+)')],2) .. '-' ..
                               dz.utils.leadingZeros(dutchDate:match('%a+%s+(%d+)'),2)
                    elseif line:find('<span class=%"afvaldescr%">') then
                        local garbageLong = line:match('%b><'):sub(2,-2)
                        if garbage[date] then
                            table.insert(garbage[date], myGarbageLabels[garbageLong:match("(%a+)%.*")])
                        else
                            garbage[date] =  { myGarbageLabels[garbageLong:match("(%a+)%.*")] }
                        end
                    end
                end
            elseif t ~= nil and next(dz.data.garbage) then
                garbage = t
            else
                errorMessage('Problem with response and no previous data')
                return
            end

            textLines = makeLines(selectRecords(garbage))
            alert(textLines)
            text(textLines)

        end

        -- Main
        if item.isHTTPResponse then
            if item.ok then
                handleResponse(item.data, true)
            else
                errorMessage('Problem with response (not ok) Using previous data')
                handleResponse(dz.data.garbage)
                collectGarbageDates(600)                                           -- response not OK, try again after 10 minutes
            end
        else
             collectGarbageDates(1)
        end
    end
}