dzVents script to delete history data of devices for which you don't want any history-log

Moderator: leecollings

Post Reply
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

dzVents script to delete history data of devices for which you don't want any history-log

Post by waaren »

requires domoticz V2020.2 build >= 13027

Warning!!
Please make sure you have a recent and tested backup as this script potentially
deletes many rows from the history tables (on my test system database size went from 23 MB to 6 MB)

Code: Select all

--[[

        Script to delete history data from database of the devices for which you don't want any historylog
        requires domoticz V2020.2 build >= 13027

        Warning!! Please make sure you have a recent and tested backup as this script potentially
                  deletes many rows from the history tables (on my test system database size went from 23 MB to 6 MB)

        there are two ways to have this script ignore devices (and thus do not touch its history)
            1. Add device ID to the comma separated variable "excludeDeviceString"
            2. Add the string "ExcludeFromHistoryDelete" to the description field of the device

        History:
        20210301 initial release
        20200302 Split JSON in used and unused to minimize risk of too many lines

]]--

local scriptVar = 'deleteHistory_'

return
{
    on =
    {
        timer =
        {
          'at 1:53', -- Preferable at the the start of a quiet period  
        },
        customEvents =
        {
            scriptVar,
        },
        httpResponses =
        {
            scriptVar .. '*',
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG, -- change to domoticz.LOG_ERROR when all ok
        marker = scriptVar,
    },

    execute = function(dz, item)

        --- enter below the comma separated list of devices that are to be excluded from deleteHistory 
        --- Or enter the string 'ExcludeFromHistoryDelete' anywhere in the device description

        local excludeDeviceString = '3,4,5,6,10,446,447,834,835,192' 

        --- No changes required below this line

        local function getDevices(used)
            dz.openURL(
            {
                url = dz.settings['Domoticz url'] .. '/json.htm?type=devices&displayhidden=1&used=' .. tostring(used == 'used'),
                callback = scriptVar .. used,
            })
        end

        local function deleteSwitchLogHistory(idx, delay)
            dz.openURL(dz.settings['Domoticz url'] .. '/json.htm?type=command&param=clearlightlog&idx=' .. idx).afterSec(delay)
            return delay + 2 -- Give command some time to be processed before next one
        end

        local function deleteCalendarHistory(idx, delay )
            dz.openURL
            (
                dz.settings['Domoticz url'] .. '/json.htm?type=command&param=deletedaterange'..
                '&idx=' .. idx ..
                '&fromdate=2010-01-01' ..
                '&todate=9999-12-31'
            ).afterSec(delay)
            return delay + 5 -- Give command some time to be processed before next one
        end

        local function vacuumDatebase(delay)
            dz.log(delay, dz.LOG_DEBUG)
            dz.openURL(dz.settings['Domoticz url'] .. '/json.htm?type=command&param=vacuumdatabase').afterSec(delay + 10 )
        end

        local function hasCalendarHistory(dv)
           return not(dv.Type:lower():find('switch') or dv.SubType:lower():find('switch') or dv.SubType =='AC' or dv.deviceSubType == 'Text')
        end

        local function createExcludedDevicesTable()
            local excludeDevices = {}
            for id in excludeDeviceString:gmatch("%d*") do
                excludeDevices[id] = true
            end
            return excludeDevices
        end

        local function processResult(rt)

            local delay = 0
            local excludeDevices = createExcludedDevicesTable()

            for key, dv in ipairs(rt) do
                if excludeDevices[dv.idx] or dv.Description:find('ExcludeFromHistoryDelete') then
                    dz.log('Exclude: '.. dv.idx .. ' -->> ' .. dv.Name .. ': ' .. dv.Type .. ', ' ..dv.SubType .. ', ' .. dv.Description, dz.LOG_DEBUG)
                elseif hasCalendarHistory(dv) then
                    dz.log('Calendar: '.. dv.idx .. ' -->> ' .. dv.Name .. ': ' .. dv.Type .. ', ' ..dv.SubType .. ', ' .. dv.Description, dz.LOG_DEBUG)
                    delay = deleteCalendarHistory(dv.idx, delay)
                else
                    dz.log('Switch: '.. dv.idx .. ' -->> ' .. dv.Name .. ': ' .. dv.Type .. ', ' ..dv.SubType .. ', ' .. dv.Description, dz.LOG_DEBUG)
                    delay = deleteSwitchLogHistory(dv.idx, delay)
                end
            end

            return delay
        end

        -- main
        if item.isTimer then
            getDevices('used')
        elseif item.isCustomEvent then
            getDevices('unused')
        elseif not(item.json) then
            dz.log('Problem retrieving data ' , dz.LOG_ERROR)
        else -- httpResponse
            local delay = processResult(item.json.result)
            if item.trigger:find('unused') then
                vacuumDatebase(delay)
            else
                dz.emitEvent(scriptVar).afterSec(delay)
            end
        end

    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
zicht
Posts: 272
Joined: Sunday 11 May 2014 11:09
Target OS: Windows
Domoticz version: 2023.1+
Location: NL
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by zicht »

Nice & thanks.
Will test this weekeind.

Any particular reason to want to shrink the db this way ?
(with short log on 1 day and lamps on 3 days, i am able to get to <8Mb, but i do not notice any discomfort when >10Mb)
Rpi & Win x64. Using : cam's,RFXCom, LaCrosse, RFY, HuE, google, standard Lua, Tasker, Waze traveltime, NLAlert&grip2+,curtains, vacuum, audioreceiver, smart-heating&cooling + many more (= automate all repetitive simple tasks)
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by EddyG »

With short log sensors on 3 days and lights/switches on 1 day I still have 40Mb, but still have no problem.
My every 5 minutes database backup folder contains almost 34 Gb of databases (3 days) and creates no problem on my NAS.
But I can use the script the other way around just to include some devices of which I do not want any history.
So Tnx @waaren
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by PieterS »

Hi Waaren,

Nice solution. But last week I was searching for a solution to delete the log of a single (text)-counter "Domoticz uptime". It creates a log that I delete manually once in a while...
This is the device:

Image

This is the script:

Code: Select all

return { 
    on = {   timer  =   {"every 5 minutes"}},  
                
    execute = function(dz)
        local uptimeTextDevice = dz.devices("Domoticz uptime")
        
        local minutes = dz.startTime.minutesAgo%60
        local days    = dz.startTime.daysAgo
        local hours   = dz.startTime.hoursAgo%24
        
        if days == 1 then
            DagText = " dag"
        else
            DagText = " dagen"
        end
        
        uptimeTextDevice.updateText(days ..DagText..", " .. hours .. " uur en " .. minutes .. " minuten"  )
    end
}
I had in mind that there is a function in DZvents to delete the entry in the log immediately and just keep the result in a variable of the IDX.. Now there are about 300 entries a day.. The log is useless in my opinion. :?:

You might have a hint?
Synology with Domoticz build (V2024.7) in Docker
User avatar
HansieNL
Posts: 964
Joined: Monday 28 September 2015 15:13
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by HansieNL »

@PieterS
I'm using this script to save uptime as a variable (thanks to @Waaren):

Code: Select all

-- dzuptime.lua
-- 
-- Set Domoticz Uptime as User Variable string
-- 

return { 
	--[[ logging = {
		level = domoticz.LOG_ERROR,   -- Select one of LOG_INFO, LOG_DEBUG, LOG_ERROR, LOG_FORCE to override system log level
		marker = "(Uptime)"
			  }, ]]--
	on = {
		timer = {
			"every minute"
				}
		 },

	execute = function(dz)
		local days	= dz.startTime.daysAgo
		local hours	= dz.startTime.hoursAgo%24
		local minutes	= dz.startTime.minutesAgo%60

		if days == 1 then DagenText = " dag" else DagenText = " dagen" end
		if hours == 1 then UrenText = " uur" else UrenText = " uren" end
		if minutes == 1 then MinutenText = " minuut" else MinutenText = " minuten" end
        
		dz.variables("Uptime").set(days ..DagenText.. ", " .. hours ..UrenText.. " en " .. minutes ..MinutenText)
	end
}
Blah blah blah
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by waaren »

zicht wrote: Tuesday 02 March 2021 19:05 Any particular reason to want to shrink the db this way ?
I prefer and use a shortlog of 14 days but I am only interested in a couple devices of the > 400 used in my system. Until now I used a bash script triggered in cron doing more or less the same as this script to keep my production database size within reasonable limits but using dzVents makes it OS independent.

And.. the backup schedule / retention I use (24 hourly backups / 31 daily backups / 12 monthly backups) make that every MB multiplies with 67 :)
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by waaren »

PieterS wrote: Tuesday 02 March 2021 19:46 I had in mind that there is a function in DZvents to delete the entry in the log immediately and just keep the result in a variable of the IDX.. Now there are about 300 entries a day.. The log is useless in my opinion. :?:

You might have a hint?
Can you try this?

Code: Select all


return {
    on = {   timer  =   {"every 5 minutes"}},

    execute = function(dz)
        local uptimeTextDevice = dz.devices("Domoticz uptime")

        local function deleteTextLog(idx)
            dz.openURL(dz.settings['Domoticz url'] .. '/json.htm?type=command&param=clearlightlog&idx=' .. idx)
        end

        local minutes = dz.startTime.minutesAgo%60
        local days    = dz.startTime.daysAgo
        local hours   = dz.startTime.hoursAgo%24

        if days == 1 then
            DagText = " dag"
        else
            DagText = " dagen"
        end

        uptimeTextDevice.updateText(days ..DagText..", " .. hours .. " uur en " .. minutes .. " minuten"  )

        if dz.time.matchesRule('every hour') then -- change to your liking
            deleteTextLog(uptimeTextDevice.id)
        end
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by PieterS »

H waaren,

Tried your script in several ways but it didn't empty the logfile by : I just put the 3 rows near the end into my script, I copied your whole script but no luck...

No errors in logfile of Domoticz..

Changed log level to Debug (everything)

Code: Select all

2021-03-03 16:59:00.496 Status: dzVents: Info: ------ Start internal script: Uptime:, trigger: "every minute"
2021-03-03 16:59:00.497 Status: dzVents: Debug: Processing device-adapter for Domoticz uptime: Text device
2021-03-03 16:59:00.497 Status: dzVents: Info: ------ Finished Uptime
2021-03-03 16:59:00.497 Status: dzVents: Debug: - UpdateDevice = {["idx"]=785, ["_trigger"]=true, ["nValue"]=0, ["sValue"]="39 dagen, 21 uur en 29 minuten"}
2021-03-03 16:59:00.683 Status: dzVents: Debug: Processing device-adapter for Domoticz uptime: Text device
2021-03-03 16:59:00.684 Status: dzVents: Debug: - Device: Domoticz uptime 
This is the log of today

Image

I use the latest stable version of Domoticz V 2020.2 (build 12231) and dzVents version 3.0.1

What is wrong? :roll:
Synology with Domoticz build (V2024.7) in Docker
User avatar
erem
Posts: 230
Joined: Tuesday 27 March 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: 2021.1
Location: Amsterdam/netherlands
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by erem »

@PieterS
PieterS wrote: Wednesday 03 March 2021 17:18 I use the latest stable version of Domoticz V 2020.2 (build 12231) and dzVents version 3.0.1
which part of
waaren wrote: Tuesday 02 March 2021 16:14 requires domoticz V2020.2 build >= 13027
did you not understand :mrgreen: :mrgreen:
Regards,

Rob
User avatar
HansieNL
Posts: 964
Joined: Monday 28 September 2015 15:13
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by HansieNL »

@PieterS
I switched from variable to this script, so I can use an Uptime device in Domoticz.
Every hour the log is cleared, so the script is working o.k.
Did you try the script as Waaren did post without changes?
Edit: I’m using stable 2020.2, but my dzvents version is 3.0.2. Did you update and upgrade your system?
Blah blah blah
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by waaren »

PieterS wrote: Wednesday 03 March 2021 17:18 Tried your script in several ways but it didn't empty the logfile by : I just put the 3 rows near the end into my script, I copied your whole script but no luck...

What is wrong? :roll:
Did you read this part of the wiki ?
__________________________________________________________________________________________________________________________
When not yet familiar with dzVents please start with reading Get started Before implementing (~ 5 minutes). Special attention please for "In Domoticz go to Setup > Settings > Other and in the section EventSystem make sure the checkbox 'dzVents enabled' is checked. Also make sure that in the Security section in the settings you allow 127.0.0.1 to not need a password. dzVents uses that port to send certain commands to Domoticz. Finally make sure you have set your current location in Setup > Settings > System > Location, otherwise there is no way to determine nighttime/daytime state."
___________________________________________________________________________________________________________________________

In below version I activated debug logging. Can you please try it? It might show enough information to see what goes wrong on your system.

Code: Select all


return {
    on = {   timer  =   {"every 5 minutes"}},
	
	logging =
	{
		level = domoticz.LOG_DEBUG,
		marker = 'upTime',
	},
	
    execute = function(dz)
        local uptimeTextDevice = dz.devices("Domoticz uptime")

        local function deleteTextLog(idx)
            dz.openURL(dz.settings['Domoticz url'] .. '/json.htm?type=command&param=clearlightlog&idx=' .. idx)
        end

        local minutes = dz.startTime.minutesAgo%60
        local days    = dz.startTime.daysAgo
        local hours   = dz.startTime.hoursAgo%24

        if days == 1 then
            DagText = " dag"
        else
            DagText = " dagen"
        end

        uptimeTextDevice.updateText(days ..DagText..", " .. hours .. " uur en " .. minutes .. " minuten"  )

        if dz.time.matchesRule('every hour') then -- change to your liking
            deleteTextLog(uptimeTextDevice.id)
        end
    end
}

Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by waaren »

erem wrote: Wednesday 03 March 2021 17:26 which part of
waaren wrote: Tuesday 02 March 2021 16:14 requires domoticz V2020.2 build >= 13027
did you not understand :mrgreen: :mrgreen:
Maybe the part where it was not in? :o

@PieterS post is about another script.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: dzVents script to delete history data of devices for which you don't want any history-log

Post by PieterS »

Thanks Waaren for quick reply. I use several dzVentz scripts. So settings should be oke.
@HansieNL: A made a mistake in the version of dzVents. Should be version 3.0.11 :oops: The host is a Synology and Domoticz is a package compiled by Jadahl.

@waaren: Now I copied your full latest script and saved it in a new name.. Changed timers in 2 and 5 minutes.
I had in mind that lines 29 till 31 should do the job.. But lines 13 and 14 are important too :? Sorry for that.
This is a part in the log what might give you some information. Says not much to me..

Code: Select all

2021-03-03 19:20:01.011 Status: dzVents: Info: upTime: ------ Start internal script: Uptime2:, trigger: "every 2 minutes"
2021-03-03 19:20:01.011 Status: dzVents: Debug: upTime: OpenURL: url = http://127.0.0.1:8084/json.htm?type=command&param=clearlightlog&idx=785
2021-03-03 19:20:01.011 Status: dzVents: Debug: upTime: OpenURL: method = GET
2021-03-03 19:20:01.011 Status: dzVents: Debug: upTime: OpenURL: post data = nil
2021-03-03 19:20:01.011 Status: dzVents: Debug: upTime: OpenURL: headers = nil
2021-03-03 19:20:01.011 Status: dzVents: Debug: upTime: OpenURL: callback = nil
2021-03-03 19:20:01.011 Status: dzVents: Info: upTime: ------ Finished Uptime2
But clearing the log is succesfull :D
Image

Thanks everyone a lot for your help :P
Synology with Domoticz build (V2024.7) in Docker
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest