Feature Request: Delete entire history of device

[akamming]

Hi,

I was dzvents scripting again with Virtual Sensors. So mostly what happens is:
- Create a new virtual device
- Start creating a script (mostly takes a few minutes beore you have it perfect), which already gives the first incorrect readings in the history of the device (cause it was not updated. Escpially in case of counters, cause if you don't give them a value in the first few minutes, domoticz increases a big increase when you give the first reading for the 1st day
- and then most of the time the upcoming days you have some more ideas to perfect it.
- Until it's perfect, but then you want to throw away the history. For which thare are 3 ways:
1. Shift-click on the graphs to remove individual readings (a lot of work!!)
2. go into the DB (eg. "remove from Meter" and "remove from Meter_ calender statements". Less work. but user unfriendly and in case of mistakes, you corrupt the DB.
3. Delete and recreate the sensor (but then you only have to a few minutes to point to the new device, otherwise you have the same problem again)

So wouldn't it be great if there was a button in the GUI, which just throws away the entire history of a sensor?

[JayDotR]

Hi,

Alternative way if you are not so common-use-DBA . As I encounter the same issues you described when doing development work on scripts to get some data to domoticz, before committing data to the sensor I use old-school programming method, ergo Log to console

Code: Select all

return {
    active = true,
    on = {
        --timer = { 'every <final schedule>' }, -- Adjust interval as needed
        timer = { 'every 1 minutes' }, -- Adjust interval as needed
        
    },
    logging = {
        --level = domoticz.LOG_DEBUG, -- this one is used when developping the plugin/script
        level = domoticz.LOG_INFO,
        marker = 'Multi-Interface Bandwidth - ',
    },
    execute = function(dz, triggeredItem)
				dz.log( 'I did not find _d.ap_table - Re-initialize.', dz.LOG_DEBUG )
				-- placeholder code for updating the sensor here

Big advantage :
- you don't mess up data in the domoticz.db
- if you are more DBA you don't need to check in which of the tables your sensor is writing to
- and DBA's best practice : never update/delete without the right select

However, a very interesting feature request : +1 requester from me to the development team

[akamming]

Hi,

Alternative way if you are not so common-use-DBA . As I encounter the same issues you described when doing development work on scripts to get some data to domoticz, before committing data to the sensor I use old-school programming method, ergo Log to console

Code: Select all

return {
    active = true,
    on = {
        --timer = { 'every <final schedule>' }, -- Adjust interval as needed
        timer = { 'every 1 minutes' }, -- Adjust interval as needed
        
    },
    logging = {
        --level = domoticz.LOG_DEBUG, -- this one is used when developping the plugin/script
        level = domoticz.LOG_INFO,
        marker = 'Multi-Interface Bandwidth - ',
    },
    execute = function(dz, triggeredItem)
				dz.log( 'I did not find _d.ap_table - Re-initialize.', dz.LOG_DEBUG )
				-- placeholder code for updating the sensor here

Big advantage :
- you don't mess up data in the domoticz.db
- if you are more DBA you don't need to check in which of the tables your sensor is writing to
- and DBA's best practice : never update/delete without the right select

However, a very interesting feature request : +1 requester from me to the development team

I agree that might be a workaround if it's just one sensor. But even then you have to make sure the value is populated between creation and the first time data is saved to the history (every 5 mins) otherwise you would already have the 1st incorrect value.

But i recently wrote a dzvents script which maintains 8 sensors, so in that case this is undoable. Cause then within 5 mins i would have to
- (re)create the 8 sensors
- adjust the code the new 8 IDX'numers of the devices
- run the script
so then it's undoable..

[waltervl]

You can also use this procedure:
1. Disable your dzvents updating script (easily done in the event system editor)
2. Remove incorrect sensor devices (you can take your time as the update script is disabled)
3. Create new sensor devices. (you can take your time as the update script is disabled)
4. Enable and if needed edit your dzvents updating script (easily done in the event system editor)

https://wiki.domoticz.com/Events

Event scripts can be enabled or disabled using the "On/Off" checkbox. Disabled events are marked red in the events list.

[akamming]

You can also use this procedure:
1. Disable your dzvents updating script (easily done in the event system editor)
2. Remove incorrect sensor devices (you can take your time as the update script is disabled)
3. Create new sensor devices. (you can take your time as the update script is disabled)
4. Enable and if needed edit your dzvents updating script (easily done in the event system editor)

https://wiki.domoticz.com/Events

Event scripts can be enabled or disabled using the "On/Off" checkbox. Disabled events are marked red in the events list.

unfortunately No, you will have the same issue: You only have max 5 mins between step 3 en 4 (including the editing of the script to reference the newly created sensors). Otherwise you will have 0 values in the history of those sensors. Which is especially nasty if they are of the type counter.

You can delete the values in the DB or the using shift click on the graph. But this is a lot of manual work. Hence the feature request making this much more intuitive and less error prone.

[waltervl]

Why is a couple of 0 counter values in the beginning nasty? I never have found this to be a problem.

[akamming]

Why is a couple of 0 counter values in the beginning nasty? I never have found this to be a problem.

Try it with a counter which stores an actual much higher value as first value. Then it counts a way too high number as first value scrwwing up the graphs and statistics

[HvdW]

My method
I have a few sensors mySwitch, myTextSensor, mySelectorSwitch, myCounter and you can add more.
I use those for experimenting.

When the script is tested and tuned I create the production sensors.

Because my patience is not endless I'm often using on devices 'mySwitch' to activate the script while testing instead of every minute.
Instead of looking at the log I also use myTextSensor a lot to display relevant data from the script.
I put these sensors in the dashboard and can act and see the results at once.

[waltervl]

Why is a couple of 0 counter values in the beginning nasty? I never have found this to be a problem.

Try it with a counter which stores an actual much higher value as first value. Then it counts a way too high number as first value scrwwing up the graphs and statistics

That has nothing to do with removing old test data. That is standard behavior when you fill a new Domoticz counter device with data from a long running counter device. Also a new API call will not solve this issue....

When I run into this use case I let the counter run for 1 day and the next day I remove the first spike from the monthly graph. Then everything runs smoothly.

[akamming]

Why is a couple of 0 counter values in the beginning nasty? I never have found this to be a problem.

Try it with a counter which stores an actual much higher value as first value. Then it counts a way too high number as first value scrwwing up the graphs and statistics

That has nothing to do with removing old test data. That is standard behavior when you fill a new Domoticz counter device with data from a long running counter device. Also a new API call will not solve this issue....

When I run into this use case I let the counter run for 1 day and the next day I remove the first spike from the monthly graph. Then everything runs smoothly.

This is exactly a reason why it has to do with this feature request: Since the 0 value which is always there when you create a new virtual device is stored in the history, in a lot of cases you will have to delete this later on to get a correct history. So that is why i mention feature request.

A better solution would be that no values are added to the history of a virtual device until the first measurement.

[waltervl]

When you add a new counter device it will probably start with 0.
And every 5 minutes it will be overwritten by the 0 value. (You can by the way switch this off in the settings/tab Log History that domoticz only stores a real changed value).

And when you load the first value from your counter, say Solar panels production already produced the last years in total 51241000 Wh
and then every 5 minutes a delta of 100 Wh due to current production, you will get a huge peak of 51241 kWh and a low total first day production of 1 kWh.

But that huge peak of 51241 kWh you will always get, whatever reset method you have chosen. This cannot be prevented. It is also not related to 1 or more 0 values in the log.

[akamming]

When you add a new counter device it will probably start with 0.
And every 5 minutes it will be overwritten by the 0 value. (You can by the way switch this off in the settings/tab Log History that domoticz only stores a real changed value).

And when you load the first value from your counter, say Solar panels production already produced the last years in total 51241000 Wh
and then every 5 minutes a delta of 100 Wh due to current production, you will get a huge peak of 51241 kWh and a low total first day production of 1 kWh.

But that huge peak of 51241 kWh you will always get, whatever reset method you have chosen. This cannot be prevented. It is also not related to 1 or more 0 values in the log.

Hence the feature request, so you can reset the history when this happens

[waltervl]

And I am going to tell you for the third time this is not possible.....

[akamming]

And I am going to tell you for the third time this is not possible.....

Why is it not possible. It should be a button on the Gui. If pressed, it executies the 2 delete statements which I now do manually

[waltervl]

Because it is the same as deleting the device and create a new one for which you say it is not working for you.....

[akamming]

Because it is the same as deleting the device and create a new one for which you say it is not working for you.....

But it’s not the same.

If you delete and re add the device it starts with the incorrect value. Furthemore you have to update your script(s) again

If you delete the history, no changing of scripts and the device “starts” with a correct value

[waltervl]

After rethinking it it seems you are not deleting the device entry from the devicestatus table as that containes the last updated value.
So you are not deleting the "entire history".

So lets see if a developer has the same issue as you and has some time to make an API that is not used in the User interface.

I think it will be faster for you to create some SQL script combined with python or something else that does the deleting of the "history" for a certain device IDX. See for example https://stackoverflow.com/questions/605 ... tead-of-id