history data size limited?

[Milifax]

Hi,

first of all, great work. I'm diving into Dzvents and am getting the hang of it. Still a beginner, so be easy on me
A strange thing occures when I use Historical variables API. Let me explain.

For a barometer, which reports to Domoticz, I want to calculate a forecast. The barometer doens't give these values and Domoticz doesn't calculate them by itself (as far as I know of, correct me if I'm wrong please).
To calculate I need the data of the last 180 minutes, getting new data every minute.
Then I want to calculate the average of 5 readings at 180 minutes and 5 readings at the moment. These values will be used to forecast. The historical variable would be perfect for it, because it shifts in new and shifts out old data.

However, it looks like the array is limited to 100 records. Once the script stored the 100 values, it just shifts out the old at (100) and enters new at (1).
Is this correct? Is the maxItems limited to 100? And is there a reason for it? Should I solve it differenlty?

Code: Select all

data = {
    
        BaroPressure = { 
        history = true, 
        maxItems = 185
        }
                          
    },

Code: Select all

 2017-07-18 21:38:00.449 dzVents: Info: ------ Start internal script: Combine_Barometer:, trigger: every 1 minutes
2017-07-18 21:38:00.487 dzVents: Info: Average BaroPressure: 1016.07 hPa
2017-07-18 21:38:00.487 dzVents: Info: Average BaroPressure: 101.607 kPa
2017-07-18 21:38:00.487 dzVents: Info: Average last 5 minutes: 101.6 kPa
2017-07-18 21:38:00.487 dzVents: Info: BaroPressure 1: 1016
2017-07-18 21:38:00.487 dzVents: Info: BaroPressure 2: 1016
2017-07-18 21:38:00.487 dzVents: Info: BaroPressure 3: 1016
2017-07-18 21:38:00.487 dzVents: Info: BaroPressure 4: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 5: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 6: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 7: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 8: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 9: 1016
2017-07-18 21:38:00.488 dzVents: Info: BaroPressure 10: 1016
2017-07-18 21:38:00.488 dzVents: Info: Size of history: 100
2017-07-18 21:38:00.488 dzVents: Info: Forecast: 0
2017-07-18 21:38:00.494 dzVents: Info: ------ Finished Combine_Barometer 

[dannybloe]

(I turned this into a new topic).

Ok, first of all, there should be a forecast to the barometer device but I'm not sure how Domoticz gets to the forecast.

Then your question. The point of limiting the history size is that the mechanism is not intended to store a large amount of data. The thing is is that all the data is stored in a file and every time the script runs (so likely every time you get a new baro reading) dzVents will load the file, update it and save it again. If there is too much data in there then you will definitely run into performance problems and perhaps even other problems. I think that 100 items is already way too many. Usually you can be smarter like doing pre-calculations before you store data in the history-store.

So, in your case I don't think that's a problem. I don't think you need baro data every minute. Wouldn't it be just fine if you store every 5 minutes or maybe even every 15 minutes and use that to calculate the difference over 180 minutes? Of you can do average calculations before you store it and only store the average.

So, reconsider

But it is good to hear someone is finding use of this. Let me know how you deal with this. (You can of course increase the max in the code but again, I advise against it).

Good luck!

Danny

[Milifax]

Thanks for the clear reply.
I will let my thoughts go over another solution. But I think it would be something like an array. The data should be shifted for 180 minutes before it can be dropped.

Will keep you posted if I think of another solution.

[dannybloe]

Yes but you dont need 180 data points for that as each datapoint will have a timestamp so you easily get the sample from 180 mins ago or closest to that moment. dzVents has methods for getting this data out of the history.

[Milifax]

Could you explain that a little bit more?
Can you retreive the data even if it wasn't stored in this historical 'array'?
Does Dzvents retreive the data out of the DB than?
I was under the impression the data was stored including a timestamp, but not retreivable by giving the timestamp.

If so, what is the way to get the data with a typical timestamp?

[dannybloe]

No, guess we misunderstood each other. You can use historical data that spans 180 minutes. All I was saying is that you don't need 180 data points in that array but you could probably do with a lot less data points to achieve the same result.If you take samples ever 10 minutes you have 18 data points which is a lot less.

[Milifax]

Ah ok, I understand.

With 18 points you can hardly filter out any false readings.
In the calcuation I use, an average is calculated over 5 samples every time the script executes. The 5 samples rotate with histroy data.
However I reduced the samples to half of 180, by measuring every 2 minutes. This reduces the samples a lot. Any more reduction is too much loss of data I think.

The thing is is that all the data is stored in a file and every time the script runs (so likely every time you get a new baro reading) dzVents will load the file, update it and save it again.

Did I understand it correctly that a physical file is used to store and read the data? If so, I will reconsider as it will not be very good for the SD-Card Domoticz is running on I guess

[dannybloe]

Then create two hisotries. A short-term for avg calc and data smoothing and a long-term one for the results.

[Milifax]

Did I understand it correctly that a physical file is used to store and read the data? If so, I will reconsider as it will not be very good for the SD-Card Domoticz is running on I guess

Ah, I see it in the wiki now. Just one paragraph further It indeed is stored in a file.

[Milifax]

Then create two hisotries. A short-term for avg calc and data smoothing and a long-term one for the results.

But still it doesn't shift the data, so I will loose the data inbetween.
Then again, maybe I'm missing what you mean but this doesn't sound like an accurate solution.

To be complete, this is what I'm trying to accomplish:

http://www.nxp.com/docs/en/application-note/AN3914.pdf

See Advanced Version, page 8 and further.

[dannybloe]

From what I understand so quickly (as I didn't look at the algorithm that much) is that it calculates an average every half hour using the past 5-minute averages. That looks exactly what I was suggesting. You create a var that collects samples every minute with a maximum of 5 samples. Then at every half hour you use that var to get an average and that's what you push into the other history var.