Validity time of svalue

[Toulon7559]

The aspect of time-validity is already/elsewhere introduced under this message, but I think it deserves it's own thread.
Reason: time-validity or time-stamping is an important aspect if you check continuity of a data stream, if you interpolate or extrapolate data, and if you check on responses.

In the section with svalues for each input a data set is registered;
look at Setup/More Options/Events/ Current States
If everything is operational, the data set is periodically refreshed, but if a malfunction occurs, then the refresh stops:
the time-value in the most righthand column becomes 'frozen'. If you apply the data later in time it produces a 'flatline'.
A time-out is upcoming as soon as that 'frozen'time and the actual clocktime deviate by a defined amount of time.

To perform a time-out check you need to read that time-value for the subject data and compare against the actual clock-time.
How to extract the time-value by means of a lua-script? Or in a Python-script? Or by PHP-script?

[Toulon7559]

In Domoticz' Wiki I revisited the section on API/JSON URLs, and got an idea to get hold of time validity and a mechanism for timeout-check etc..

Consider the following example from my configuration.
RID = 89 is for a data set coming from an ESP8266, providing a light-value.
When making the call http://<controller-ip-address>:8080/json.htm?type=devices&rid=89
the resulting string is

Code: Select all

{
   "ActTime" : 1501360852,
   "ServerTime" : "2017-07-29 22:40:52",
   "Sunrise" : "05:50",
   "Sunset" : "21:28",
   "result" : [
      {
         "AddjMulti" : 1.0,
         "AddjMulti2" : 1.0,
         "AddjValue" : 0.0,
         "AddjValue2" : 0.0,
         "BatteryLevel" : 255,
         "CustomImage" : 0,
         "Data" : "73 Lux",
         "Description" : "",
         "Favorite" : 1,
         "HardwareID" : 4,
         "HardwareName" : "Dummies",
         "HardwareType" : "Dummy (Does nothing, use for virtual switches only)",
         "HardwareTypeVal" : 15,
         "HaveTimeout" : true,
         "ID" : "82089",
         "LastUpdate" : "2017-07-29 21:16:22",
         "Name" : "ESP8266C_Licht1",
         "Notifications" : "false",
         "PlanID" : "0",
         "PlanIDs" : [ 0 ],
         "Protected" : false,
         "ShowNotifications" : true,
         "SignalLevel" : "-",
         "SubType" : "Lux",
         "Timers" : "false",
         "Type" : "Lux",
         "TypeImg" : "lux",
         "Unit" : 1,
         "Used" : 1,
         "XOffset" : "0",
         "YOffset" : "0",
         "idx" : "89"
      }
   ],
   "status" : "OK",
   "title" : "Devices"
}

In my opinion, for this subject the interesting 'elements' are ActTime, ServerTime, Data, LastUpdate and perhaps HaveTimeOut and Type:
- ActTime is the actual time expresses as condensed UTC-label
- Servertime is the actual time related to the response on the call for the contents of the RID
- Data is the value, and Type contains the textstring identifying the kind of parameter [for baro, hum and temp sometimes even simpler just a number, but that is RID/device-dependent]
- LastUpdate tells at which time the contents were valid, and in this example considerably deviating from the ServerTime
- HaveTimeOut seems to be an indication (already present in Domoticz!) that a defined time-out has occurred

The check on a general time-out on the uplad from the device could be:
1) Check ActTime against actual SystemTime
2) If difference > threshold X, then time-out => 'raise alarm'

If those 'elements' are applicable, the solution for a check on time-out of the device seems:
1) check on presence of "HaveTimeout" : true, or "HaveTimeout" : false,
2) If false => OK = no time-out
3) If true => time-out value = ServerTime - LastUpdate
3a) if time-out value > Y, then 'raise alarm'

Less alarming application is check of staleness within the response (regardless of "HaveTimeout"):
- Staleness = ServerTime - LastUpdate

Or - if you want to extrapolate (or check change of values) - apply 2 (or more) successive datasets:
1) make first json-call and from response extract the value from Data and the time from LastUpdate, and store as uservariableset1
2) make second json-call and from response extract the value from Data and the time from LastUpdate, and store as uservariableset2
3) subtract the Data and the LastUpdate, and you can calculate the tendency of the data

Only problem now is how to extract those values & text from the JSON-response in a format which can be easily processed:
help appreciated for working lua-scriptlines, python-scriptlines or by PHP-scrip for any of above applications
Or is a simpler, more direct access possible to this information as stored under the svalues?

[Toulon7559]

Some background for the previous message.

[Toulon7559]

Except for the first message in this thread, the ideas expressed in the messages are just work-arounds if no direct access to time-validity is possible/available.

Anybody an idea or 'basic' working example how to extract the latest valid time-value from an svalue by means of a dzVents/lua-script?
UTC-format for time would be OK, and would simplest for subsequent calculations to determine 'age' of the data in that svalue.
First application:
check that the data in the svalue is meaningful, considering age.

Variant available for Python-script or for PHP-script?

[Toulon7559]

Other phrasing of the question, directly aimed at realisation

The example lua-scriptline below extracts the sensor-data from an svalue for rain

Code: Select all

sRainRate, sRainCount = otherdevices_svalues[RainMeter]:match("([^;]+);([^;]+)")

What is the lua-scriptline to get the time-value related to this svalue?

[waaren]

Other phrasing of the question, directly aimed at realisation

The example lua-scriptline below extracts the sensor-data from an svalue for rain

Code: Select all

sRainRate, sRainCount = otherdevices_svalues[RainMeter]:match("([^;]+);([^;]+)")

What is the lua-scriptline to get the time-value related to this svalue?

There is no time-value directly related to an sValue. In the database you will find the LastUpdate field reflecting the time/date the nValue or sValue was updated (not necessarily changed) via the standard domoticz mechanisms. It will not change if you update these values directly in the database.
In memory you can find the date/time a device was last seen using <domoticz IP>:<domoticz port>/json.htm?type=devices&rid=<device id> The last seen value is in the LastUpdate field. Fact that the same fieldname is used for different info is a bit confusing for me ) the last seen value does reflect a change to the nValue/ sValue but also when the hardware send a keep-alive signal as some hardware do.
The database value is what you get when you use domoticz.devices(<device id>).lastUpdate.rawDate
The value from memory (last seen) must be retrieved from the JSON/ API return.

[Toulon7559]

@Waaren

Thanks for the explanation.
Apparently have been looking for something not available (or not at expected place).

Just wondering, whether/how a 'simple' call to domoticz.devices(<device id>).lastUpdate.rawDate could produce a numeric time-value:
probably requires a certain structure, because some first experiments only gave error reports that 'domoticz' not recognized.

[waaren]

Just wondering, whether/how a 'simple' call to domoticz.devices(<device id>).lastUpdate.rawDate could produce a numeric time-value:

If you are looking for a numeric time Value have a look at this dzVents script. Would not know how to do this in classic domoticz Lua

Code: Select all

local scriptVar = 'lastSeen'

return {
    on = { timer =  {'every minute'}, httpResponses = { scriptVar }},
    logging = { level   = domoticz.LOG_INFO, marker  = scriptVar },
    
    execute = function(dz, item) 
        
        if not (item.isHTTPResponse) then
            dz.openURL({ 
                url = dz.settings['Domoticz url'] .. '/json.htm?type=devices&used=true',
                callback = scriptVar })
        else
            local Time = require('Time')
            for _, node in pairs(item.json.result) do
            
                local lastSeen = Time(node.LastUpdate).secondsAgo
                local lastUpdated = dz.devices(tonumber(node.idx)).lastUpdate.secondsAgo
                local lastUpdatedDate = dz.devices(tonumber(node.idx)).lastUpdate.dDate
                local delta = lastSeen - lastUpdated
                dz.log('id '..  node.idx .. '('  ..node.Name .. ') Delta ' .. delta .. ' Numeric lastUpdated ' .. lastUpdatedDate,dz.LOG_FORCE)
            
            end
        end
    end
}

[Toulon7559]

@ Waaren
a single lua-scriptline would be nice-to-have , but
at one hand that seems farfetched as a translation for your multi-line example dzvents-script, and
on the other hand a single lua-scriptline calling domoticz.devices(<device id>).lastUpdate.rawDate does not seem easy either.

For a 'classic' lua-solution and for a Python-variant, the simplest approach seems
a) JSON-calling plus dissect of the result-string to find the numeric value of lastUpdate
b) compare with actual time.

To avoid any hickups, for 'universal' setup best to apply UTC as time-format.

[Toulon7559]

With time (and another key word for search) comes a solution:
viewtopic.php?t=36189&sid=da1c78a489757 ... cbfb0a39ae
Another gem from waaren's hand .....