[SOLVED] how to deal with this kind of (solar-)data?

[groovy]

Hi,

I found a website providing some kind of solar forecast which I want to use in domoticz with a (simple) dzVents script.
The API is free to use for basic information (hourly 2 days forecast for watts, watt hours and watt hours/day ) and restricted to 12 calls per hour.
The problem I'm facing is, that the data for that 2 days comes as one single json string and I have no idea how to deal with this.
I assume I have to store the data in a temporary file and process that file every hour to feed a virtual sensor.
I would really appreciate if someone could push me in the right direction on how to do this. Or maybe there is an even simpler way to deal with this.
Any ideas?

The description of API calls can be found here: http://doc.forecast.solar/doku.php?id=api:estimate
The relevant parameters for the example below are:

51.3 - lat - latitude of location, -90 (south) … 90 (north)
12.4 - lon - longitude of location, -180 (west) … 180 (east)
45 - dec - plane declination, 0 (horizontal) … 90 (vertical)
0 - az - plane azimuth, -180 … 180 (-180 = north, -90 = east, 0 = south, 90 = west, 180 = north)
0.03 (for my 30W panel) kwp - installed modules power in kilo watt

Here is an example of the API call from bash:

Code: Select all

curl -H "Accept: text/json" https://api.forecast.solar/estimate/51.3/12.4/45/0/0.03

And this is what the answer looks like:

Code: Select all

{
    "result": {
        "watts": {
            "2019-04-08 06:22:00": 0,
            "2019-04-08 06:56:00": 0.18,
            "2019-04-08 07:30:00": 1.2,
            "2019-04-08 08:00:00": 2.19,
            "2019-04-08 09:00:00": 4.98,
            "2019-04-08 10:00:00": 8.37,
            "2019-04-08 11:00:00": 11.88,
            "2019-04-08 12:00:00": 14.64,
            "2019-04-08 13:00:00": 21.75,
            "2019-04-08 14:00:00": 21.48,
            "2019-04-08 15:00:00": 19.5,
            "2019-04-08 16:00:00": 14.52,
            "2019-04-08 17:00:00": 8.73,
            "2019-04-08 18:00:00": 3.63,
            "2019-04-08 19:01:00": 0.51,
            "2019-04-08 20:02:00": 0,
            "2019-04-09 06:20:00": 0,
            "2019-04-09 06:55:00": 0.24,
            "2019-04-09 07:30:00": 1.74,
            "2019-04-09 08:00:00": 3.45,
            "2019-04-09 09:00:00": 8.31,
            "2019-04-09 10:00:00": 13.89,
            "2019-04-09 11:00:00": 18.96,
            "2019-04-09 12:00:00": 22.41,
            "2019-04-09 13:00:00": 23.76,
            "2019-04-09 14:00:00": 22.68,
            "2019-04-09 15:00:00": 19.17,
            "2019-04-09 16:00:00": 14.22,
            "2019-04-09 17:00:00": 8.49,
            "2019-04-09 18:00:00": 3.54,
            "2019-04-09 19:02:00": 0.51,
            "2019-04-09 20:03:00": 0
        },
        "watt_hours": {
            "2019-04-08 06:22:00": 0,
            "2019-04-08 06:56:00": 0.09,
            "2019-04-08 07:30:00": 0.78,
            "2019-04-08 08:00:00": 1.89,
            "2019-04-08 09:00:00": 6.87,
            "2019-04-08 10:00:00": 15.24,
            "2019-04-08 11:00:00": 27.12,
            "2019-04-08 12:00:00": 41.76,
            "2019-04-08 13:00:00": 63.51,
            "2019-04-08 14:00:00": 84.99,
            "2019-04-08 15:00:00": 104.49,
            "2019-04-08 16:00:00": 119.01,
            "2019-04-08 17:00:00": 127.74,
            "2019-04-08 18:00:00": 131.37,
            "2019-04-08 19:01:00": 131.88,
            "2019-04-08 20:02:00": 131.88,
            "2019-04-09 06:20:00": 0,
            "2019-04-09 06:55:00": 0.15,
            "2019-04-09 07:30:00": 1.17,
            "2019-04-09 08:00:00": 2.88,
            "2019-04-09 09:00:00": 11.19,
            "2019-04-09 10:00:00": 25.08,
            "2019-04-09 11:00:00": 44.04,
            "2019-04-09 12:00:00": 66.45,
            "2019-04-09 13:00:00": 90.21,
            "2019-04-09 14:00:00": 112.89,
            "2019-04-09 15:00:00": 132.06,
            "2019-04-09 16:00:00": 146.28,
            "2019-04-09 17:00:00": 154.77,
            "2019-04-09 18:00:00": 158.31,
            "2019-04-09 19:02:00": 158.85,
            "2019-04-09 20:03:00": 158.85
        },
        "watt_hours_day": {
            "2019-04-08": 131.88,
            "2019-04-09": 158.85
        }
    },
    "message": {
        "code": 0,
        "type": "success",
        "text": "",
        "info": {
            "place": "04328 Sellerhausen-St\u00fcnz, Kreisfreie Stadt Leipzig, Sachsen, DE",
            "distance": 6.36,
            "pid": "4th683nz"
        },
        "ratelimit": {
            "limit": 12,
            "remaining": 2
        }
    }
}

[waaren]

I found a website providing some kind of solar forecast which I want to use in domoticz with a (simple) dzVents script.
The API is free to use for basic information (hourly 2 days forecast for watts, watt hours and watt hours/day ) and restricted to 12 calls per hour.
The problem I'm facing is, that the data for that 2 days comes as one single json string and I have no idea how to deal with this.
I assume I have to store the data in a temporary file and process that file every hour to feed a virtual sensor.
I would really appreciate if someone could push me in the right direction on how to do this. Or maybe there is an even simpler way to deal with this.
Any ideas?

The description of API calls can be found here: http://doc.forecast.solar/doku.php?id=api:estimate

Happy to help but what do you want to do with the data ?
Do you want to sum it and store it in counter or text device or somewhere else ?

[groovy]

Thanks for having a look.
My idea would be a dummy device of type "P1 Smart Meter, Energy", so I can have the actual W every hour and the daily summary of the solar energy production.

[waaren]

Thanks for having a look.
My idea would be a dummy device of type "P1 Smart Meter, Energy", so I can have the actual W every hour and the daily summary of the solar energy production.

OK will come up with something. Probably tomorrow morning.

[groovy]

Thanks a lot in advance.

[waaren]

Thanks for having a look.
My idea would be a dummy device of type "P1 Smart Meter, Energy", so I can have the actual W every hour and the daily summary of the solar energy production.

Could be something like this.
The script updates a P1 meter and a usage electric sensor with the estimated produced energy in watt from the time nearest to current time. Just see what works for you and if not needed just comment one of the two.
A daily summary in the P1 meter is something domoticz calculate automatically.

Please feel free to comment or ask for clarification when needed.

Code: Select all

--[[ 
        get solar data from forecast.solar and store in a virtual P1 sensor.
--]] 

local webResponse = "solarData"

return  {   on =    {  
                        timer           = { 'every 10 minutes' },
                       httpResponses   = { webResponse },
                    },

        logging =   {
                        level       = domoticz.LOG_DEBUG,  -- switch to LOG_ERROR when all OK
                        marker      = webResponse
                    },
       
    execute = function(dz, item)

        local solarDelivery = dz.devices("solarDelivery")   -- Create this virtual sensor as "usage Electric"
        local solarP1 = dz.devices("solarP1")             -- Create this virtual sensor as "P1 Smart Meter, Electric"
        local latitude =  dz.settings.location.latitude   -- these are the lat/lon set in domoticz (dzVents >= 2.4.14 required)
        local longitude = dz.settings.location.longitude  -- or set to the value you want
        local planeDeclination = 45                       -- 0 (horizontal) … 90 (vertical)
        local planeAzimuth = 0                            --   -180 … 180 (-180 = north, -90 = east, 0 = south, 90 = west, 180 = north)
        local installedModulesPower =  0.03 -- in kilo watt

        local function getSolarData()
            local url = "https://api.forecast.solar/estimate" .. 
                        "/" .. latitude .. 
                        "/" .. longitude ..
                        "/" .. planeDeclination ..
                        "/" .. planeAzimuth ..
                        "/" .. installedModulesPower 
            dz.openURL  (   
                            {
                                url = url,
                                headers = { ['Accept'] = 'application/json' },
                                method = 'GET',
                                callback = webResponse
                            }        
                        )
        end
        
        local function getWattFromNearestTime(t)
            local Time = require('Time')
            local loopValue, delta = 0, 99999
            local nearest
            
            for key, value in pairs(t) do
                loopValue = dz.time.compare(Time(key)).secs  -- get absolute diff between now and date/time in response JSON
                if loopValue < delta then 
                    nearest = key
                    delta = loopValue
                end
            end
            
            return nearest -- return date/time string nearest to now
        end
        
        local function updateSensors(produced)
            solarP1.updateP1(solarP1.usage1,             -- keep current data
                             solarP1.usage2,            -- keep current data
                             produced,      -- Update to new value
                             solarP1.return2,           -- keep current data
                             solarP1.usage,             -- keep current data
                             solarP1.usagDelivered)     -- keep current data
             solarDelivery.updateEnergy(produced)
        end
        
        if not item.isHTTPResponse then
            getSolarData()
        elseif item.ok then
            local wattTable = item.json.result.watts 
            local nearest = getWattFromNearestTime(wattTable)
            dz.log("Produced watt at: " .. nearest .. " ==>> " .. wattTable[nearest] ,dz.LOG_DEBUG) 
            updateSensors(wattTable[nearest])
        else
            dz.log("Problem with the JSON response",dz.LOG_ERROR)
        end
    end
}

[Toulon7559]

@groovy
Interesting website you found.

For owners of solar-systems just the thing missing in the meteo-forecasts.
The chart-function allows visual looking forward on available energy:
useful for domotics, because you could plan the application of washing machines etc.

The info in the CSV-, JSON- or XML-file could already be crude basis for automated scheduling:
'just' find/make software to determine thresholds & durations for decisions & switching.
Although some practical margins must be taken:
forecasts have a limited accuracy, and any unexpected cloud may completely spoil the scheduling ..........

Unfortunately their chart-function is part of the Professional account:
'moneysavers' have to build an own graphic solution, e.g. comparable with this type of layout
Somebody with a simple, 'ready' script-solution for that, applying the info in the CSV-, JSON- or XML-file obtained with the API-call?

[groovy]

@waaren

Thanks a lot, this almost worked out of the box. Great way also to learn something, especially how you managed to get the nearest time. Great work!
I just decided to finally use a sensor of type: "Electric (Instant+Counter)" and modified the script to update the sensor via the JSON API, because also I want to send the data to an InfluxDB. The sensor calculates the daily energy himself and I even get a nice Icon

Bildschirmfoto von 2019-04-09 10-33-09.png (12.96 KiB) Viewed 2075 times

So, I'm very happy now

Unfortunately their chart-function is part of the Professional account:
'moneysavers' have to build an own graphic solution, e.g. comparable with this type of layout
Somebody with a simple, 'ready' script-solution for that, applying the info in the CSV-, JSON- or XML-file obtained with the API-call?

no "ready script", but have a look at Grafana for nice graphs. There is even a JSON-Plugin available, so you could easily feed the data directly into Grafana. See here:

https://grafana.com/plugins/simpod-json-datasource

[Toulon7559]

On my ToDo-list is (for a long time) further study of Domoticz with application of InfluxdB and Grafana.
This API-call may be a nice/interesting incentive to get it started ......

[Toulon7559]

Periodically extracting the JSON-file or XML-file and putting it at my webserver is easy enough:
now next step = how to easily make a nice graph?

The introduction of Canvasjs tells that it can deal with JSON, CSV and XML, just as groovy's website delivers.
Homework to read&'translate', and then experiments to make the practical link between the file and the plugin.
Practical example(s) would be handy to shorten the learning curve .....
Somebody a hint?

Seeming slightly easier & simpler than Grafana, https://canvasjs.com/javascript-charts/ ... jax-chart/ could be an alternative entry for making the related graphs.

[PieterS]

Very interesting post for me; to plan my green footage: when to start heavy equipement as washingmachine or dishwasher of irrigation.

@Waaren: scripting dzvents is not that easy for me. So for me this case is not solved.. Can you help me out with creating a working script for me?

Up till now I made one virtual sensor named solarDelivery, Type Delivery, Energy read From device.
Guess that will do because i have ferrarismeter and no sensor for that.
Maybe I need another because i suggest to fill one for forecast today and second for forecast tomorrow?

It think it is clear to me how to fill the url https://api.forecast.solar/estimate/51/5/45/10/3.5 When I fill that in the adresbar of my browser i get result. I know how to be more precisely.

From line 60 in the script it starts with editing the value of a P1-sensor.... I am not familiar with that type... Sorry. I think I should use another type.

Would you help me with an example? I would be glad when I can represent two values on my dashboard in Domoticz: estimate watt_hours_day like
2019-04-12 16544.5
2019-04-13 18819.5

Maybe I need two virtual sensors? Type Custom? And they make a graph with the results in watts and watt_hours?

I hope you can help me out. Many thanks in advance!

Pieter

[Toulon7559]

@waaren/ @ PieterS

Graphs for forecast of Watts and of Watthours over today and tomorrow would be nice:
in fact my aim, and reason to study the indicated charting-software.
But (unless I am mistaken) the read-out as proposed by waaren only provides the actual forecast for this moment, and (due to the way that Domoticz operates) the resulting graph is a historic trail of that forecast.
If we want a forecast-graph based on the forecast-info , it requires to take the complete json-file coming from Estimate/Watts and/or Estimate/Watthours and translate into a chart.
Good practical result could be a chart for today and for tomorrow combining hourly Forecasted Power and Energy!
And the 'Ultimate' layout might be that (in the same graph) as graphics line 3 and 4 you get from your PV-system the actual/historic PV-Power and PV-Energy.
'Smells' as combination of InfluxDB + Grafana + ?

For simplicity seems best to start with periodic making of a graph for forecasts only as indicated above.
Somebody having ready ideas?

[PieterS]

Let's start my first goal to fill two virtual sensors in my Domoticz dashboard with values of forecast of today and tomorrow.
I study and use the luftdatenscript as example to extract those values.

Code: Select all

local FQDN = 'luftdaten.xxxx.xx'

return {
        active = true,
        on = {
                timer = { 'every minute' },
                httpResponses = { 'luftdatenRetrieved' } -- matches callback string below
        },
        execute = function(domoticz, item)

                if (item.isTimer) then
                        domoticz.openURL({
                                url = 'http://192.168.x.xx/data.json',
                                method = 'GET',
                                callback = 'luftdatenRetrieved'
                        })

                elseif (item.isHTTPResponse) then
                        if (item.ok and item.isJSON) then -- statusCode == 2xx
                                if tonumber(item.json.age) < 60 then
-- 1: SDS_P1 PM10, 2: SDS_P2 PM2.5, 3: DHT22 temp, 4: DHT22 hum
                                        domoticz.devices('Luftdaten PM10').updateCustomSensor(item.json.sensordatavalues[1].value)
                                        domoticz.devices('Luftdaten PM2.5').updateCustomSensor(item.json.sensordatavalues[2].value)
                                        domoticz.devices('Buitentemperatuur').updateTempHum(item.json.sensordatavalues[3].value,item.json.sensordatavalues[4].value,0)
                                end
                        else
                                -- oops
                                domoticz.log('Error fetching Luftdaten data', domoticz.LOG_ERROR)
                                domoticz.log(item.data, domoticz.LOG_ERROR)
                        end
                end
        end
}

I would use the command: updateCustomSensor(item.json.sensordatavalues[x].value)

But the first problem is: the layout of those files are different in my browser... Why? This is result in JSON I get from my luft-sensor.

It's told that If no Accept header is defined, JSON will delivered by default.. How to get a result from Forecast.solar in the same layout?

When I give curl -H "Accept: application/json" https://api.forecast.solar/estimate/51/5/45/10/3.5 in putty on my Synology I get a long output and no numbers for sensordata.
any help appreciated..

[waaren]

Let's start my first goal to fill two virtual sensors in my Domoticz dashboard with values of forecast of today and tomorrow.
any help appreciated..

What do you see if you add line

Code: Select all

domoticz.log(item.data,domoticz.LOG_FORCE)

just before

Code: Select all

elseif (item.isHTTPResponse) then

?

[PieterS]

@waaren: Thanks for reply!
Wondering why I had to add that line in script Luftdaten. That script works fine and delivers adequate info. I would use that as example for another script. Sorry for misunderstanding.
Anyway, I did what you asked and edit script Luftdaten. At first it did not make sense to the logfile in Domoticz. SoI changed in Settings the loglevel of dzVents to debug everything.

Code: Select all

 2019-04-14 11:41:00.325 Status: dzVents: Debug: Dumping domoticz data to /usr/local/domoticz/var/scripts/dzVents/domoticzData.lua
2019-04-14 11:41:00.341 Status: dzVents: Debug: Event triggers:
2019-04-14 11:41:00.341 Status: dzVents: Debug: - Timer
2019-04-14 11:41:00.362 Status: dzVents: Info: ------ Start internal script: Luftdaten:, trigger: every minute
2019-04-14 11:41:00.363 Status: dzVents: Debug: OpenURL: url = http://192.168.1.41/data.json
2019-04-14 11:41:00.363 Status: dzVents: Debug: OpenURL: method = GET
2019-04-14 11:41:00.363 Status: dzVents: Debug: OpenURL: post data = nil
2019-04-14 11:41:00.363 Status: dzVents: Debug: OpenURL: headers = nil
2019-04-14 11:41:00.363 Status: dzVents: Debug: OpenURL: callback = luftdatenRetrieved
2019-04-14 11:41:00.363 Status: dzVents: !Info: nil
2019-04-14 11:41:00.363 Status: dzVents: Info: ------ Finished Luftdaten
2019-04-14 11:41:00.372 Status: dzVents: Debug: Commands sent to Domoticz:
2019-04-14 11:41:00.372 Status: dzVents: Debug: - OpenURL = {["_trigger"]="luftdatenRetrieved", ["method"]="GET", ["URL"]="http://192.168.1.41/data.json"}
2019-04-14 11:41:00.372 Status: dzVents: Debug: - UpdateDevice = {["sValue"]="30 dagen, 19 uur en 16 minuten", ["idx"]=785, ["_trigger"]=true, ["nValue"]=0}
2019-04-14 11:41:00.372 Status: dzVents: Debug: =====================================================
2019-04-14 11:41:00.577 Status: dzVents: Debug: Dumping domoticz data to /usr/local/domoticz/var/scripts/dzVents/domoticzData.lua
2019-04-14 11:41:00.593 Status: dzVents: Debug: dzVents version: 2.4.6
2019-04-14 11:41:00.594 Status: dzVents: Debug: Event triggers:
2019-04-14 11:41:00.594 Status: dzVents: Debug: - HTTPResponse: luftdatenRetrieved
2019-04-14 11:41:00.676 Status: dzVents: Info: Handling httpResponse-events for: "luftdatenRetrieved
2019-04-14 11:41:00.676 Status: dzVents: Info: ------ Start internal script: Luftdaten: HTTPResponse: "luftdatenRetrieved"
2019-04-14 11:41:00.687 Status: dzVents: Debug: Processing device-adapter for Luftdaten PM10: Custom sensor device adapter
2019-04-14 11:41:00.687 Status: dzVents: Debug: Processing device-adapter for Luftdaten PM2.5: Custom sensor device adapter
2019-04-14 11:41:00.688 Status: dzVents: Debug: Processing device-adapter for Buitentemperatuur: Temperature+humidity device adapter
2019-04-14 11:41:00.688 Status: dzVents: Info: ------ Finished Luftdaten
2019-04-14 11:41:00.688 Status: dzVents: Debug: Commands sent to Domoticz:
2019-04-14 11:41:00.688 Status: dzVents: Debug: - UpdateDevice = {["sValue"]="32.10", ["idx"]=742, ["_trigger"]=true, ["nValue"]=0}
2019-04-14 11:41:00.688 Status: dzVents: Debug: - UpdateDevice = {["sValue"]="27.50", ["idx"]=743, ["_trigger"]=true, ["nValue"]=0}
2019-04-14 11:41:00.688 Status: dzVents: Debug: - UpdateDevice = {["sValue"]="7.90;99.90;0", ["idx"]=748, ["_trigger"]=true, ["nValue"]=0}
2019-04-14 11:41:00.688 Status: dzVents: Debug: =====================================================
2019-04-14 11:41:00.706 Status: dzVents: Debug: Dumping domoticz data to /usr/local/domoticz/var/scripts/dzVents/domoticzData.lua
2019-04-14 11:41:00.731 Status: dzVents: Debug: Processing device-adapter for Domoticz uptime: Text device
2019-04-14 11:41:00.731 Status: dzVents: Debug: dzVents version: 2.4.6
2019-04-14 11:41:00.731 Status: dzVents: Debug: Event triggers:
2019-04-14 11:41:00.731 Status: dzVents: Debug: - Device: Domoticz uptime
2019-04-14 11:41:00.834 Status: dzVents: Debug: Dumping domoticz data to /usr/local/domoticz/var/scripts/dzVents/domoticzData.lua
2019-04-14 11:41:00.860 Status: dzVents: Debug: Processing device-adapter for Luftdaten PM10: Custom sensor device adapter
2019-04-14 11:41:00.860 Status: dzVents: Debug: Processing device-adapter for Luftdaten PM2.5: Custom sensor device adapter
2019-04-14 11:41:00.861 Status: dzVents: Debug: Processing device-adapter for Buitentemperatuur: Temperature+humidity device adapter
2019-04-14 11:41:00.861 Status: dzVents: Debug: dzVents version: 2.4.6
2019-04-14 11:41:00.861 Status: dzVents: Debug: Event triggers:
2019-04-14 11:41:00.861 Status: dzVents: Debug: - Device: Luftdaten PM10
2019-04-14 11:41:00.861 Status: dzVents: Debug: - Device: Luftdaten PM2.5
2019-04-14 11:41:00.861 Status: dzVents: Debug: - Device: Buitentemperatuur

Can you explain why I had to add that line in this Luftdaten script? I would expect that we would make adjustments to the script that you wrote for Groovy.

Wondering for next steps.

[waaren]

@waaren: Thanks for reply!
Wondering why I had to add that line in script Luftdaten. That script works fine and delivers adequate info. I would use that as example for another script. Sorry for misunderstanding.

Probably my mistake (mixing up the topics) Can you open a new topic and elaborate on what you try to achieve and any issue that you face in the process of creating a working script for this ?

[Toulon7559]

@waaren

'Learning by reading' trying to understand the setup of the script in this thread.
Therefore this question is just for understanding.

Your script applies 2 virtual sensors:
1* "usage" plus 1* "P1", with "P1" according to https://www.domoticz.com/wiki/Domoticz_ ... mart_meter
I see the same value coming out of both.

Then why not "P1" alone?
"P1" has plenty of application-fields both for energy and for power, but (unless mistaken) I see only Energy-this-hour coming out.
Or "usage" alone, also dedicated to Energy-this-hour?

[waaren]

@waaren

'Learning by reading' trying to understand the setup of the script in this thread.
Therefore this question is just for understanding.

Your script applies 2 virtual sensors:
1* "usage" plus 1* "P1", with "P1" according to https://www.domoticz.com/wiki/Domoticz_ ... mart_meter
I see the same value coming out of both.

Then why not "P1" alone?
"P1" has plenty of application-fields both for energy and for power, but (unless mistaken) I see only Energy-this-hour coming out.
Or "usage" alone, also dedicated to Energy-this-hour?

@Toulon7559,

No other reason then to give the option to choose one of them. That's why I suggested to comment the one that was not needed.

[Toulon7559]

Clear!

Just one more question for understanding:
if the script is looking for "Watts", why the data-output in P1-field RETURN1, and not in P1-field PROD?

[waaren]

Clear!

Just one more question for understanding:
if the script is looking for "Watts", why the data-output in P1-field RETURN1, and not in P1-field PROD?

My thinking was that because the returned estimate in Watts is an average estimate for one hour, It could be used as WattHour. Here we only need to look at this estimate and not what the the result of produced - consumed is. (In the real P1 meter either produced = 0 when consumed > 0 and vice versa) Domoticz internal processing should take care of the calculations while transferring the values from devicestatus table to multimeter table and from multimeter table to multimeter_calendar table.
Could well be that I did not take everything into account. My primary goal here was to get the data from forecast.solar into a virtual domoticz device and I am not using a forecast myself yet. That might change in the future when my new electric powered car arrives. ( ETA 2028 )