Forecast of the yield of solar panels  [Solved]

Easy to use, 100% Lua-based event scripting framework.

Moderator: leecollings

PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Forecast of the yield of solar panels

Post by PieterS »

As a continuation of solved topic https://www.domoticz.com/forum/viewtopi ... 59&t=27565 I start a new one to achieve another goal: I want to represent the estimation of my solar panels for today and tomorrow in the dashboard of Domoticz.

I thought:
fill a virtual sensor with estimated Watts for this hour and total Watt-hours_today
and a second sensor with estimated Watt-hours expected tomorrow.

It should be possible with the result of a call to http://doc.forecast.solar/. The request https://api.forecast.solar/estimate/51/5/45/10/3.5 sends a lot of information. :D

Problem is that the layout from http://forcast.solar is quite different than the for me known layout of JSON.. As an example: When I ask the results of my dustsensor http://192.168.1.41/data.json it gives:
Image

I would use the command: updateCustomSensor(item.json.sensordatavalues[x].value)

So that I can rebuild a working script that I now use for reading my dustsensor and which does a great job!

Code: Select all

local FQDN = 'luftdaten.xxxx.xx'
return {
        active = true,
        on = {
                timer = { 'every minute' },
                httpResponses = { 'luftdatenRetrieved' } -- matches callback string below
        },
        execute = function(domoticz, item)
                if (item.isTimer) then
                        domoticz.openURL({
                                url = 'http://192.168.1.41/data.json',
                                method = 'GET',
                                callback = 'luftdatenRetrieved'
                        })
                elseif (item.isHTTPResponse) then
                        if (item.ok and item.isJSON) then -- statusCode == 2xx
                                if tonumber(item.json.age) < 60 then
-- 1: SDS_P1 PM10, 2: SDS_P2 PM2.5, 3: DHT22 temp, 4: DHT22 hum
                                        domoticz.devices('Luftdaten PM10').updateCustomSensor(item.json.sensordatavalues[1].value)
                                        domoticz.devices('Luftdaten PM2.5').updateCustomSensor(item.json.sensordatavalues[2].value)
                                        domoticz.devices('Buitentemperatuur').updateTempHum(item.json.sensordatavalues[3].value,item.json.sensordatavalues[4].value,0)
                                end
                        else
                                -- oops
                                domoticz.log('Error fetching Luftdaten data', domoticz.LOG_ERROR)
                                domoticz.log(item.data, domoticz.LOG_ERROR)
                        end
                end
        end
}
But the first problem is: the layout of those files from my dustsensor (shown above) and that from forecast.solar (a kind of textfile) are different in my browser... Why?

When I give curl -H "Accept: application/json" https://api.forecast.solar/estimate/51/5/45/10/3.5 in putty on my Synology I get long output and no numbers for sensordata. Can you explain this to me? And give a hint for a solution for a script?

Use Domoticz V4.9700 on a Synology DS718+

Any help appreciated..
Synology with Domoticz build (V2024.7) in Docker
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Forecast of the yield of solar panels

Post by waaren »

PieterS wrote: Sunday 14 April 2019 15:26 As a continuation of solved topic https://www.domoticz.com/forum/viewtopi ... 59&t=27565 I start a new one to achieve another goal: I want to represent the estimation of my solar panels for today and tomorrow in the dashboard of Domoticz.

I thought:
fill a virtual sensor with estimated Watts for this hour and total Watt-hours_today
and a second sensor with estimated Watt-hours expected tomorrow.
Can you test attached ?
The script update 3 custom sensors (KWh) as I don't know how to combine two values (hour and day) in one custom sensor.
Please feel free to ask if anything is not clear or not working as expected.

Code: Select all

--[[ 
        get solar data from forecast.solar and store in a virtual P1 sensor.
--]] 

local webResponse = "solarData"

return  {   on =    {  
                       timer           = { 'every hour' },
                       httpResponses   = { webResponse },
                    },

        logging =   {
                        level       = domoticz.LOG_DEBUG,  -- switch to LOG_ERROR when all OK
                        marker      = webResponse
                    },
       
    execute = function(dz, item)
        
        local function logWrite(str,level)
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
        
        local solarEstimateCurrentHour = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateToday = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateTomorrow = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        
        -- local latitude =  dz.settings.location.latitude   -- these are the lat/lon set in domoticz (dzVents >= 2.4.14 required)
        local latitude =  51 -- set to your latitude 
        -- local longitude = dz.settings.location.longitude  -- or set to the value you want
        local longitude = 5  -- set to your longitude
        local planeDeclination = 45                       -- 0 (horizontal) … 90 (vertical)
        local planeAzimuth = 10                            --   -180 … 180 (-180 = north, -90 = east, 0 = south, 90 = west, 180 = north)
        local installedModulesPower =  3.5 -- in kilo watt

        local function getSolarData()
            local url = "https://api.forecast.solar/estimate" .. 
                        "/" .. latitude .. 
                        "/" .. longitude ..
                        "/" .. planeDeclination ..
                        "/" .. planeAzimuth ..
                        "/" .. installedModulesPower 
            dz.openURL  (   
                            {
                                url = url,
                                headers = { ['Accept'] = 'application/json' },
                                method = 'GET',
                                callback = webResponse
                            }        
                        )
        end
        
        local function getNearestHour(t)
            local Time = require('Time')
            local loopValue, delta = 0, 99999
            local nearest
            
            for key, value in pairs(t) do
                loopValue = dz.time.compare(Time(key)).secs  -- get absolute diff between now and date/time in response JSON
                if loopValue < delta then 
                    nearest = key
                    delta = loopValue
                end
            end
            
            return nearest -- return date/time string nearest to now
        end
        
        local function updateSensors(thisHour, today, tomorrow)
             solarEstimateCurrentHour.updateCustomSensor(thisHour)
             solarEstimateToday.updateCustomSensor(today)
             solarEstimateTomorrow.updateCustomSensor(tomorrow)
        end
        
        if not item.isHTTPResponse then
            getSolarData()
        elseif item.ok then
            local resultTable = item.json.result
            local watts = resultTable.watts
            local wattHours = resultTable.watt_hours_day
            
            local thisHour = getNearestHour(watts)
            local today = dz.time.rawDate
            local tomorrow = os.date("%Y-%m-%d",os.time()+24*60*60)
            
            local thisHourWatt = dz.utils.round(watts[thisHour] / 1000,3)
            local todayWattHours = dz.utils.round(wattHours[dz.time.rawDate] / 1000,3)
            local tomorrowWattHours = dz.utils.round(wattHours[tomorrow] / 1000,3)
            
            logWrite("thishour:                 " .. thisHour)
            logWrite("watts:                     " .. watts[thisHour])
            logWrite("watt-hours_day today:     " .. todayWattHours)
            logWrite("watt-hours_day tomorrow:     " .. tomorrowWattHours)
            
            updateSensors(thisHourWatt, todayWattHours, tomorrowWattHours)
            
        else
            dz.log("Problem with the JSON response",dz.LOG_ERROR) 
        end
    end
}
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1968
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: Forecast of the yield of solar panels

Post by FireWizard »

Hi,

Have you thought to use Node-Red for that goal?
You can load a node called node-red-contrib-solar-power-forecast-plus.
Write a simple flow and push the data in the form you like with MQTT into Domoticz.
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

@PieterS

The display on your browserscreen may have the contents of the JSON-file (or XML-file) with PVForecasts originating from forecast.solar, but the layout is different, because it went through additional processing for display:
what you see on the screen is not a JSON-file, nor XML-file!

As described here, the application of the Luftdaten JSON- or XML-output as input for Domoticz needs filtering to be performed on a file with repeated use of label-set value-type and value.
For easier filtering in some way you have to get rid of that repeating of the identical labels:
initially a 2-step solution is described here
In the subsequent messages following in the mentioned thread, in the end also showing a 1step-approach (which you apparently apply for your Dust-sensor). For the PVForecasts have also been looking to that latter 1step-approach, but not yet found a easy adaptation/translation for the PVForecasts, due to different structure of the JSON-/XML-file from forecast.solar.

For my PV-setup aiming at the forecast of Sum_Wh for today and for tomorrow, the JSON-file (and the corresponding XML-file) resulting from API-call to forecast.solar for /estimate/watthours/day/ with Step1 of my Luftdaten-approach would look like

Code: Select all

{"message": {"info": {"distance": 0, "pid": "h453eb5q", "place": "7559 KW Hengelo, Overijssel, NL"}, "text": "", "ratelimit": {"limit": 12, "remaining": 9}, "code": 0, "type": "success"}, "result": {"2019-04-13": 22248.24, "2019-04-12": 25376.4}}

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?><root><message type="dict"><info type="dict"><distance type="int">0</distance><pid type="str">h453eb5q</pid><place type="str">7559 KW Hengelo, Overijssel, NL</place></info><text type="str"></text><ratelimit type="dict"><limit type="int">12</limit><remaining type="int">9</remaining></ratelimit><code type="int">0</code><type type="str">success</type></message><result type="dict"><key type="float" name="2019-04-13">22248.24</key><key type="float" name="2019-04-12">25376.4</key></result></root>
Unlike a direct extraction from forecast.solar, this method of extraction has the handicap that in the resulting JSON- and XML-files the order/sequence of the segments cannot be controlled, and in the above result you see that the order/sequence of dates is not chronological.
Filtering on the above JSON-file also has a different twist related to the Luftdaten-approach, because 1] there are no clearly identical labels to get rid off, but 2] also it has no clearly visible, alternative labels to apply for sorting.
However, for extraction from the JSON-file above you might apply the idea from Step2 of the Luftdaten-approach by selection&search of unique textstrings marking the start of the segments and the end of the 'result'-field.
That method does not work on the XML-version, because in the XML-version, the 2 segments have identical format <key type = .......</key>

Summarized: yes, following 'my' Lufdaten-approach a download of a JSON-file is possible, but data-extraction is another story .......
Last edited by Toulon7559 on Friday 19 April 2019 17:05, edited 2 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: Forecast of the yield of solar panels

Post by PieterS »

Sorry, took some time to react.
Thanks to all for reply. I like to keep it simple. So for now I choose for the easiest solution: a script which is almost errorfree. :D

For now I start with solution of waaren.
I made three virtual custom sensors with Axis kWh named thisHour, today and tomorrow and copied the code in a new script in Domoticz. Saved as PV-forecast.
Image
In the logfile I got:

Code: Select all

 2019-04-15 18:30:00.504 Status: dzVents: Info: solarData: ------ Start internal script: PV-forecast:, trigger: every 5 minutes
2019-04-15 18:30:00.505 Status: dzVents: Debug: solarData: OpenURL: url = https://api.forecast.solar/estimate/51/5/45/10/3.5
2019-04-15 18:30:00.505 Status: dzVents: Debug: solarData: OpenURL: method = GET
2019-04-15 18:30:00.505 Status: dzVents: Debug: solarData: OpenURL: post data = nil
2019-04-15 18:30:00.505 Status: dzVents: Debug: solarData: OpenURL: headers = {["Accept"]="application/json"}
2019-04-15 18:30:00.505 Status: dzVents: Debug: solarData: OpenURL: callback = solarData
2019-04-15 18:30:00.505 Status: dzVents: Info: solarData: ------ Finished PV-forecast
2019-04-15 18:30:01.379 Status: dzVents: Info: Handling httpResponse-events for: "solarData
2019-04-15 18:30:01.379 Status: dzVents: Info: solarData: ------ Start internal script: PV-forecast: HTTPResponse: "solarData"
2019-04-15 18:30:01.386 Status: dzVents: Debug: solarData: thishour: 2019-04-15 19:00:00
2019-04-15 18:30:01.386 Status: dzVents: Debug: solarData: watts: 203
2019-04-15 18:30:01.386 Status: dzVents: Debug: solarData: watt-hours_day today: 14.98
2019-04-15 18:30:01.386 Status: dzVents: Debug: solarData: watt-hours_day tomorrow: 21.648
2019-04-15 18:30:01.386 Status: dzVents: Error (2.4.6): solarData: An error occured when calling event handler PV-forecast
2019-04-15 18:30:01.386 Status: dzVents: Error (2.4.6): solarData: ...cz/var/scripts/dzVents/generated_scripts/PV-forecast.lua:69: attempt to call field 'updateCustomSensor' (a nil value)
2019-04-15 18:30:01.386 Status: dzVents: Info: solarData: ------ Finished PV-forecast
There are some problems:
You mention in comment that dzVents >= 2.4.14 is required. In my logfiles it says dzVents version 2.4.6. Is that a problem? The script can create a url... There are results written in the log..

But: The sensors are not updated in the dashboard..

Almost there! :D
Synology with Domoticz build (V2024.7) in Docker
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Forecast of the yield of solar panels

Post by waaren »

PieterS wrote: Monday 15 April 2019 18:36 You mention in comment that dzVents >= 2.4.14 is required. In my logfiles it says dzVents version 2.4.6. Is that a problem?
No the comment " dzVents >= 2.4.14 is required" is is only for using the latitude / longitude as set in domoticz. If you enter those directly as values in the script it should work without problem.
But: The sensors are not updated in the dashboard.
I don't see all expected loglines. If you can share all loglines from one cycle (script start twice (second time shortly after the first one) and the complete script as it is now (you can blank the latitude / longitude) I will have a look.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

Based on personal experience/preference and available source-info, experimenting with a lua-script operating on the XML-files originating from forecast.solar.

First experiment operating on the Wh_per_day file:
#0 download of the XML_string
#1 find the start & end of each value-segment
a. definition of search-string
b. perform search to get position references
#2 extract contents from the value segments
#3 convert contents into output to Domoticz

The start-section of my script for #0 provides in Domoticz' Log from a printcommand a nicely formatted XML-string:
that seems sign that download-output is OK.

For Step #1 the relevant 1st part of the script

Code: Select all

-- Line 069, Make Search-strings for Date&Time; filled with info from date = os.date("*t")
datestring_day1 = year .. "-" .. month .. "-" .. day1 -- for Today
datestring_day2 = year .. "-" .. month .. "-" .. day2 -- for Tomorrow
timestring1 = year .. "-" .. month .. "-" .. day1 .. " " .. hour .. ":" .. minutes

timestring1 not yet required, but applicable for the other files to find various time-related value-segments
Print of the strings shows the info-fields as expected, matching the 'headers' in the value-segments, such as
2019-04-16

Relevant next part of the script

Code: Select all

-- Line 089, Find in the XML_string the info-fields based on their labels
       d1 = string.find(XML_string, datestring_day1)    -- read start reference position for today's segment
       d2 = string.find(XML_string, datestring_day2)     -- read start reference position for tomorrow's segment
       d3 = string.find(XML_string,"</result>")             -- read end reference position of value container
Until scriptline 090 no problem.
For scriptlines 090~092 no error report (which implies that the layout is 'legal' [?]), but reading the error-reports in the Log, scriptline 90 doesn't produce a number for d1, but a 'nil', and in the subsequent calculation of the exact character-positions that causes a problem.
Tracing back, therefore the reason must be somewhere in the scriptlines shown above, with from my side a suspicion to lines 90~92 related to handling of the Search-strings.

Any ideas what might be wrong in the layout of those scriptlines?
Last edited by Toulon7559 on Thursday 18 April 2019 13:16, edited 11 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: Forecast of the yield of solar panels

Post by PieterS »

waaren wrote: Monday 15 April 2019 23:03
PieterS wrote: Monday 15 April 2019 18:36 You mention in comment that dzVents >= 2.4.14 is required. In my logfiles it says dzVents version 2.4.6. Is that a problem?
No the comment " dzVents >= 2.4.14 is required" is is only for using the latitude / longitude as set in domoticz. If you enter those directly as values in the script it should work without problem.
But: The sensors are not updated in the dashboard.
I don't see all expected loglines. If you can share all loglines from one cycle (script start twice (second time shortly after the first one) and the complete script as it is now (you can blank the latitude / longitude) I will have a look.
Hi waaren,
Thanks for reply.

I didn't change anything in your script, excect the timer (every 10 minutes) in line 8. I made three virtual sensors as named in line 23, 24 and 25. See screenshot earlier.

Code: Select all

--[[ 
        get solar data from forecast.solar and store in a virtual P1 sensor.
--]] 

local webResponse = "solarData"

return  {   on =    {  
                      timer           = { 'every 10 minutes' },
                       httpResponses   = { webResponse },
                    },

        logging =   {
                        level       = domoticz.LOG_DEBUG,  -- switch to LOG_ERROR when all OK
                        marker      = webResponse
                    },
       
    execute = function(dz, item)
        
        local function logWrite(str,level)
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
        
        local solarEstimateCurrentHour = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateToday = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateTomorrow = dz.devices(nnn)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        
        -- local latitude =  dz.settings.location.latitude   -- these are the lat/lon set in domoticz (dzVents >= 2.4.14 required)
        local latitude =  xx -- set to your latitude 
        -- local longitude = dz.settings.location.longitude  -- or set to the value you want
        local longitude = xx  -- set to your longitude
        local planeDeclination = 45                       -- 0 (horizontal) … 90 (vertical)
        local planeAzimuth = 10                            --   -180 … 180 (-180 = north, -90 = east, 0 = south, 90 = west, 180 = north)
        local installedModulesPower =  3.5 -- in kilo watt

        local function getSolarData()
            local url = "https://api.forecast.solar/estimate" .. 
                        "/" .. latitude .. 
                        "/" .. longitude ..
                        "/" .. planeDeclination ..
                        "/" .. planeAzimuth ..
                        "/" .. installedModulesPower 
            dz.openURL  (   
                            {
                                url = url,
                                headers = { ['Accept'] = 'application/json' },
                                method = 'GET',
                                callback = webResponse
                            }        
                        )
        end
        
        local function getNearestHour(t)
            local Time = require('Time')
            local loopValue, delta = 0, 99999
            local nearest
            
            for key, value in pairs(t) do
                loopValue = dz.time.compare(Time(key)).secs  -- get absolute diff between now and date/time in response JSON
                if loopValue < delta then 
                    nearest = key
                    delta = loopValue
                end
            end
            
            return nearest -- return date/time string nearest to now
        end
        
        local function updateSensors(thisHour, today, tomorrow)
             solarEstimateCurrentHour.updateCustomSensor(thisHour)
             solarEstimateToday.updateCustomSensor(today)
             solarEstimateTomorrow.updateCustomSensor(tomorrow)
        end
        
        if not item.isHTTPResponse then
            getSolarData()
        elseif item.ok then
            local resultTable = item.json.result
            local watts = resultTable.watts
            local wattHours = resultTable.watt_hours_day
            
            local thisHour = getNearestHour(watts)
            local today = dz.time.rawDate
            local tomorrow = os.date("%Y-%m-%d",os.time()+24*60*60)
            
            local thisHourWatt = dz.utils.round(watts[thisHour] / 1000,3)
            local todayWattHours = dz.utils.round(wattHours[dz.time.rawDate] / 1000,3)
            local tomorrowWattHours = dz.utils.round(wattHours[tomorrow] / 1000,3)
            
            logWrite("thishour:                 " .. thisHour)
            logWrite("watts:                     " .. watts[thisHour])
            logWrite("watt-hours_day today:     " .. todayWattHours)
            logWrite("watt-hours_day tomorrow:     " .. tomorrowWattHours)
            
            updateSensors(thisHourWatt, todayWattHours, tomorrowWattHours)
            
        else
            dz.log("Problem with the JSON response",dz.LOG_ERROR) 
        end
    end
}


I am wondering which extra info should be seen in the logfile. There is no more info related to the script. I filter on "solarData".
In Domoticz, Settings I changed the loglevel to Debug (everything) but no difference.

Don't know how to start the script twice in Domoticz, shortly after the first one... Sorry

Code: Select all

 2019-04-16 19:58:00.277 Status: dzVents: Info: solarData: ------ Start internal script: PV-forecast:, trigger: every minute
2019-04-16 19:58:00.277 Status: dzVents: Debug: solarData: OpenURL: url = https://api.forecast.solar/estimate/xxx/45/10/3.5
2019-04-16 19:58:00.277 Status: dzVents: Debug: solarData: OpenURL: method = GET
2019-04-16 19:58:00.277 Status: dzVents: Debug: solarData: OpenURL: post data = nil
2019-04-16 19:58:00.277 Status: dzVents: Debug: solarData: OpenURL: headers = {["Accept"]="application/json"}
2019-04-16 19:58:00.277 Status: dzVents: Debug: solarData: OpenURL: callback = solarData
2019-04-16 19:58:00.277 Status: dzVents: Info: solarData: ------ Finished PV-forecast
2019-04-16 19:58:00.277 Status: dzVents: Debug: - OpenURL = {["method"]="GET", ["URL"]="https://api.forecast.solar/estimate/xx/xx/45/10/3.5", ["headers"]={["Accept"]="application/json"}, ["_trigger"]="solarData"}
2019-04-16 19:58:00.717 Status: dzVents: Debug: - HTTPResponse: solarData
2019-04-16 19:58:00.788 Status: dzVents: Info: Handling httpResponse-events for: "solarData
2019-04-16 19:58:00.788 Status: dzVents: Info: solarData: ------ Start internal script: PV-forecast: HTTPResponse: "solarData"
2019-04-16 19:58:00.794 Status: dzVents: Debug: solarData: thishour: 2019-04-16 19:53:00
2019-04-16 19:58:00.794 Status: dzVents: Debug: solarData: watts: 31.5
2019-04-16 19:58:00.794 Status: dzVents: Debug: solarData: watt-hours_day today: 16.352
2019-04-16 19:58:00.794 Status: dzVents: Debug: solarData: watt-hours_day tomorrow: 7.711
2019-04-16 19:58:00.794 Status: dzVents: Error (2.4.6): solarData: An error occured when calling event handler PV-forecast
2019-04-16 19:58:00.794 Status: dzVents: Error (2.4.6): solarData: ...cz/var/scripts/dzVents/generated_scripts/PV-forecast.lua:69: attempt to call field 'updateCustomSensor' (a nil value)
2019-04-16 19:58:00.794 Status: dzVents: Info: solarData: ------ Finished PV-forecast 
Have you been helped with this?
Synology with Domoticz build (V2024.7) in Docker
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Forecast of the yield of solar panels

Post by waaren »

PieterS wrote: Tuesday 16 April 2019 21:10 I didn't change anything in your script, except the timer (every 10 minutes) in line 8. I made three virtual sensors as named in line 23, 24 and 25. See screenshot earlier.
Did you change the nnn in line 23,24 and 25 to the actual device numbers and the xx on line 28 and 30 to the actual latitude, longitude?
Don't know how to start the script twice in Domoticz, shortly after the first one... Sorry
this is no action for you. It is just how the script is triggered. First time by the timer and the second time by the http response.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: Forecast of the yield of solar panels

Post by PieterS »

@waaren: You did it!!!

Your first hint about the device numbers made the difference. I did not know I had to fill them in. Still learning :|

Thank you very much!
Synology with Domoticz build (V2024.7) in Docker
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

waaren's latest script is nice, producing the actual value for forecasts for this hour, for today and for tomorrow:
;-) is a working solution, while my lua-script (at this moment) stuck at the read-out .....
But this comment is still applicable:
the widgets in Domoticz show actual info, and the related graphs in Domoticz trace the historic changes in the forecasts due to the changing inputs.
Matter of taste, but more interesting would be a 'look-ahead'-display, like we are accustomed for meteo-forecasts.
Would it be possible to adapt waaren's script also for online making such 'prospect-graphs' for 2 days?
Implies periodically
- reading the 2 files with forecast-power-per-hour and forecast-(cum)energy-per-hour,
- putting the hourly data in an array (or table) of at least 3 rows [time, forecast-power, forecast-(cum)energy], and
- produce a graph for that array/table.
In that way you get a picture similar to any other aspect of a 2day meteo-forecast.

Thinking ahead, an output-graph in the form of jpg-file would be easiest for further application in webpages.

Thinking even further ahead, for incorporation of the actual-power and actual-energy (or any other actual info), it implies addition of related rows filled each hour 'on-the-fly' with latest info from Domoticz. In that perspective a 1-day-graph for today is best.

;-) Some more challenges ....
Last edited by Toulon7559 on Saturday 20 April 2019 11:27, edited 4 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
PieterS
Posts: 197
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: NL
Contact:

Re: Forecast of the yield of solar panels

Post by PieterS »

@Toulon7559: Reached my first goal and yes, you are right. Your idea / proposal could be next. :D But no idea if it is possible to realize in this application. Conditions should be there.
Synology with Domoticz build (V2024.7) in Docker
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

Those graphs may be a remote goal, especially the dynamic 4-line graph for today's PV-info.
No preference which way of realization: by dzVents-script, by 'ordinary' lua-script or by Python-script, or other .....

With 3 Domoticz-interfaces with forecasted values try to make the translation to some gauges showing in a webpage the forecasted Power/Energy_ThisHour and Total_Energy_Today.
A gauge with Total_Energy_Today will be very static: Total_Energy_Today is just the 'ceiling-value' to be checked at the end of Today.
The more dynamic accumulating_Energy_Today would require to take the 'growing' Wh-values from the Watthours-table coming from forecast.solar: for that extraction need to add some lines to waaren's script and an extra Virtual Device.
At my website also having graphs with actual Power & Energy per day, that has some value for actual comparison between expectation and practise of PV-production.
With RRDTool (or some other charting software) it must be possible to compile a running 'comparison'-chart showing forecasted Power&Energy vs. actual Power&Energy: the goal mentioned in line1 of this message may be within reach.

;-) Progressing step by step .....

Addition April, 19, 2019
After a remark of waaren have quick-checked the info coming in the info-tables from forecast.solar.
Indeed, it seems that the forecasted W-powerlevel indicated per hour, and then taken for 1 hour, results in the Wh-step for that hour in the table with accumulating watthours per day => the hourly info in the Watts-table can also be taken as Wh-info for that hour.
Meanwhile adapted waaren's script in such way that it now also reads the hourly accumulating Energy-value per day to a 4th Virtual Sensor in Domoticz.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

Some fiddling with scripts and RRDTool resulted in a comparison-graph:
- the blue line is the forecasted 'PV-goal' per day
- the red and green lines are respectively the forecasted and actual power/energy values per hour
- the purple and black line are in the same way the accumulating energy values over the day
At the end of the day these latter 2 lines should be close to the blue line.

PV_Comparison_Graph
PV_Comparison_Graph
pipowerPVF1_20190426 [640x480].png (114.42 KiB) Viewed 5000 times

The background data may be correct, but in practise the resulting picture is a bit 'strange':
- the PV-forecasts are refreshed per hour in relation to the actual meteo-forecast in that hour
Refresh & fluctuations show as distinct steps.
- the actual PV_Power and actual PV_Energy in comparison are very detailed.

Conclusion: other (more appealing) way of presentation needed without those 'steps'.
For cloudy days sometimes the gap between Forecast and Practise is 'rather large'.
On the other hand, those 'steps' are clear visual sign that the estimation of values has a limited accuracy, and that such graphline is not 'the absolute truth'.
Last edited by Toulon7559 on Tuesday 02 July 2019 11:26, edited 1 time in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

After some experiments made a separate webpage aimed at comparison of Forecasted and Actual PV-production.

In addition to the graph discussed in the previous message, the present version has 3 gauges to show the forecasted values
[with a 4-gauge version to be called by weblink on this page].

Better layout seems if the gauges are in horizontal row.
Some read-outs could be combined, if it would be possible to have gauges with 2 pointers:
then you get immediate visual judgement whether 'forecasted' and 'actual' values are close or remote.

Has somebody seen a Java/Ajax-script for such 2-pointer gauges?
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
User avatar
FireWizard
Posts: 1968
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: Forecast of the yield of solar panels

Post by FireWizard »

Hi

I use waaren 's script now for some days.
In general it works "out of the box".
You have to fill in some project information, such as installed kWp, azimuth, declination and sensor idx.

The URL https://api.forecast.solar gives the data from approx 5-8 minutes before sunrise and after sunset.
In the "dark" hours the URL does not give any data, but the polling continues for every hour.
Of course that is useless but the script reports also an error.

2019-06-28 02:01:13.068 Error: Error opening url: https://api.forecast.solar/estimate/52. ... 10/-14/2.4
2019-06-28 02:01:13.223 Error: dzVents: Error: (2.4.24) solarData: response: 0 ==>>
2019-06-28 02:01:13.270 Error: dzVents: Error: (2.4.24) solarData: Problem with the JSON response
2019-06-28 03:01:02.001 Error: Error opening url: https://api.forecast.solar/estimate/52. ... 10/-14/2.4
2019-06-28 03:01:02.389 Error: dzVents: Error: (2.4.24) solarData: response: 0 ==>>
2019-06-28 03:01:02.394 Error: dzVents: Error: (2.4.24) solarData: Problem with the JSON response
2019-06-28 07:01:01.642 Error: Error opening url: https://api.forecast.solar/estimate/52. ... 10/-14/2.4
2019-06-28 07:01:01.897 Error: dzVents: Error: (2.4.24) solarData: response: 0 ==>>
2019-06-28 07:01:01.907 Error: dzVents: Error: (2.4.24) solarData: Problem with the JSON response

You see the error at 2 o'clock and 3 o'clock in the night. Why it happens also at 7, I have no idea.

Does anyone know a simple way to change the script, so that it is only polling, let's say 15 minutes before sunrise until 15 minutes after sunset?

In his addition of April 19, 2019 Toulon7559 mentions that he has adapted waaren's script in such way that it now also reads the hourly accumulating Energy-value per day to a 4th Virtual Sensor in Domoticz. As I'm also interested in that I ask if you can publish that script?

Thanks in advance

Regards
Last edited by FireWizard on Thursday 06 August 2020 14:09, edited 2 times in total.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Forecast of the yield of solar panels

Post by waaren »

FireWizard wrote: Friday 28 June 2019 18:33 Does anyone know a simple way to change the script, so that it is only polling, let's say 15 minutes before sunrise until 15 minutes after sunset?
Change line 8 to

Code: Select all

                    timer = { 'every 10 minutes between 15 minutes before sunrise and 15 minutes after sunset' },
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1968
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: Forecast of the yield of solar panels

Post by FireWizard »

Thank you, waaren.
I have updated the script and it ran overnight error free.
I didn' t know that it was so simple.
It updates now 3 sensors, but I was thinking to implement a fourth sensor, in order to read the hourly accumulated energy value per day. Any hints or suggestions?

Thanks in advance
Regards
Toulon7559
Posts: 859
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: <2025
Location: Hengelo(Ov)/NL
Contact:

Re: Forecast of the yield of solar panels

Post by Toulon7559 »

@FireWizard

As requested, my version of the script expanded for inclusion of a 4th device for accumulating energy per hour:|
obviously to be adapted for application with your configuration.

Code: Select all

--[[ 
        get solar data from forecast.solar and split & store in virtual sensors: script-version = script_time_PVForecast_4CS_dzVents.lua of 190419
--]] 

local webResponse = "solarData"

return  {   on =    {  
                      timer           = { 'every 30 minutes' },
                       httpResponses   = { webResponse },
                    },

        logging =   {
                        level       = domoticz.LOG_DEBUG,  -- switch to LOG_ERROR when all OK
                        marker      = webResponse
                    },
       
    execute = function(dz, item)
        
        local function logWrite(str,level)
            dz.log(tostring(str),level or dz.LOG_DEBUG)
        end
        
        local solarEstimateAccumHour = dz.devices(3148)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateCurrentHour = dz.devices(3136)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateToday = dz.devices(3126)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        local solarEstimateTomorrow = dz.devices(3127)   -- Create this virtual sensor as "Custom Sensor" (x-as KWh)
        
        -- local latitude =  dz.settings.location.latitude   -- these are the lat/lon set in domoticz (dzVents >= 2.4.14 required)
        local latitude =  52.29626465 -- set to your latitude (signed decimal degrees) 
        -- local longitude = dz.settings.location.longitude  -- or set to the value you want
        local longitude = 6.80555010  -- set to your longitude (signed decimal degrees)
        local planeDeclination = 35                       -- 0 (horizontal) … 90 (vertical)
        local planeAzimuth = -17                            --   -180 … 180 (-180 = north, -90 = east, 0 = south, 90 = west, 180 = north)
        local installedModulesPower =  4.0 -- in kilo watt (EFFECTIVE power, NOT the nominal power!)

        local function getSolarData()
            local url = "https://api.forecast.solar/estimate" .. 
                        "/" .. latitude .. 
                        "/" .. longitude ..
                        "/" .. planeDeclination ..
                        "/" .. planeAzimuth ..
                        "/" .. installedModulesPower 
            dz.openURL  (   
                            {
                                url = url,
                                headers = { ['Accept'] = 'application/json' },
                                method = 'GET',
                                callback = webResponse
                            }        
                        )
        end
        
        local function getNearestHour(t)
            local Time = require('Time')
            local loopValue, delta = 0, 99999
            local nearest
            
            for key, value in pairs(t) do
                loopValue = dz.time.compare(Time(key)).secs  -- get absolute diff between now and date/time in response JSON
                if loopValue < delta then 
                    nearest = key
                    delta = loopValue
                end
            end
            
            return nearest -- return date/time string nearest to now
        end
        
        -- local function updateSensors(thisHour, today, tomorrow)
        local function updateSensors(thisHour, accum, today, tomorrow)
             solarEstimateCurrentHour.updateCustomSensor(thisHour)
             solarEstimateAccumHour.updateCustomSensor(accum)
             solarEstimateToday.updateCustomSensor(today)
             solarEstimateTomorrow.updateCustomSensor(tomorrow)
        end
        
        if not item.isHTTPResponse then
            getSolarData()
        elseif item.ok then
            local resultTable = item.json.result
            local watts = resultTable.watts
            local wattHour = resultTable.watt_hours
            local wattHours = resultTable.watt_hours_day
            
            local thisHour = getNearestHour(watts)
            local accum = getNearestHour(wattHour)
            local today = dz.time.rawDate
            local tomorrow = os.date("%Y-%m-%d",os.time()+24*60*60)
            
            local thisHourWatt = dz.utils.round(watts[thisHour] / 1000,3)
            local thisHourWh = dz.utils.round(wattHour[accum] / 1000,3)
            local todayWattHours = dz.utils.round(wattHours[dz.time.rawDate] / 1000,3)
            local tomorrowWattHours = dz.utils.round(wattHours[tomorrow] / 1000,3)
            
            logWrite("thishour:                 " .. thisHour)
            logWrite("watts:                    " .. thisHourWatt)
            logWrite("watt-hours_cum today:     " .. thisHourWh)
            logWrite("watt-hours_day today:     " .. todayWattHours)
            logWrite("watt-hours_day tomorrow:  " .. tomorrowWattHours)

            -- updateSensors(thisHourWatt, todayWattHours, tomorrowWattHours)
            updateSensors(thisHourWatt, thisHourWh, todayWattHours, tomorrowWattHours)
            
        else
            dz.log("Problem with the JSON response",dz.LOG_ERROR) 
        end
    end
}
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
User avatar
FireWizard
Posts: 1968
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: Forecast of the yield of solar panels  [Solved]

Post by FireWizard »

@Toulon7559

Thank you very much for submitting your version of the script.

In the mean time I had started already to look at it and to modify it. It will not surprise you that your version and my version of the script are more or less identical, except of the names of the variables.

My script differs from yours at the following.

1. If your version of dzVents support it, you can get the lat/lon from Domoticz
Change the following

local latitude = dz.settings.location.latitude -- these are the lat/lon set in domoticz (dzVents >= 2.4.14 required)
-- local latitude = 51 -- set to your latitude
local longitude = dz.settings.location.longitude -- or set to the value you want
-- local longitude = 5 -- set to your longitude

2. To my opinion the following should be in W(atts) instead of kWh, because the table presents the electrical production at that specific time and not the energy produced over a specific time.
local solarEstimateCurrentHour = dz.devices(3136) -- Create this virtual sensor as "Custom Sensor" (x-as KWh)

You can simple change the X-ax to kW.
See the table for today:
watts: object
2019-07-02 05:14:00: 0
2019-07-02 05:37:00: 10
2019-07-02 06:00:00: 62
2019-07-02 07:00:00: 223
2019-07-02 08:00:00: 454
2019-07-02 09:00:00: 725
2019-07-02 10:00:00: 1003
2019-07-02 11:00:00: 1202
2019-07-02 12:00:00: 1853
2019-07-02 13:00:00: 1956
2019-07-02 14:00:00: 1918
2019-07-02 15:00:00: 1738
2019-07-02 16:00:00: 1457
2019-07-02 17:00:00: 1104
2019-07-02 18:00:00: 725
2019-07-02 19:00:00: 382
2019-07-02 20:00:00: 142
2019-07-02 21:04:00: 19
2019-07-02 22:08:00: 0

If you want it to be presented as Watt instead of kW change the following line:
local thisHourWatt = dz.utils.round(watts[thisHour] / 1000,3)
to
local thisHourWatt = dz.utils.round(watts[thisHour])

3. In addition I use waarens suggestion regarding the poll frequency of the script.
As you have 12 free calls/hour to forecast.solar you can poll every 10 minutes,
but 30 minutes are fine and even 1 hour will do.
In my opinion it is unnecessary to poll the URL before sunrise and after sunset.
And it avoids an error from the script timer = { 'every 10 minutes between 15 minutes before sunrise and 15 minutes after sunset' },

Thank you once more and best regards.
Last edited by FireWizard on Thursday 06 August 2020 14:15, edited 3 times in total.
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest