Protocol for Upload to Sonnenertrag.eu

[Toulon7559]

Many ways seem available to upload data to Sonnenertrag.eu
Before the 'big shift' at Sonnenertrag.eu in 2018 I had (historically developed) 3 PV-subsystems running based on the importlink with PVOutp.org, and those 3 subsystems happily restarted after 'revival' of Sonnenertrag.eu
For my latest addition to the PV-configuration I tried to link also the subject PVOutput-sid to Sonnenertrag.eu, but unfortunately the setup for id+sid was refused by Sonnenertrag.eu for reason which basically translates to 'Sonnenertrag cannot read the pvo-account: power-setting not matching'.
Surprising, because no problem for the 3 earlier PVO-sids.
Looking at the example-info would not be surprised if Sonnenertrag.eu also 'stumbles' at my latest PVO-addition having an sid of 5 digits, and (according to example-text) Sonnenertrag only assuming an input of 4 digits.
Annoying that for 1 PV-subsystem no compatible automatic companion upload from PVOutput.org of info to Sonnenertrag.eu

In the setup-menu for upload of PV-inverter info to Sonnenertrag.eu is an Option for 'Self-made' datalogger.
Implicitly it means that then a protocol must be applied compatible with Sonnenertrag's data-entry portal.
Should be in the WiKi of Sonnenertrag, but calling wiki-url = 'no response' = blank screen and waiting forever .......
Access to Sonnenertrag by GoogleChrome is generally slightly quicker/better than with Firefox, but not in this case.

19July2021:
First section from next message shifted into this message.
'No response' probably is an aspect related to the O.S in combination with Browsertype.
Above comment is applicable for Windows@PC, but not for iOS@iPad: see next message.

[Toulon7559]

As indicated, need to find an alternative solution for data-upload.
Recently revisiting this question, and with some different Google-searchkeywords, now seem to have found access to the wiki of Sonnenertrag.eu and to some related forum-messages.
Do not ask why, but not on my PC&Windows, but on my iPad I can reach the wiki of Sonnenertrag.eu (through a direct call to lower level of the wiki):
e.g. https://wiki.sonnenertrag.eu/datenimport:json and https://wiki.sonnenertrag.eu/datenimport:selfmade and https://wiki.sonnenertrag.eu/datenimpor ... g:solarlog
The required general setup is best summarized in wiki-section https://wiki.sonnenertrag.eu/datenimpor ... aussetzung and it's subsections
On file-handling some practical example-info from users at https://www.photovoltaikforum.com/threa ... -erzeugen/

The description below is resulting from my experiments using bits & pieces of info to get (with try & error) a working implementation of 'Selfmade' upload.
A rather long story, but no reason that 'successors' in this forum meet the same pitfalls and/or need to reinvent the wheel:
summary below with the bloody details to be found in the spoiler .....

1. Manual filling of data-gaps
Manual filling of the gaps is applicable if you already have some registration of values before you try to start automatic upload towards Sonnenertrag.eu, and would like to include historic data into that registration. Useful to get acquainted with Sonnenertrag.eu
Is basic setup if you just want manual data entry per completed month (considering automatic upload a luxury)
2 Possible layouts for data-processing:
either highly detailed by insertion of value-per-day, or by value-per-month with min. & max.
Can be selected in bottom rows of upper right corner of UserMenu, and
then manually enter the required data for the (incomplete) month(s) in the database of Sonnenertrag.eu
2. Filling of data-gaps by Upload of Month-infotable
For automated upload of month-data, goto setup-window for your installation [= top rows of upper right corner of UserMenu]
For 'Selfmade'-import it is required that in the 3rd setup-window you insert the hosting-url = http://<hosting-folder>/months.js
<hosting-folder> is your privately selected designated folder at a server at internet to which your system uploads the file, to be grabbed by Sonnenertrag.eu at a later time [= next morning].
2 Files are needed as inputs, to be placed at <hosting-folder>:
- File base_vars.js is required for the configuration setup of the 'Selfmade' data-import.
- File months.js contains data per month: from months.js each morning only latest 3 months in the file are being read by Sonnenertrag.eu
3. Filling of data-gaps by Upload of Day-infotable
For this mode of operation it is required that in the setup-window for 'Selfmade'-import you keep the hosting-url http://<hosting-folder>/months.js
- File base_vars.js is required same as in 2.
- File days_hist.js contains data per day: from days_hist.js each morning only latest day is being read by Sonnenertrag.eu

Spoiler: show

To arrive at application of an automatic 'Selfmade' uploadfunction the following multistep slow-train' approach might be Run-up incl. learning curve:
1. Manual filling of data-gaps
Manual filling of the gaps is applicable if you already have some registration before you try to start automatic upload towards Sonnenertrag.eu, and would like to include historic data into that registration. Useful to get acquainted with Sonnenertrag.eu
Is basic setup if you just want manual data entry per completed month (considering automatic upload a luxury)
Also for bulk filling of any data-gaps over 2018 till today unfortunately the quickest method is this laboursome procedure using the setup web-pages of Sonnenertrag.eu for 'processing of data' [= upper righhand corner of 'Usermenu'].
2 Possible layouts for data-processing: either highly detailed by insertion of value-per-day, or by value-per-month with min. & max.
manually enter the required data for the (incomplete) month(s) in the database of Sonnenertrag.eu
2. Filling of data-gaps by Upload of Month-infotable
Goto setup-window for your installation [= upper right corner of UserMenu]
For 'Selfmade'-import it is required that in the 3rd setup-window you insert the hosting-url = http://<hosting-folder>/months.js
<hosting-folder> is your privately selected designated folder at a server at internet to which your system uploads the file to be grabbed by Sonnenertrag.eu at a later time [= next morning].
File base_vars.js is required for the setup of the 'Selfmade' data-import. Described in wiki as mandatory, but it is duplicate of info already inserted by the web-setup (anyway, a simple & once-only effort to make and upload this file base_vars.js with info compatible with the web-setup)
File months.js contains data per month: from months.js each morning only latest 3 months in the file are being read by Sonnenertrag.eu
(Manually) make file months.js starting after the last complete month in database of Sonnenertrag.eu, as filled with 1. and upload file months.js to the <hosting-folder>. Sonnenertrag.eu grabs the file in next morning.
=> check upload & pick-up from <hosting-folder> by Sonnenertrag.eu => if OK for file with 1 month, then
in a series of days (manually) expanding file months.js with max. 3 months/day till database filled upto/incl.last month.
Once this method has been proven over several days, to accelerate the filling of data-gaps, useful to revert to use of method 1. for filling per month for remaining 'empty' months.
May be candidate for automation by script if your PV-system provides montly data for PV-production, but
own experience (looking at results in the webtable for manual insertion, with manual insertion halfway in a month) is that this function in this way happily reports that everything OK, but only makes a strange filling for minimal month-value, although the file seems to meet the prescribed format.
Emperically found that apparently no completed months are allowed later than the remaining gap.
Quick&Dirty solution:
- in the webtable for manual insertion clear the data-windows for all subject months till running month,
- untick all visible boxes 'autofill',
- 'save' this cleaned webtable,
- manually fill all data-windows,
- tick the 'autofill'-boxes and
- save.
3. Filling of data-gaps by Upload of Day-infotable
For this mode of operation it is required that in the setup-window for 'Selfmade'-import you keep the hosting-url http://<hosting-folder>/months.js
File base_vars.js is required same as in 2.
File days_hist.js contains data per day: from days_hist.js each morning only latest day is being read by Sonnenertrag.eu
(Manually) make days_hist.js starting after the last month in database of Sonnenertrag.eu, as filled with 2. and upload to the hosting-url. Sonnenertrag.eu grabs the file in next morning.
=> check upload & pick-up from <hosting-folder> by Sonnenertrag.eu => if OK for file with 1 day, then
in a series of days (manually) expanding file day_hist.js with 1 day/day.
Once this method has been proven over several days you have checked the functional mechanism, but you will not hook on to the actual data-registration (because of pace of 1day/day):
to accelerate the filling of data-gaps till 'yesterday', useful to revert to use of method 1. for filling per day of remaining 'empty' days.
Candidate for automation by script for upload of daily PV-production.
For 'Express' way of testing, see further down this section.
4. Automatic dailly upload
The text outside the spoiler is all needed for this mode of operation.

During test, for check of correct operation of automatic file-upload, next morning do read-back of the hosting-url (dependent on the setting of your <hosting-folder>)
by means of http://<hosting-folder>/months.js and http://<hosting-folder>/days_hist.js
or by https://<hosting-folder>/months.js and http://<hosting-folder>/days_hist.js

Possible alternative 'Express' way of testing for 3.:
a. Manually compile all files mentioned under 4. and perform upload to the <hosting folder>
b. Login to Sonnenertrag.eu
c. In upper right corner under 'Installatiebeheer' choose the applicable installation
d. 'walk' the setup pages,
d1. check&fill as required the first 2 pages
d2. fill-in the setup-page3 for 'Self-made' mode (for howto see 2. above)
d3. check at setup-page 3 that plausibility-test is OK
d4. click 'trigger automatic import'
e. check at display of graphs whether data shows as expected

4. Automatic dailly upload/update
For subsequent automatic upload starting 'today' then 'only' the development is required of a small script [using lua or Python or dzVents or ...] generating the required js-files and daily uploading them to the tested <hosting-folder> for pick-up by Sonnenertrag.eu in the next morning of upload-files:
- months.js (to be generated/uploaded at the end of each day)
- days_hist.js (also to be generated/uploaded at end of each day)
- base_vars.js (once-only effort to make and upload this file, as configuration-reference)
You have choice for implementation using either months.js (with daily update for production of running month), or days.hist.js (reporting daily production)

Just wondering whether & why to generate & upload the other file min_day.js as mentioned in wiki and forum-messages:
- min_day.js only seems required if you like to join the competition to be the daily topstation at Sonnenertrag.eu
[it also keeps your system busy with an upload desired every 5 minutes]

July 15th till July 20th, 2021:
Editing to include some more explanation, for own experience with automated month-input, for alignment with previous & next messages, shifting experimental details to the spoiler

[Toulon7559]

Devil is in details

The startup-section of a lua-script for setting, reading & compilation of Time & PV-values is 'as usual', and therefore not described in this message.
Input/upload of day-values being the simplest approach, next segment of script is intended to generate the required outputfile days_hist.js

Code: Select all

-- Line099 = Generate the upload-file in the period between 23:45 till 00:00
-- Upload-file days_hist.js must have contents/line like da[dx++]="01.07.21|210;81"  [for info: ASCII-code for " is B2]
if ((hourvalue > 22) and (minutes > 45)) then
    file = io.open("/home/pi/days_hist.js", "w+")
   -- Opens a working file named day_hist.js (stored under the designated sub-folder of Domoticz)
   -- not (yet) in append mode for growing list, but as rewrite [= 1 line with latest, actual info]
-- Line105 = write line to opened file
    file:write("da[dx++]=" .. """ .. day .. "." .. month .. "." .. year_short .. "|" .. actldayenergy0580 .. ";".. maxdaypower0580  .. """ .. "\n")
    file:close() -- closes the open file
-- Line 108 = Next line is required to set permissions for open&copy&shift of the file 
    os.execute("chmod a+rw /home/pi/days_hist.js")
end

But that causes an error-report

Code: Select all

 2021-07-06 10:29:00.482 Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_SE0580_Ginlong_upload_energy00_RPI3B4.lua: ...ua/script_time_SE0580_Ginlong_upload_energy00_RPI3B4.lua:104: ')' expected near '" .. day .. "'

Obviously 3* " in row is not acceptable for lua-script.
Have tried a trick working for upload to WUnderground, by insertion of the ASCII-code for "

Code: Select all

    file:write("da[dx++]=" .. "%B2" .. day .. "." .. month .. "." .. year_short .. "|" .. actldayenergy0580 .. ";".. maxdaypower0580  .. "%B2" .. "\n")

=> no problems generating the output-file, but the resulting file is not accepted at Sonnenertrag.eu

Anybody knowing a method to generate that " as part of the outputstring?

23July2021:
Top 2 lines edited, and next info added as explanation.
At PVOutput.org we are accustomed that an upload results in refresh of data in tables and graphs with short delay, enabling display of 'Live' data.
Be aware that the server(s) of Sonnenertrag.eu grab(s) data from the uploaded files only once a day for values of the previous day (or further back in history).
Detected impact of that aspect, because now using 2 different protocols for upload to Sonnenertrag.eu.
The wiki explains and tells at which grabbing-times for which upload-configuration [ROM = somewhere between midnight and noon].
Also to bear in mind that Sonnenertrag.eu at 14:30 each day has another, general grabbing-run to include any upload missed in the first grabbing-run!!!
Implication: generate & upload the day's value at the end of the day and keep unchanged until next evening.
= Only update & upload the actual values after sunset!!!!!

[EddyG]

Normally this could done by ""' (single double single).
Sometimes it also works with an extra (fourth) " (double).
I looked at that site, lot of the things on the website don't even work properly, that does not promise anything good for the data part.
I upload to pvoutput and that works fine.

[Toulon7559]

@EddyG

Like you, using PVOutput.org as main collector, register & presenter for my PV-data.

Indeed at Sonnenertrag.eu quite a few functions seem inactive since custodianship of the organisation shifted to other hands.
Surprising that some functions don't work via Windows@PC, but on other hand can be used via iOS@iPad.
Things also do not always work according to the available descriptions, or not at all ......
Observations during first month of operation:
Manually inserted missing month-values [if without any supporting day-values in the background] for unclear reasons sometimes disappear from the database
=> Important to keep your notes, enabling a quick re-insertion of values!
=> more permanent solution is the (laboursome) manual insertion of missing background day-values.
Also the automatically online-inserted day-values do not seem to cumulate inside the database to the related, expected actual month-value.
=> Requires constant watch on progress etc.
Lacking & lagging maintenance and instability of the website??

However, 3 subsystems for more than 10 years running (without problems) towards that database (in fact by reading through link from PVOut.org) is a reason to still try to add the 4th subsystem to that configuration: 'more of the same' and keeping the long history at one place.
Essential is that the registration of values for the own PV-system is OK, even if 'some' doubt about the values for country & province.
Functional argument to stay:
- their graphs expressing in kW/Wp are a nice way for relative comparison over the life-cycle and to other systems.

Some message in the Photovoltaik Forum (which is labelled to be the forum supporting Sonnenertrag.eu) seems to hint that other 'separators' than "(=double) are allowed.
Tried 2 variations towards Sonnenertrag.eu:

Code: Select all

da[dx++]=01.07.21|210;81

Code: Select all

da[dx++]='01.07.21|210;81'

Unfortunately in this aspect Sonnenertrag.eu is very stubborn, and only accepts "(=double) as 'separator'.
Your hint to '"" (single-double-single) did the trick in the lua-script, now producing the acceptable string.
Therefore the subject scriptline to be changed into

Code: Select all

file:write("da[dx++]=" .. '"' .. day .. "." .. month .. "." .. year_short .. "|" .. actldayenergy0580 .. ";".. maxdaypower0580  .. '"' .. "\n")

The examples in their wiki show integer values, but real values are accepted as well:
output of integer values requires insertion of some script-lines with a rounding function for those values.

The end-result counts:
basic form is a working script for automated upload of daily values by means of 'Selfmade' datalogger in Domoticz' lua-script
[you just need to add 'head&tail' to the segment mentioned in the previous messasage, as amended in this message]
Based on experience, seems required that monthly value is checked at end of the finished month:
each month a manual action for end-of-month value is quick & easy, while a scripting solution to automatically generate the end-of-month-value is much more effort.

August01, 2021:
inserted redmarked and italic text
August11, 2021:
added bottom line