Meteo data in Cumulus uploadfile

Toulon7559 · Post by **Toulon7559** » Thursday 07 January 2021 15:09

Many meteo stations periodically generate themselves files for upload to WUnderground.
In this forum also a wide variety of scripts to generate similar upload files for other organisations, such as PWSW and WOW, or quite different files for upload to AWEKAS.
Now have the requirement to mimick a Cumulus-like upload-file to feed a certain non-standard upload from Domoticz towards the database of HetWeerActueel, a Benelux-organisation of meteo-enthousiasts.

Intended general layout (for maximum coverage)

Code: Select all

BOF

$stationDate = "<#day>-<#month>-<#year>";
$stationTime = "<#hour>:<#minute>";

$tempUnit = "<#tempunit>";
$humUnit = "%";
$barUnit = "<#pressunit>";
$rainUnit = "<#rainunit>";
$rateUnit = "<#rainunit>/hr";
$windUnit = "<#windunit>";

$sunriseTime = "<#sunrise>";
$sunsetTime = "<#sunset>";

$outsideTemp = "<#temp>";
$hiOutsideTemp = "<#tempTH>";
$lowOutsideTemp = "<#tempTL>";

$lowOutsideTempTime = "<#TtempTL>";
$hiOutsideTempTime = "<#TtempTH>";

$lowMonthlyOutsideTemp = "---";
$hiMonthlyOutsideTemp = "---";
$hiYearlyOutsideTemp = "---";
$lowYearlyOutsideTemp = "---";

$outsideHumidity = "<#hum>";
$lowHumidity = "---";
$hiHumidity = "---";
$lowHumTime = "---";
$hiHumTime = "---";
$hiMonthlyHumidity = "---";
$lowMonthlyHumidity = "---";
$hiYearlyHumidity = "---";
$lowYearlyHumidity = "---";

$outsideDewPt = "<#dew>";
$hiDewpoint = "---";
$lowDewpoint = "---";
$hiDewpointTime = "---";
$lowDewpointTime = "---";
$hiMonthlyDewpoint = "---";
$lowMonthlyDewpoint = "---";
$hiYearlyDewpoint = "---";
$lowYearlyDewpoint = "---";

$windSpeed = "<#wlatest>";
$wind10Avg = "<#wspeed>";
$hiWindSpeed = "<#windTM>";
$hiWindSpeedTime = "<#TwindTM>";
$hiMonthlyWindSpeed = "---";
$hiYearlyWindSpeed = "---";

$windDir = "<#bearing>";
$windDirection = "<#wdir>";

$windChill = "<#wchill>";
$lowWindchill = "---";
$lowWindchillTime = "---";
$lowMonthlyWindchill = "---";
$lowYearlyWindchill = "---";

$outsideHeatIndex = "<#heatindex>";
$hiHeatindex = "---";
$hiHeatindexTime = "---";
$hiMonthlyHeatindex = "---";
$hiYearlyHeatindex = "---";

$thw = "---";

$hiTHSWindex = "---";
$hiTHSWindexTime = "---";
$hiMonthlyTHSWindex = "---";
$hiYearlyTHSWindex = "---";

$barometer = "<#press>";
$barTrend = "<#presstrend>";
$lowBarometer = "<#pressTL>";
$hiBarometer = "<#pressTH>";
$lowMonthlyBarometer = "---";
$hiMonthlyBarometer = "---";
$lowYearlyBarometer = "---";
$hiYearlyBarometer = "---";
$lowBarometerTime = "<#TpressTL>";
$hiBarometerTime = "<#TpressTH>";

$dailyRain = "<#rfall>";
$stormRain = "---";
$monthlyRain = "<#rmonth>";
$totalRain = "<#ryear>";

$rainRate = "<#rrate>";
$hiRainRate = "<#rrateTM>";
$hiRainRateTime = "<#TrrateTM>";
$hiRainRateHour = "---";
$hiMonthlyRainRate = "---";
$hiYearlyRainRate = "---";

$solarRad = "<#SolarRad>";
$hiSolarRad = "---";
$hiSolarRadTime = "---";
$hiMonthlySolarRad = "---";
$hiYearlySolarRad = "---";

$uv = "<#UV>";
$hiUV = "---";
$hiUVTime = "---";
$hiMonthlyUV = "---";
$hiYearlyUV = "---";

$insideTemp = "---";
$hiInsideTemp = "---";
$lowInsideTemp = "---";
$hiInsideTempTime = "---";
$lowInsideTempTime = "---";
$hiMonthlyInsideTemp = "---";
$lowMonthlyInsideTemp = "---";
$hiYearlyInsideTemp = "---";
$lowYearlyInsideTemp = "---";

$insideHumidity = "---";
$hiInsideHumidity = "---";
$lowInsideHumidity = "---";
$hiInsideHumidityTime = "---";
$lowInsideHumidityTime = "---";
$hiMonthlyInsideHumidity = "---";
$lowMonthlyInsideHumidity = "---";
$hiYearlyInsideHumidity = "---";
$lowYearlyInsideHumidity = "---";

$insideDewPt = "---";

$insideHeatIndex = "---";

$forecast = "<#forecast>";

$cumulusversion = "Versie <#version>, build <#build>"

EOF

Most fields can be filled quite straightforward by taking actual data from Domoticz, either reading local sensor data or reading the related external data (such as from WU).
Some derived data is another story:
- high/low value per day for temp/hum/baro/wind/rain
- times of these high/low values
- cumulative rain this week/month
Has somebody found a solution to get these data without installing & maintaining a separate database?
Some of these derived data somewhere 'hidden' in Domoticz' database or elsewhere in Domoticz?
Manipulation of a (long) list of uservariables might be a practical solution to produce those values, but have the feeling that more 'hightech & lean' is possible.
Script type is not a requirement, and lua-script would be OK, although Python or dzVents would be nicer.

waaren · Post by **waaren** » Thursday 07 January 2021 15:43

Toulon7559 wrote: Thursday 07 January 2021 15:09 Many meteo stations periodically generate themselves files for upload to WUnderground.
In this forum also a wide variety of scripts to generate similar upload files for other organisations, such as PWSW and WOW, or quite different files for upload to AWEKAS.
Has somebody found a solution to get these data without installing & maintaining a separate database?

Nice project !
My approach would be to use a combination of a bash script and dzVents.

The bash script to query the current devicestates and short and long term database history for temp, baro, wind, etc using sqlite commands and when ready send to it to dzVents as JSON formatted data to be combined with the other data.

Toulon7559 · Post by **Toulon7559** » Friday 08 January 2021 9:50

Waaren,

Good suggestion, not using an extra database!

Have to investigate Domoticz' database for the presense & layout of those history records, and need to dust off my knowledge of sqlite commands.
In the run-up (till the read-out of the database is OK & Complete), I could temporarily use the mentioned list of user variables as fill-in-for-testing for the most important variables.
Determining the lowest & highest per day is not difficult, just scripting-work, and functionally could be as simple as:
- define 4 user-variables [day_low, day_high, time_low, time_high]
- at 00:00 reset & fill with then-actual values
- periodically check the actual values against the user variables
. if actual value higher than day_high, replace day_high by actual value + update time_high
. if actual value lower than day_low, replace day_low by actual value + update time_low
=> min & max available as floating values with time-resolution defined by interval for checking

Slightly more clever would be application of an array per variable.
That construction also allows easy checking of values per past hour, per past 3 hours or per past 24 hour, and calculation of average value over past 10 minutes: comparison over last hour and last 3 hours to determine tendency of air pressure, and average value over last 10 minutes for windspeed_average.
Functionality:
- periodically write the actual value&time into the array
- at (other) interval of choice, check the array for min & max & average in combination with time
=> min & max & average available with validity and time-resolution defined by the grid of array

Even with the minimal list of entries for Cumulus for this application the required list of user variables or the list of arrays will grow rather fast, and calling the database may be more lean & mean. However a learning curve for the latter solution .....

waaren · Post by **waaren** » Friday 08 January 2021 17:39

Toulon7559 wrote: Friday 08 January 2021 9:50 Good suggestion, not using an extra database!
...
Even with the minimal list of entries for Cumulus for this application the required list of user variables or the list of arrays will grow rather fast, and calling the database may be more lean & mean. However a learning curve for the latter solution.

Happy to work with you to get this going but to make it interesting for me, only when you are on dzVents >= 3.1.0 ( because of the new executeShellCommand() feature ) and if we share the final result on the public part of the forum.

Just send me a PM if you are interested.

Toulon7559 · Post by **Toulon7559** » Wednesday 13 January 2021 13:55

Whatever the scripttype is for the generation of the required uploadfile,
the effort starts with analysis which data is required, which data is available, and which data must be 'made' by calculation and/or from other sources.

In the file below a preliminary 'translation-list'.
Quite on purpose the fields to be 'life' have been minimized, to reduce the need for historic data which certainly is not directly available.
Filling of (empty) fields now marked with "---" is OK, but only if without additional effort = data already available and easy to access.

Would most appreciate if 'Domoticz-fellows' would look & comment for the columns with titles "Domoticz" and "Script-operation", to see whether a simplication would be possible (or a filling of empty fields as mentioned above)

Toulon7559 · Post by **Toulon7559** » Monday 20 September 2021 16:53

Got a first concept-version from Waaren as dzvents-script and this summer have been working on it, unfortunately without the expert-help of the originator.
Once understood how the script operates, will show in this forum for further cooperative clean-up & release.

Toulon7559 · Post by **Toulon7559** » Thursday 10 February 2022 10:45

Understanding another person's software is always a challenge, and especially products from a superior programmer like waaren (dearly missed), while myself not well-versed in dzVents.
Over time, tried several approaches, but without success:
kindly requested to contribute any ideas/thoughts what he might have intended, and how to realise.

His proposal for compilation of a HWA-file from Domoticz-data has the following setup.

Code: Select all

        this dzVents trigger an offline bashscript to get some meteo data for emulating a Cumulus file

            The bash script will:
                Copy the domoticz database to /tmp/cCopy.db
                Collect historic Data
                It will then report it's findings back to dzVents.

        Requires: dzVents 3.1.0

        Tested for database version 147 = domoticz version V2020.2 (build 12822 - 12850)

His initial proposal has been succesfully running in his setups with demonstrated correct output, but my configuration apparently different enough to cause havoc: have also the nagging feeling that different version of dzvents may play a role ......
Have achieved (working backwards) that the script starts segmentwise running (during testing of segments), but not proceeding very far from the 'normal, frontend start', and stumbling/stuck at the aspect described below.

The script-declaration has been changed from waaren's version to match the appropriate locations within my configuration

Code: Select all

 -- Controlvars           (Insert here the location of files, sources & destinations)
                callback = scriptVar,
                sqlite = '/usr/bin/sqlite3',                    -- '/usr/local/bin/sqlite3', -- set to location of your sqlite3 binary
                targetLocation = '/tmp',                        -- working directory
                domoticzDatabaseDirectory = '/home/pi/domoticz',   -- '/opt/domoticz', -- where can the script find your database to copy it ?
                output = '/tmp/Cumulus.txt',                    -- file location for cumulus emulation

In the bash-file the process comes to a halt with error-report pointing to the bottom-scriptline in this segment

Spoiler: show: #/bin/bash
#
# Save the parameter send to this script
#
parms=$1

#
#debug
#
#echo $parms > parms.debug

#
# convert parm (JSON string) to script variables
#
for parm in $(echo $parms | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]"); do
export $parm
echo $parm >> out.data # debug / development
done

# Line 20
# shorthands
#
result=$targetLocation/Cumulusresult_$$.json
workingDatabase=$targetLocation/domoticz_$$.db

#
# Prepare (nullify) resultfile
#
truncate -s 0 $result
# line 030
#
# Create working copy of the database. Using the .backup command to ensure database will not get blocked
#
sudo $sqlite $domoticzDatabaseDirectory/domoticz.db ".backup $workingDatabase"

#
# convert CSV formatted data to json
#
miller()
{
echo "$@" | mlr --c2j --jlistwrap --ofs ':' cat >> $result
}

collect()
{
res=$($sqlite --csv --header $workingDatabase "SELECT $@"); miller "$res"
}

The error-report is

Code: Select all

sudo: /domoticz.db: opdracht niet gevonden
./buildCumulusData.sh: 46: ./buildCumulusData.sh: --csv: not found
mlr: unacceptable empty CSV key at file "(stdin)" line 1.
./buildCumulusData.sh: 46: ./buildCumulusData.sh: --csv: not found
mlr: unacceptable empty CSV key at file "(stdin)" line 1.
.
<repeating same 2 lines till end, and therefore removed in this extract>
.
./buildCumulusData.sh: 46: ./buildCumulusData.sh: --csv: not found
mlr: unacceptable empty CSV key at file "(stdin)" line 1.
rm: kan '/domoticz_2063.db' niet verwijderen: Bestand of map bestaat niet

'Brute/manual' insertion of a copy of domoticz.db makes the bash-script piecewise running, but if the 'automatic, periodic backup' is not working that not of much further use .......
It has looks that no workable copy of domoticz.db becomes present,
but tracing back, the scriptlines related to backup of domoticz.db seem correct.
Anybody an idea?

Question on correctness of the subject scriptline also dropped in dzVents-section of the forum, as more direct 'invitation' for dzVents-experts.
Has resulted in working bash-script.

Toulon7559 · Post by **Toulon7559** » Wednesday 16 February 2022 11:47

The (draft)-batchfile mentioned in the previous message produces a json-file as contained in the spoiler below.
Keep in mind, that the batchfile is aimed at collecting historic data:
the main script reads the actual data from the virtual devices of Domoticz.

Spoiler: show: [
{ "barometer_max": 1003, "Time": "00:00" }
,
{ "barometer_min": 994, "Time": "08:00" }
,
{ "barometer_avg": 997.9 }
,
{ "barometer_mtd_max": 1020, "Date": "2022-02-12" }
,
{ "barometer_mtd_min": 999, "Date": "2022-02-14" }
,
{ "barometer_mtd_avg": 1008.7 }
,
{ "barometer_ytd_max": 1020, "Date": "2022-02-12" }
,
{ "barometer_ytd_min": 999, "Date": "2022-02-14" }
,
{ "barometer_ytd_avg": 1008.7 }
,
{ "barometer_month_min": 999, "Date": "2022-02-14" }
,
{ "barometer_month_max": 1020, "Date": "2022-02-12" }
,
{ "barometer_month_avg": 1008.7 }
,
{ "barometer_year_max": 1020, "Date": "2022-02-12" }
,
{ "barometer_year_min": 999, "Date": "2022-02-14" }
,
{ "barometer_year_avg": 1008.7 }
,
{ "insideTemperature_max": 10.32, "Time": "10:25" }
,
{ "insideTemperature_min": 8.15, "Time": "00:05" }
,
{ "insideTemperature_avg": 9.04 }
,
{ "insideTemperature_mtd_max": 13.55, "Date": "2022-02-02" }
,
{ "insideTemperature_mtd_min": -1.3, "Date": "2022-02-12" }
,
{ "insideTemperature_mtd_avg": 7.23 }
,
{ "insideTemperature_ytd_max": 14.54, "Date": "2022-01-01" }
,
{ "insideTemperature_ytd_min": -1.3, "Date": "2022-02-12" }
,
{ "insideTemperature_ytd_avg": 6.41 }
,
{ "insideTemperature_month_max": 13.55, "Date": "2022-02-02" }
,
{ "insideTemperature_month_min": -1.3, "Date": "2022-02-12" }
,
{ "insideTemperature_month_avg": 6.56 }
,
{ "insideTemperature_year_max": 27.69, "Date": "2021-09-14" }
,
{ "insideTemperature_year_min": -5.72, "Date": "2021-12-26" }
,
{ "insideTemperature_year_avg": -2.06 }
,
{ "outsideTemperature_max": 9.6, "Time": "10:15" }
,
{ "outsideTemperature_min": 7.5, "Time": "00:00" }
,
{ "outsideTemperature_avg": 8.39 }
,
{ "outsideTemperature_mtd_max": 11.4, "Date": "2022-02-14" }
,
{ "outsideTemperature_mtd_min": 0.0, "Date": "2022-02-12" }
,
{ "outsideTemperature_mtd_avg": 6.54 }
,
{ "outsideTemperature_ytd_max": 11.4, "Date": "2022-02-14" }
,
{ "outsideTemperature_ytd_min": 0.0, "Date": "2022-02-12" }
,
{ "outsideTemperature_ytd_avg": 6.54 }
,
{ "outsideTemperature_month_max": 11.4, "Date": "2022-02-14" }
,
{ "outsideTemperature_month_min": 0.0, "Date": "2022-02-12" }
,
{ "outsideTemperature_month_avg": 6.54 }
,
{ "outsideTemperature_year_max": 11.4, "Date": "2022-02-14" }
,
{ "outsideTemperature_year_min": 0.0, "Date": "2022-02-12" }
,
{ "outsideTemperature_year_avg": 3.83 }
,
{ "insideHumidity_max": 100, "Time": "00:00" }
,
{ "insideHumidity_min": 100, "Time": "00:00" }
,
{ "insideHumidity_avg": 100.0 }
,
{ "insideHumidity_mtd_max": 100, "Date": "2022-02-04" }
,
{ "insideHumidity_mtd_min": 68, "Date": "2022-02-13" }
,
{ "insideHumidity_mtd_avg": 90.5 }
,
{ "insideHumidity_ytd_max": 100, "Date": "2022-01-01" }
,
{ "insideHumidity_ytd_min": 68, "Date": "2022-02-13" }
,
{ "insideHumidity_ytd_avg": 95.6 }
,
{ "insideHumidity_month_max": 100, "Date": "2022-01-17" }
,
{ "insideHumidity_month_min": 68, "Date": "2022-02-13" }
,
{ "insideHumidity_month_avg": 93.9 }
,
{ "insideHumidity_year_max": 100, "Date": "2021-10-03" }
,
{ "insideHumidity_year_min": 62, "Date": "2021-09-13" }
,
{ "insideHumidity_year_avg": 62.0, "Date": "2021-09-13" }
,
{ "outsideHumidity_max": 95, "Time": "00:00" }
,
{ "outsideHumidity_min": 94, "Time": "00:40" }
,
{ "outsideHumidity_avg": 94.8 }
,
{ "outsideHumidity_mtd_max": 82, "Date": "2022-02-15" }
,
{ "outsideHumidity_mtd_min": 66, "Date": "2022-02-12" }
,
{ "outsideHumidity_mtd_avg": 73.3 }
,
{ "outsideHumidity_ytd_max": 82, "Date": "2022-02-15" }
,
{ "outsideHumidity_ytd_min": 66, "Date": "2022-02-12" }
,
{ "outsideHumidity_ytd_avg": 73.3 }
,
{ "outsideHumidity_month_max": 82, "Date": "2022-02-15" }
,
{ "outsideHumidity_month_min": 66, "Date": "2022-02-12" }
,
{ "outsideHumidity_month_avg": 73.3 }
,
{ "outsideHumidity_year_max": 82, "Date": "2022-02-15" }
,
{ "outsideHumidity_year_min": 66, "Date": "2022-02-12" }
,
{ "outsideHumidity_year_avg": 66.0, "Date": "2022-02-12" }
,
{ "dewpoint_max": 8.84, "Time": "10:15" }
,
{ "dewpoint_min": 6.75, "Time": "00:00" }
,
{ "dewpoint_avg": 7.6 }
,
{ "dewpoint_mtd_max": 3.53, "Date": "2022-02-14" }
,
{ "dewpoint_mtd_min": -9.2, "Date": "2022-02-12" }
,
{ "dewpoint_mtd_avg": -1.3 }
,
{ "dewpoint_ytd_max": 3.53, "Date": "2022-02-14" }
,
{ "dewpoint_ytd_min": -9.2, "Date": "2022-02-12" }
,
{ "dewpoint_ytd_avg": -1.3 }
,
{ "dewpoint_month_max": 3.53, "Date": "2022-02-14" }
,
{ "dewpoint_month_min": -9.2, "Date": "2022-02-12" }
,
{ "dewpoint_month_avg": -1.3 }
,
{ "dewpoint_year_max": 3.53, "Date": "2022-02-14" }
,
{ "dewpoint_year_min": -9.2, "Date": "2022-02-12" }
,
{ "dewpoint_year_avg": -9.2, "Date": "2022-02-12" }
,
{ "windspeed_max": 40, "Time": "08:45" }
,
{ "windspeed_avg": 13.9 }
,
{ "windspeed_mtd_max": 24, "Date": "2022-02-15" }
,
{ "windspeed_ytd_max": 24, "Date": "2022-02-15" }
,
{ "windchill_min": 6.2, "Time": "00:00" }
,
{ "windchill_mtd_min": -0.7, "Date": "2022-02-13" }
,
{ "windchill_ytd_min": -0.7, "Date": "2022-02-13" }
,
{ "heatindex_max": 31.5, "Time": "00:00" }
,
{ "heatindex_mtd_max": 62.0, "Date": "2022-02-12" }
,
{ "heatindex_ytd_max": 62.0, "Date": "2022-02-12" }
,
{ "rain_mtd_total": 4.88 }
,
{ "rain_ytd_total": 4.88 }
,
{ "rain_month_total": 4.88 }
,
{ "rain_year_total": 4.88 }
,
{ "rainrate_today_max": 75, "Time": "08:10" }
,
{ "rainrate_hour_max": 75, "Time": 8 }
,
{ "rainrate_mtd_max": 249, "Date": 2 }
,
{ "rainrate_ytd_max": 249, "Date": 2022 }
,
{ "rainrate_month_max": 249, "Date": 2 }
,
{ "rainrate_year_max": 249, "Date": 2022 }
,
{ "solarradiation_today_max": 420, "Time": "09:50" }
,
{ "solarradiation_mtd_max": "", "Date": "" }
,
{ "solarradiation_ytd_max": "", "Date": "" }
,
{ "solarradiation_month_max": "", "Date": "" }
,
{ "solarradiation_year_max": "", "Date": "" }
,
{ "uv_today_max": 0.12, "Time": "09:50" }
,
{ "uv_mtd_max": 1.14, "Date": "2022-02-14" }
,
{ "uv_ytd_max": 1.14, "Date": "2022-02-14" }
,
{ "uv_month_max": 1.14, "Date": "2022-02-14" }
,
{ "uv_year_max": 1.14, "Date": "2022-02-14" }
,
{ "wind_maxGust": 64, "Time": "08:45" }
,
{ "uv_radiation_max": "", "Time": "" }
]

Scanning the contents is useful as a check that
# we found all required historic data,
# identifiers and IDXes have been properly inserted referring to virtual devices in Domoticz,
# (as result) which&how fields have been filled in the json-file.
Observations in the JSON-file:
# Windspeed-info limited to actual values, not (historic) extremes and their time of occurrence,
[to be checked whether real wind-values or WS10 and WG10 as Domoticz uses in it's API/JSON]
# Info for rain and rainrate requires check and rescale, because
- Rainrates have to be divided by 100 (due to Domoticz using Rainrate100), and
- Rain-listing is confusing with actual values from other gauges at 35/february and 120/ytd, but the json-file having
{ "rain_mtd_total": 4.88 },
{ "rain_ytd_total": 4.88 },
{ "rain_month_total": 4.88 },
{ "rain_year_total": 4.88 }
# Some unexpected empty fields, and
# Some contents which (at first glance) make no sense, such as { "heatindex_max": 31.5, "Time": "00:00" }
which looks as if result is in Fahrenheit, although formula acc. to Wiki_for_Celsius. Calculation needing scaling-algorithm upfront?
.
=> Info collected which fields have meaningful contents, suitable for transfer to the Cumulus uploadfile, and what is amiss.

Next steps:
a) check identifiers in the batchfile to get proper filling for the JSON-file, comparing with SQL-database-identifiers
b) tuning the dzVents-script for extraction from this JSON-file and for direct read-out from the virtual devices, aiming to filling as much as possible according to the list in the first message of this thread
c) if not all fields can be filled, check whether 'mandatory' or 'nice-to-have', because a slightly limited upload-file might be OK for basic application at the other side of the interface. If required, fill fields with placeholder --- , indicating that no value available.

Toulon7559 · Post by **Toulon7559** » Tuesday 22 February 2022 19:37

After crash-course in dzVents and much try&error, now a working release of a package which compiles a Cumulus-type uploadfile from Domoticz-data.

Can be used by those daring to experiment!
'Full version' not yet ready for full deployment, because of need for final clean-up
(as described under 'Observations' in previous message and under 'Status&ToDo' in this message)!
'Light version' in next message should be OK, because avoiding 'aspects_under_discussion':
different dzVents-script, but same bashfile & User-instructions.

Consists of 3 software-scripts:
a) Main script in dzVents for control and for compilation
b) Bashfile for collection & calculation of historic data, as Annex serving the Main script
c) Auxiliary lua-script to copy the result-file to aother desired location at the host

The scripts are contained in the spoilers below.

Main dzVents-script

Spoiler: show: --[[
-- Line 002
this dzVents-script triggers an offline bashscript to get some meteo data for compilation of an emulated Cumulus file

The bash script will:
Copy the domoticz database to /tmp/cCopy.db as working copy
Collect historic Data
It will then report it's findings back to dzVents, and erase the working copy.

Requires: dzVents > 3.1.0

Tested (the draft-version, by Waaren) for database version 147 = domoticz version V2020.2 (build 12822 - 12850)
Tuned & further tested by Toulon7559. Latest version installed as /home/pi/domoticz/scripts/dzVents/scripts/script_time_CumulusGen_dzVents03.lua

History:
20210113: Waaren, Start coding the dzVents part
20210113: Waaren, Start coding the bash script part
20210117: Waaren, first Alfa version for testing, request for comments
20210607: Toulon7559, second Alfa version for testing
20211228: Toulon7559, third Alfa version for testing
20220222: first public Beta release open for testing and comments

-- Line 025
]]--

local scriptVar = 'buildCumulus'
local checkBashReturn = 'CumulusCheck'

-- Line 031
return {
on = { timer = { "every 10 minutes" }, -- interval for running script
devices = { 'cumulusTrigger'}, -- during test and development [reason for inclusion????]
customEvents = {checkBashReturn}, -- this will be triggered to check if the bash script returned timely
shellCommandResponses = {scriptVar}, -- this will receive the response of the bash script
},
-- Line 038
data = { bashStarted = { initial = 0},
},

--logging = { level = domoticz.LOG_DEBUG, -- switch to LOG_ERROR when all OK
-- marker = scriptVar,
-- },

execute = function(dz, item)

-- Line 048
-- ======================= Settings below this line =================
-- Identify the bashfile by name and location
local bashScript = '/home/pi/domoticz/scripts/dzVents/scripts/buildCumulusData02.sh'

-- this table will be parsed as JSON to the bash script
local tParms =
{
-- Line 056, Controlvars (Insert here the location of files, sources & destinations as valid for your configuration)
callback = scriptVar,
sqlite = '/usr/bin/sqlite3', -- '/usr/local/bin/sqlite3', -- set to location of your sqlite3 binary
targetLocation = '/tmp', -- working directory
domoticzDatabaseDirectory = '/home/pi/domoticz', -- '/opt/domoticz', -- where can the script find your database to copy it ?
output = '/tmp/HWAFile.txt', -- '/tmp/Cumulus.txt' -- file location for cumulus emulation

-- Line 063, Weather devices (IDX: insert deviceIDs as valid for your configuration; same deviceID possible for various attributes of 1 IDX)
wind = 97,
windchill = 97,
gust = 97,
heatindex = 69,
insideHeatIndex = 85, -- Placeholder
rain = 96,
rainrate = 96,
uv = 98,
solar = 99,
outsideHumidity = 95,
insideHumidity = 85, -- Placeholder
dewpoint = 95,
insideDewpoint = 85, -- Placeholder
outsideTemperature = 95,
insideTemperature = 85, -- Placeholder
barometer = 95,
forecast = 74, -- text
}

-- Line 083 ======================= No changes required below this line ==================

local parms = dz.utils.toJSON( tParms) -- Convert Lua table to JSON string
parms = " '" .. parms .. "'" -- sinqle quotes around JSON

local BASH_STARTED = 'Bash Started'
local BASH_RETURNED = 'Bash Returned'

local function getDevices( t)
t.dv = {}
t.dv.wind = dz.devices( t.tParms.wind)
t.dv.windchill = dz.devices( t.tParms.windchill)
t.dv.outsideTemperature = dz.devices( t.tParms.outsideTemperature)
t.dv.insideTemperature = dz.devices( t.tParms.insideTemperature)
t.dv.barometer = dz.devices( t.tParms.barometer)
t.dv.rain = dz.devices( t.tParms.rain)
t.dv.uv = dz.devices( t.tParms.uv)
t.dv.solar = dz.devices( t.tParms.solar)
t.dv.outsideHumidity = dz.devices( t.tParms.outsideHumidity)
t.dv.insideHumidity = dz.devices( t.tParms.insideHumidity)
t.dv.humidity = dz.devices( t.tParms.humidity)
t.dv.dewpoint = dz.devices( t.tParms.dewpoint)
t.dv.insideDewpoint = dz.devices( t.tParms.insideDewpoint)
t.dv.forecast = dz.devices( t.tParms.forecast)
return t
end

local function makeFileTable(t)
t.fileLayout =
{
{'BOF'},
{''},
{'$stationDate = "' .. t['Local Date'] .. '";' },
{'$stationTime = "' .. t['Local time'] .. '";' },
{''},
{'$tempUnit = "°C";' },
{'$humUnit = "%";'},
{'$barUnit = "hPa";'},
{'$rainUnit = "mm";'},
{'$rateUnit = "mm/hr";'},
{'$windUnit = "km/h";' },
{''},
{'$sunriseTime = ' .. '"' .. t.sunrise .. ';"'},
{'$sunsetTime = ' .. '"' .. t.sunset .. ';"'},
{''},
{'outsideTemp','outsideTemperature'},
{'hiOutsideTemp','outsideTemperature_max'},
{'lowOutsideTemp','outsideTemperature_min'},
{''},
{'lowOutsideTempTime','outsideTemperature_max','time'},
{'hiOutsideTempTime','outsideTemperature_min','time'},
{'lowMonthlyOutsideTemp','outsideTemperature_mtd_min'},
{'hiMonthlyOutsideTemp','outsideTemperature_mtd_max'},
{'hiYearlyOutsideTemp','outsideTemperature_ytd_max'},
{'lowYearlyOutsideTemp','outsideTemperature_ytd_min'},
{''},
{'outsideHumidity','outsideHumidity'},
{'lowHumidity','outsideHumidity_min'},
{'hiHumidity','outsideHumidity_max'},
{'lowHumTime','outsideHumidity_min','time'},
{'hiHumTime','outsideHumidity_max','time'},
{'hiMonthlyHumidity','outsideHumidity_mtd_max'},
{'lowMonthlyHumidity','outsideHumidity_mtd_min'},
{'hiYearlyHumidity','outsideHumidity_ytd_max'},
{'lowYearlyHumidity','outsideHumidity_ytd_min'},
{''},
{'outsideDewPt','dewpoint'},
{'hiDewpoint','dewpoint_min'},
{'lowDewpoint','dewpoint_max'},
{'hiDewpointTime','dewpoint_min','time'},
{'lowDewpointTime','dewpoint_max','time'},
{'hiMonthlyDewpoint','dewpoint_mtd_max'},
{'lowMonthlyDewpoint','dewpoint_mtd_min'},
{'hiYearlyDewpoint','dewpoint_ytd_max'},
{'lowYearlyDewpoint','dewpoint_ytd_min'},
{''},
{'windSpeed','windspeed'},
{'wind10Avg','windspeed_avg'},
{'hiWindSpeed','windspeed_max'},
{'hiWindSpeedTime','windspeed_max','time'},
{'hiMonthlyWindSpeed','windspeed_mtd_max'},
{'hiYearlyWindSpeed','windspeed_ytd_max'},
{''},
{'windDir','winddirection_number'},
{'windDirection','winddirection_string'},
{''},
{'windChill','windchill'},
{'lowWindchill','windchill_min'},
{'lowWindchillTime','windchill_min','time'},
{'lowMonthlyWindchill','windchill_mtd_min'},
{'lowYearlyWindchill','windchill_ytd_min'},
{''},
{'outsideHeatIndex','heatindex'},
{'hiHeatindex','heatindex_max'},
{'hiHeatindexTime','heatindex_max','time'},
{'hiMonthlyHeatindex','heatindex_mtd_max'},
{'hiYearlyHeatindex','heatindex_ytd_max'},
{''},
{'barometer','barometer'},
{'barTrend','forecast'}, -- 'barometer_trend'
{'lowBarometer','barometer_min'},
{'hiBarometer','barometer_max'},
{'lowMonthlyBarometer','barometer_mtd_min'},
{'hiMonthlyBarometer','barometer_mtd_max'},
{'lowYearlyBarometer','barometer_ytd_min'},
{'hiYearlyBarometer','barometer_ytd_max'},
{'lowBarometerTime','barometer_min','time'},
{'hiBarometerTime','barometer_max','time'},
{''},
{'dailyRain','rain_today'},
{'$stormRain = "---";'}, --t.rain_storm.value
{'monthlyRain','rain_mtd_total'},
{'totalRain','rain_ytd_total'},
{''},
{'rainRate','rainrate_hour'},
{'hiRainRate','rainrate_today_max'},
{'hiRainRateTime','rainrate_today_max','time'},
{'hiRainRateHour','rainrate_hour_max'},
{'hiMonthlyRainRate','rainrate_mtd_max'},
{'hiYearlyRainRate','rainrate_ytd_max'},
{''},
{'solarRad','solarradiation'},
{'hiSolarRad','solarradiation_today_max'},
{'hiSolarRadTime','solarradiation_today_max','time'},
{'hiMonthlySolarRad','solarradiation_mtd_max'},
{'$hiYearlySolarRad = "---";'}, --t.solarradiation_ytd_max.value
{''},
{'uv','uv'},
{'hiUV','uv_today_max'},
{'hiUVTime','uv_today_max','time'},
{'hiMonthlyUV','uv_mtd_max'},
{'hiYearlyUV','uv_ytd_max'},
{''},
{'insideTemp','insideTemperature'},
{'hiinsideTemp','insideTemperature_max'},
{'lowinsideTemp','insideTemperature_min'},
{'hiinsideTempTime','insideTemperature_max','time'},
{'lowinsideTempTime','insideTemperature_min','time'},
{'hiMonthlyinsideTemp','insideTemperature_mtd_max'},
{'lowMonthlyinsideTemp','insideTemperature_mtd_min'},
{'hiYearlyinsideTemp','insideTemperature_ytd_max'},
{'lowYearlyinsideTemp','insideTemperature_ytd_min'},
{''},
{'insideHumidity','insideHumidity'},
{'lowInsideHumidity','insideHumidity_min'},
{'hiHumidity','insideHumidity_max'},
{'lowInsideHumidityTime','insideHumidity_min','time'},
{'hiInsideHumidityTime','insideHumidity_max','time'},
{'hiMonthlyInsideHumidity','insideHumidity_mtd_max'},
{'lowMonthlyInsideHumidity','insideHumidity_mtd_min'},
{'hiYearlyInsideHumidity','insideHumidity_ytd_max'},
{'lowYearlInsideHumidity','insideHumidity_ytd_min'},
{''},
{'insideDewPt', 'insideDewpoint'},
{''},
{'insideHeatIndex', 'insideHeatindex'},
{''},
{'forecast', 'forecast'},
{''},
{'$cumulusversion = "Domoticz_dzVents, test 03"'},
{''},
{'EOF'},
{''}
}
return t
end

local function processBashResult(result)
tMeteo = {}
local time,date,mKey
for _, item in ipairs(result) do
for key, value in pairs(item) do
if key == 'Date' then
date = value
elseif key == 'Time' then
time = value
else
tMeteo[key] = {}
tMeteo[key].value = value
mKey = key
end
end
tMeteo[mKey].date = date
tMeteo[mKey].time = time

date, time = nil, nil

end

return tMeteo
end

local function notify(message, priority)
dz.log('Function notify: ' .. tostring(message) .. ' Pri ' .. tostring(priority) , dz.LOG_DEBUG)
return
end

local function makeMeteoEntry( t, device, attribute, key, decimals)
t[key] = {}
t[key].value = t.dv[device][attribute]
if decimals and type( t[key].value) == 'number' then
t[key].value = dz.utils.round( t[key].value, decimals )
end
t[key].time = t.dv[device].lastUpdate.rawTime:sub(1,5)
end

local function minutesToTime(minutes)
local h = math.floor(minutes / 60)
local hh = ('0' .. h):sub(-2)
local mm = ('0' .. minutes - (h * 60)):sub(-2)
return hh .. ':' .. mm
end

local function heatindex( T, H) -- in Celsius
local c1 = -8.78469475556
local c2 = 1.61139411
local c3 = 2.33854883889
local c4 = -0.14611605
local c5 = -0.012308094
local c6 = -0.0164248277778
local c7 = 0.002211732
local c8 = 0.00072546
local c9 = -0.000003582

return dz.utils.round( ( c1 + c2*T + c3*H + c4*T*H + c5*T^2 + c6*H^2 + c7*T^2*H + c8*T*H^2 + c9*T^2*H^2) ,2)
end

local function createCalculatedContent( t)
t.sunrise = minutesToTime(dz.time.sunriseInMinutes)
t.sunset = minutesToTime(dz.time.sunsetInMinutes)

t.heatindex = {}
t.heatindex.value = heatindex( t.outsideTemperature.value, t.outsideHumidity.value)

t.insideHeatindex = {}
t.insideHeatindex.value = heatindex( t.insideTemperature.value, t.insideHumidity.value)

t.barometer_trend = {}
t.barometer_trend = ( ( t.barometer_max.value == t.barometer_min.value ) and 'stable' ) or ( ( t.barometer_max.time > t.barometer_min.time ) and 'rising' ) or 'dropping'
return t
end

local function writeFile( t)
local file = assert(io.open( t.tParms.output, "w"))
file:write( t.file)
file:close()
end

local function makeFile ( t)
local CR = '\n'
local CRS = ';\n'

local function format(value)
if type(value) == 'table' then return ('"---";' .. CR ) end -- protection against nil
local value = value or '---'
if tostring(value):match('%.%d+') and tonumber(value) then value = tostring(value):gsub('%.',','):gsub('%s','') end -- convert decimal point to decimal comma
return '"' .. value .. '"' .. ';' .. CR
end

t.file = ''
for idx, record in ipairs(t.fileLayout) do
-- dz.log(idx .. dz.utils.toStr(record))
if not(record[2]) then
t.file = t.file .. record[1] .. CR
else
t.file = t.file .. '$' .. record[1] .. ' = ' .. format ( t[record[2]][record[3] or 'value' ])
end
end
return t
end

local function getNativeContent( t)
t['Local Date'] = dz.time.rawDate
t['Local time'] = dz.time.rawTime
-- device attribute key decimals
makeMeteoEntry( t, 'wind', 'speedMs', 'windspeed',1)
makeMeteoEntry( t, 'wind', 'direction', 'winddirection_number')
makeMeteoEntry( t, 'wind', 'directionString', 'winddirection_string')
makeMeteoEntry( t, 'windchill', 'chill', 'windchill',1)
makeMeteoEntry( t, 'rain', 'rain', 'rain_today',2)
makeMeteoEntry( t, 'rain', 'rainRate', 'rainrate_hour', 2)
makeMeteoEntry( t, 'uv', 'uv', 'uv', 1)
makeMeteoEntry( t, 'solar', 'radiation', 'solarradiation', 1)
makeMeteoEntry( t, 'outsideTemperature', 'temperature', 'outsideTemperature', 2)
makeMeteoEntry( t, 'insideTemperature', 'temperature', 'insideTemperature', 2)
makeMeteoEntry( t, 'barometer', 'barometer', 'barometer', 0)
makeMeteoEntry( t, 'insideHumidity', 'humidity', 'insideHumidity', 0)
makeMeteoEntry( t, 'outsideHumidity', 'humidity', 'outsideHumidity', 0)
makeMeteoEntry( t, 'dewpoint', 'dewPoint', 'dewpoint', 1)
makeMeteoEntry( t, 'insideDewpoint', 'dewPoint', 'insideDewpoint', 1)
makeMeteoEntry( t, 'forecast', 'text', 'forecast' )

return t
end

local function startBash()
dz.executeShellCommand(
{
command = 'sudo ' .. bashScript .. parms ,
callback = scriptVar,
timeout = 120,
})
end

--main code
if item.isTimer or item.isDevice then
dz.data.bashStarted = BASH_STARTED
startBash()
dz.emitEvent(checkBashReturn).afterSec(120) -- schedule control for Bash return in 2 minutes
elseif item.isCustomEvent then
if item.trigger == checkBashReturn then -- control for bash return
if dz.data.bashStarted ~= BASH_RETURNED then
notify(BASH_FAILED)
handleResult(result)
end
return -- No further action needed
else
dz.data.bashStarted = BASH_RETURNED
end

elseif item.isShellCommandResponse then
dz.data.bashStarted = BASH_RETURNED
if item.json then
local tMeteo = processBashResult(item.json)
tMeteo.tParms = tParms -- Add the parms
-- tMeteo = getDevices( tMeteo) -- add the devices
-- tMeteo = getNativeContent( getDevices( tMeteo) ) --
-- tMeteo = createCalculatedContent( getNativeContent( getDevices( tMeteo) ) )
writeFile( makeFile( makeFileTable( createCalculatedContent( getNativeContent( getDevices( tMeteo) ) ) ) ) )
else
notify('Bash script returned with statusCode ' .. item.statusCode)
dz.log('No valid response from Bash script: ' .. item.statusCode, dz.LOG_ERROR)
dz.log(item, dz.LOG_DEBUG)
end

end
end
}

Bashfile

Spoiler: show: #/bin/bash
# Tested installed at/as /home/pi/domoticz/scripts/dzVents/scripts/buildCumulusData02.sh
# Short description
: '
This bash script is part of the mechanism to collect, prepare and send Meteo data to Cumulus.
The script is triggered by a dzVents script and will report back to the same script with the collected information
extracted from the domoticz history tables.

The device_IDXes are send by the dzVents script by means of a JSON string containing type of information to
be collected (eg. wind, rain, etc) and the numeric IDX as key for the sql statement to search in the history
tables.

Programs used in this script:

bash // linux Shell
sqlite3 // SQL (lite) command line interpreter
jq // to interpret the parms send to the script in a JSON string and convert these into shell parms
miller // to convert the comma separated values to an, for dzVents, easy to read JSON string

The script has been tested (by Waaren) on Linux Debian buster and on bullseye, sqlite 3.35.0, jq 1.6 and miller 5.10
Tuning & test by Toulon7559 continued on Raspian_Buster, sqlite, jq and miller in latest releases.

Test the bash-file with below command where the numbers should be adapted to be device_IDX AS APPLICABLE FOR YOUR CONFIGURATION!
File-locations also as applicable for your configuration!!!!

sudo ./buildCumulusData.sh '{"callback":"buildCumulus","dewpoint":17,"domoticzDatabaseDirectory":"/opt/domoticz","gust":11,"insideHumidity":17,"outsideHumidity":17,"barometer":10,"rain":13,"rainrate":13,"uv":794,"sqlite":"/usr/local/bin/sqlite3","targetLocation":"/tmp","outsideTemperature":421,"insideTemperature":10,"uv":12,"wind":11,"windchill":11,"heatindex":17"forecast":778,"insideHeatindez":564 }'

After this test take care that you apply exactly same inserted IDX-values/file-locations for the tParms-table in thedzVents-script!!!!
'

# Use below command to get latest available versions of the required programs on a linux system
#sudo apt install sqlite3 miller jq

#
# Save the parameter send to this script
#
parms=$1

#
# debug
#
echo $parms > parms.debug

#
# convert parm (JSON string) to script variables
#
for parm in $(echo $parms | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]"); do
export $parm
echo $parm >> out.data # debug / development
done

# Line 20
# shorthands
#
result=$targetLocation/Cumulusresult_$$.json
workingDatabase=$targetLocation/domoticz_$$.db

#
# Prepare (nullify) resultfile
#
truncate -s 0 $result
# line 030
#
# Create working copy of the database. Using the .backup command [or backupdatabase.php] to ensure database will not get blocked
#
sudo $sqlite $domoticzDatabaseDirectory/domoticz.db ".backup $workingDatabase"

#
# convert CSV formatted data to json
#
miller()
{
echo "$@" | mlr --c2j --jlistwrap --ofs ':' cat >> $result
}

collect()
{
res=$($sqlite --csv --header $workingDatabase "SELECT $@"); miller "$res"
}

# line 050
#
# SQLite views
#

createHeatIndexViews()
{
#
# Create Heatindex VIEW on table Temperature
#
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS HeatindexShortView;
CREATE VIEW HeatindexShortView (DeviceRowID, Heatindex, Date ) AS
SELECT DeviceRowID,
ROUND(
-8.78469475556 +
1.61139411 * Temperature +
2.33854883889 * Humidity +
-0.14611605 * Temperature * Humidity +
-0.012308094 * Temperature * Temperature +
-0.0164248277778 * Humidity * Humidity +
0.002211732 * Temperature * Temperature * Humidity +
0.00072546 * Temperature * Humidity * Humidity +
-0.000003582 * Temperature * Temperature * Humidity * Humidity
,1), Date
FROM Temperature;")

#
# Create Heatindex VIEW on table Temperature_Calendar
#
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS HeatindexLongView;
CREATE VIEW HeatindexLongView (DeviceRowID, Heatindex, Date ) AS
SELECT DeviceRowID,
ROUND(
-8.78469475556 +
1.61139411 * Temp_AVG +
2.33854883889 * Humidity +
-0.14611605 * Temp_AVG * Humidity +
-0.012308094 * Temp_AVG * Temp_AVG +
-0.0164248277778 * Humidity * Humidity +
0.002211732 * Temp_AVG * Temp_AVG * Humidity +
0.00072546 * Temp_AVG * Humidity * Humidity +
-0.000003582 * Temp_AVG * Temp_AVG * Humidity * Humidity
,1), Date
FROM Temperature_Calendar;")
}

createRainRateViews()
{
#
# Create RainRate view on table rain [division by 100 due to Domoticz using Rainrate100]
#

# Hour
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS RainRateMaxHourView;
CREATE VIEW RainRateMaxHourView (DeviceRowID, Rate, Hour) AS
SELECT DeviceRowID,
MAX(Rate / 100), LTRIM(STRFTIME('%H',Date), '0') FROM Rain WHERE DATE(Date) = DATE('now') GROUP BY DeviceRowID, STRFTIME('%H',Date);")

# MTD
rs=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS RainRateMaxMTDView;
CREATE VIEW RainRateMaxMTDView (DeviceRowID, Rate, Month) AS
SELECT DeviceRowID,
MAX(Rate / 100), LTRIM(STRFTIME('%m',Date), '0') FROM Rain WHERE STRFTIME('%m',Date) = STRFTIME('%m','now') GROUP BY DeviceRowID, STRFTIME('%m',Date);")

# Month
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS RainRateMaxMonthView;
CREATE VIEW RainRateMaxMonthView (DeviceRowID, Rate, Month) AS
SELECT DeviceRowID,
MAX(Rate / 100), LTRIM(STRFTIME('%m',Date), '0') FROM Rain WHERE date > date('now','-1 MONTH') GROUP BY DeviceRowID, STRFTIME('%m',Date);")

#YTD
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS RainRateMaxYTDView;
CREATE VIEW RainRateMaxYTDView (DeviceRowID, Rate, Year) AS
SELECT DeviceRowID,
MAX(Rate / 100), STRFTIME('%Y',Date) FROM Rain WHERE STRFTIME('%Y',Date) = STRFTIME('%Y','now') GROUP BY DeviceRowID, STRFTIME('%Y',Date);")

#Year
res=$($sqlite --csv --header $workingDatabase "DROP VIEW IF EXISTS RainRateMaxYearView;
CREATE VIEW RainRateMaxYearView (DeviceRowID, Rate, Year) AS
SELECT DeviceRowID,
MAX(Rate / 100), STRFTIME('%Y',Date) FROM Rain WHERE date > date('now','-1 MONTH') GROUP BY DeviceRowID, STRFTIME('%Y',Date);")

}

#
# create detail lines in $result file
#
# barometer
if [ ! -z ${barometer+x} ]; then
collect "MAX(Barometer) barometer_max, STRFTIME('%H:%M',Date) Time FROM temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $barometer"
collect "MIN(Barometer) barometer_min, STRFTIME('%H:%M',Date) Time FROM temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $barometer"
collect "ROUND(AVG(Barometer),1) barometer_avg FROM temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $barometer"
collect "MAX(Barometer) barometer_mtd_max, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $barometer"
collect "MIN(Barometer) barometer_mtd_min, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $barometer"
collect "ROUND(AVG(Barometer),1) barometer_mtd_avg FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $barometer"
collect "MAX(Barometer) barometer_ytd_max, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $barometer"
collect "MIN(Barometer) barometer_ytd_min, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $barometer"
collect "ROUND(AVG(Barometer),1) barometer_ytd_avg FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $barometer"
collect "MIN(Barometer) barometer_month_min, Date FROM Temperature_Calendar WHERE date > DATE('now','-1 MONTH') AND DeviceRowID = $barometer"
collect "MAX(Barometer) barometer_month_max, Date FROM Temperature_Calendar WHERE date > DATE('now','-1 MONTH') AND DeviceRowID = $barometer"
collect "ROUND(AVG(Barometer),1) barometer_month_avg FROM Temperature_Calendar WHERE date > DATE('now','-1 MONTH') AND DeviceRowID = $barometer"
collect "MAX(Barometer) barometer_year_max, Date FROM Temperature_Calendar WHERE date > DATE('now','-1 YEAR') AND DeviceRowID = $barometer"
collect "MIN(Barometer) barometer_year_min, Date FROM Temperature_Calendar WHERE date > DATE('now','-1 YEAR') AND DeviceRowID = $barometer"
collect "ROUND(AVG(Barometer),1) barometer_year_avg FROM Temperature_Calendar WHERE date > DATE('now','-1 YEAR') AND DeviceRowID = $barometer"
fi

#temperatures

collectTemperatures()
{
temperature=$1
idx=$2
if [ ! -z ${temperature+x} ]; then
t="${temperature}_max" ; collect "MAX(Temperature) $t, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${temperature}_min" ; collect "MIN(Temperature) $t, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${temperature}_avg" ; collect "ROUND(AVG(Temperature),2) $t FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${temperature}_mtd_max" ; collect "MAX(Temp_Max) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $idx"
t="${temperature}_mtd_min" ; collect "MIN(Temp_Min) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $idx"
t="${temperature}_mtd_avg" ; collect "ROUND(AVG(Temp_Avg),2) $t FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $idx"
t="${temperature}_ytd_max" ; collect "MAX(Temp_Max) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${temperature}_ytd_min" ; collect "MIN(Temp_Min) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${temperature}_ytd_avg" ; collect "ROUND(AVG(Temp_Avg),2) $t FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${temperature}_month_max" ;collect "MAX(Temp_Max) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${temperature}_month_min" ;collect "MIN(Temp_Min) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${temperature}_month_avg" ;collect "ROUND(AVG(Temp_Avg),2) $t FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${temperature}_year_max" ;collect "MAX(Temp_Max) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
t="${temperature}_year_min" ;collect "MIN(Temp_Min) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
t="${temperature}_year_avg" ;collect "ROUND(MIN(Temp_Avg),2) $t FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
fi
}

collectTemperatures insideTemperature $insideTemperature
collectTemperatures outsideTemperature $outsideTemperature

collectHumidities()
{
humidity=$1
idx=$2

#humidities
if [ ! -z ${humidity+x} ]; then
t="${humidity}_max" ; collect "MAX(Humidity) $t, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${humidity}_min" ; collect "MIN(Humidity) $t, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${humidity}_avg" ; collect "ROUND(AVG(Humidity),1) $t FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $idx"
t="${humidity}_mtd_max" ; collect "MAX(Humidity) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $idx"
t="${humidity}_mtd_min" ; collect "MIN(Humidity) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $idx"
t="${humidity}_mtd_avg" ; collect "ROUND(AVG(Humidity),1) $t FROM Temperature_Calendar WHERE STRFTIME('%m',date) = STRFTIME('%m','now') AND DeviceRowID = $idx"
t="${humidity}_ytd_max" ; collect "MAX(Humidity) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${humidity}_ytd_min" ; collect "MIN(Humidity) $t, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${humidity}_ytd_avg" ; collect "ROUND(AVG(Humidity),1) $t FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $idx"
t="${humidity}_month_max" ; collect "MAX(Humidity) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${humidity}_month_min" ; collect "MIN(Humidity) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${humidity}_month_avg" ; collect "ROUND(AVG(Humidity),1) $t FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $idx"
t="${humidity}_year_max" ; collect "MAX(Humidity) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
t="${humidity}_year_min" ; collect "MIN(Humidity) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
t="${humidity}_year_avg" ; collect "ROUND(MIN(Humidity),1) $t, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $idx"
fi
}

collectHumidities insideHumidity $insideHumidity
collectHumidities outsideHumidity $outsideHumidity

#dewpoint
if [ ! -z ${dewpoint+x} ]; then
collect "MAX(dewpoint) dewpoint_max, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $dewpoint"
collect "MIN(dewpoint) dewpoint_min, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $dewpoint"
collect "ROUND(AVG(dewpoint),1) dewpoint_avg FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $dewpoint"
collect "MAX(dewpoint) dewpoint_mtd_max, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $dewpoint"
collect "MIN(dewpoint) dewpoint_mtd_min, Date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $dewpoint"
collect "ROUND(AVG(dewpoint),1) dewpoint_mtd_avg FROM Temperature_Calendar WHERE STRFTIME('%m',date) = STRFTIME('%m','now') AND DeviceRowID = $dewpoint"
collect "MAX(dewpoint) dewpoint_ytd_max, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $dewpoint"
collect "MIN(dewpoint) dewpoint_ytd_min, Date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $dewpoint"
collect "ROUND(AVG(dewpoint),1) dewpoint_ytd_avg FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $dewpoint"
collect "MAX(dewpoint) dewpoint_month_max, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $dewpoint"
collect "MIN(dewpoint) dewpoint_month_min, Date FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $dewpoint"
collect "ROUND(AVG(dewpoint),1) dewpoint_month_avg FROM Temperature_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $dewpoint"
collect "MAX(dewpoint) dewpoint_year_max, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $dewpoint"
collect "MIN(dewpoint) dewpoint_year_min, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $dewpoint"
collect "ROUND(MIN(dewpoint),1) dewpoint_year_avg, Date FROM Temperature_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $dewpoint"
fi

#wind [division by 10 due to Domotiz saving values as WS10 = 10*WS and WG10 = 10*WG, and WS and WG in m/s, while HWA expecting kmh => kmh =3.6* m/s]
if [ ! -z ${wind+x} ]; then
collect "MAX(Speed *3.6/ 10) windspeed_max, STRFTIME('%H:%M',Date) Time FROM wind WHERE DATE(Date) = DATE('now') AND DeviceRowID = $wind"
collect "ROUND(AVG(Speed *3.6/ 10),1) windspeed_avg FROM wind WHERE DATE(Date) = DATE('now') AND DeviceRowID = $wind"
collect "MAX(Speed_Max *3.6/ 10) windspeed_mtd_max, date FROM wind_calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $wind"
collect "MAX(Speed_Max *3.6/ 10) windspeed_ytd_max, date FROM wind_calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $wind"
fi

#windchill
if [ ! -z ${windchill+x} ]; then
collect "MIN(Chill) windchill_min, STRFTIME('%H:%M',Date) Time FROM Temperature WHERE DATE(Date) = DATE('now') AND DeviceRowID = $windchill"
collect "MIN(Chill_Min) windchill_mtd_min, date FROM Temperature_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $windchill"
collect "MIN(Chill_Min) windchill_ytd_min, date FROM Temperature_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $windchill"
fi

#heatindex
if [ ! -z ${heatindex+x} ]; then
createHeatIndexViews # formula is part of the view

collect "ROUND(MAX(Heatindex),1) heatindex_max, STRFTIME('%H:%M',Date) Time FROM HeatindexShortView WHERE DATE(Date) = DATE('now') AND DeviceRowID = $heatindex"
collect "ROUND(MAX(Heatindex),1) heatindex_mtd_max, Date FROM HeatindexLongView WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $heatindex"
collect "ROUND(MAX(Heatindex),1) heatindex_ytd_max, Date FROM HeatindexLongView WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $heatindex"
fi

#rain
if [ ! -z ${rain+x} ]; then
collect "SUM(Total) rain_mtd_total FROM Rain_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $rain"
collect "SUM(Total) rain_ytd_total FROM Rain_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $rain"
collect "SUM(Total) rain_month_total FROM Rain_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $rain"
collect "SUM(Total) rain_year_total FROM Rain_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $rain"
fi

#rainrate
if [ ! -z ${rainrate+x} ]; then
createRainRateViews

collect "MAX(Rate /100) rainrate_today_max, STRFTIME('%H:%M',Date) Time FROM Rain WHERE DATE(Date) = DATE('now') AND DeviceRowID = $rainrate"
collect "MAX(Rate /100) rainrate_hour_max, Hour Time FROM RainRateMaxHourView WHERE DeviceRowID = $rainrate"
collect "MAX(Rate /100) rainrate_mtd_max, Month Date FROM RainRateMaxMTDView WHERE DeviceRowID = $rainrate"
collect "MAX(Rate /100) rainrate_ytd_max, Year Date FROM RainRateMaxYTDView WHERE DeviceRowID = $rainrate"
collect "MAX(Rate /100) rainrate_month_max, Month Date FROM RainRateMaxMonthView WHERE DeviceRowID = $rainrate"
collect "MAX(Rate /100) rainrate_year_max, Year Date FROM RainRateMaxYearView WHERE DeviceRowID = $rainrate"
fi

#uv
if [ ! -z ${solar+x} ]; then
collect "MAX(Value) solarradiation_today_max, STRFTIME('%H:%M',Date) Time FROM Meter WHERE DATE(Date) = DATE('now') AND DeviceRowID = $solar"
collect "MAX(Value) solarradiation_mtd_max, Date FROM Meter_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $solar"
collect "MAX(Value) solarradiation_ytd_max, Date FROM Meter_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $solar"
collect "MAX(Value) solarradiation_month_max, Date FROM Meter_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $solar"
collect "MAX(Value) solarradiation_year_max, Date FROM Meter_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $solar"
fi

#UV
if [ ! -z ${uv+x} ]; then
collect "MAX(Level) uv_today_max, STRFTIME('%H:%M',Date) Time FROM UV WHERE DATE(Date) = DATE('now') AND DeviceRowID = $uv"
collect "MAX(Level) uv_mtd_max, Date FROM UV_Calendar WHERE STRFTIME('%Y%m',date) = STRFTIME('%Y%m','now') AND DeviceRowID = $uv"
collect "MAX(Level) uv_ytd_max, Date FROM UV_Calendar WHERE STRFTIME('%Y',date) = STRFTIME('%Y','now') AND DeviceRowID = $uv"
collect "MAX(Level) uv_month_max, Date FROM UV_Calendar WHERE date > date('now','-1 MONTH') AND DeviceRowID = $uv"
collect "MAX(Level) uv_year_max, Date FROM UV_Calendar WHERE date > date('now','-1 YEAR') AND DeviceRowID = $uv"
fi

collect "MAX(Gust *3.6/10) wind_maxGust, STRFTIME('%H:%M',Date) Time FROM wind WHERE DATE(Date) = DATE('now') AND DeviceRowID = $gust"
collect "ROUND(MAX(value / 10 ),1) uv_radiation_max, STRFTIME('%H:%M',Date) Time FROM Meter WHERE DATE(Date) = DATE('now') AND DeviceRowID = $uv"

#
# Combine detail lines in one JSON and send it to stdout
#
cat $result | sed ':a;N;$!ba;s/\].\[/\,/g'

#
#cleanup
#
rm $workingDatabase
cp $result $targetLocation/Cumulusresult_old.json
rm $result

Auxiliary lua-script

Spoiler: show: -- Script_time lua-file to copy HWAFile from /tmp/ to /home/pi/
commandArray = {}
print ('Start of script 00 HWAFileCopy')
-- next line is required to set permissions for open&copy&shift&execution of the file
os.execute("sudo chmod a+rwx /tmp/HWAFile.txt")
-- next line copies file to /home/pi/ and sets permissions allowing for further processing
os.execute("sudo cp /tmp/HWAFile.txt /home/pi/HWAFile.txt")
os.execute("sudo chmod a+rwx /home/pi/HWAFile.txt")

print ('End of script 00 HWAFileCopy')

return commandArray

Status & ToDo:

1) Both the dzVents-script and the bash-file have an introductory description explaining 'who, when, what & where', not 'why'.
Also some instructions included how to install & setup & sync for each part. Obviously, the lua-script must be matched to the dzVents-script.
The UserInstructions in the spoiler below provide more info on background & operation.

Spoiler: show: 1. Survey
The required package comprises 3 elements:
- dzVents-script as main Controller & Compiler
- bashfile to collect & compile required historic data from the database domoticz.db
- upload-script to put the resulting file in position for pickup as HWA-file by the HWA-server
[The presently included lua-script is a minimal implementation which only copies the result-file from the folder /tmp/ and puts in 'more public' area of the folders.
At same time the filename can be adapted. To be replaced by upload-function inside the dzVents-script]

2. Flow description

The main steps in the functional flow are:
a) timing by the dzVents-script. Now set for run every 10 minutes.
b) pickup of historic data by the bashfile and generation of output as JSON-file
c) processing of data by the dzVents-script
- reception of the JSON-file from the bashfile
- readout of virtual devices for actual data
- processing of data for scaling and for more required derivative data etc.
- merging & compilation of data into the 'Cumulus-list'
- translation of the 'Cumulus-list' into a resulting file which is a txt-type file
d) processing for upload of the resulting file

3. Description of elements

The dzVents-script triggers an offline bashscript to get some meteo data for compilation of an emulated Cumulus file

The bash script will:
Copy the domoticz database to /tmp/cCopy.db as working copy
Collect historic Data
It will then report it's findings back to dzVents as JSON-file, and will erase the working copy.

The dzvents-script subsequently combines the historic data and the actual data, adds derived data and merges & compiles these data into the 'Cumulus-list'.
Ultimately saves the 'Cumulus-list' at temporary location.

The upload-script changes the attributes of the 'Cumulus-list' to allow further processing & transfer, reads the 'Cumulus-list', renames as required,
and uploads to designated location for pickup by the HWA-server (or any other designated user).

4. User instructions

a) Make backup of the extracted files and of working copies
b) Before any action for tuning read this instruction and read the scriptfiles (especially the introductions).
c) Collect the following information, because next needed to tune the table in the dzVents-script and the test-CLI-line in the bashfile and for setup of the upload-script:
[The contents of the fill-in consist of examples for a Raspberry with Domoticz: important to check what is VALID FOR YOUR CONFIGURATION!!!!!]
sqlite = '/usr/bin/sqlite3' => set to location of your sqlite3 binary
targetLocation = '/tmp' => working directory to which the database is copied
domoticzDatabaseDirectory => '/home/pi/domoticz' => where can the bashfile-script find your database to copy it ?
output = '/tmp/Cumulus.txt' => file location for result-file from dzVents-script for cumulus-list emulation
upload-url = https://<www.yourhost.nl>/hwa-txt => url/webspace in domain <www.yourhost.nl> to which the result-file is uploaded for subsequent pick-up by the HWA-server
IDXes of Weather devices => insert deviceIDs as valid for your configuration; same deviceID possible for various attributes of 1 IDX)
d) Insert the above information into the working copies of the dzVents-script, into the bashfile and into the upload-script
e) Make backup of the working copies
f) Install the working copy of the bashfile at destination, and make test-run as described inside that batch-file.
g) Install the working copy of the dzVents-script at destination, and check that resultfile appears at designated location (= upload-url).
h) Install the working copy of the upload-script at destination, and check that the uploaded file appears and can be read (by calling the upload-url)
i) Apply at www.hetweeractueel.nl for membership of the forum => accountname
j) When your station-setup has been running for at least 2 months without problems,
apply for inclusion of your station in the stationlist of HWA, for which you require a working upload-url & accountname.

2) The main dzVents-script as reminder shows a number of (now commented) settings applied by waaren during his testing, replaced by settings related to my configuration.
The present version of the script is doing the job, producing an output file called HWAFile.txt, but needs improvement, because
- Logging is not yet working, reporting error.
- Some output data needs more checking, because has 'questionable' values as described in previous message, probably due to wrong scaling.
- Upload-function to internet-destination not yet incorporated, but testingly file-transfer performed by the auxiliary lua-script towards a 'local' destination.
See 4) for further information.
- Information on inside-sensors (although effective & correct) is 'placeholder'.
For shorter version of the output-file, the related lines of the table might be deleted/outcommented.
- This is complete list 'as-if-Cumulus-output', but for application as basic uploadfile to the server of HetWeerActueel a condensed version is desirable => compilation of alternative/second version with short result-file [done: see next message].

3) The bash-file is the collector & compiler for historic data from the database.
Incorporated comment-description tells what extra software is needed to make the file work, and how to test-standalone the bashfile:
for test-standalone take care that you have correct IDXes and correct file-locations, synced with the table in the dzVents-script!!!!

4) The auxiliary lua-script as placeholder extracts the result-file from folder /tmp/ and puts copy in /home/pi/
Desirable would be
- that this function is integrated into the main dzVents-script and
- that destination of the copy at remote webserver can be part of setup-table of the dzVents-script.

Especially for items 2) and 4) assistance for clean-up will be much appreciated.

Toulon7559 · Post by **Toulon7559** » Friday 25 February 2022 4:35

It is not mandatory to use the 'full version' of the HWA-file to transfer meteo-data from a weatherstation to the database of HetWeerActueel.
Convenient to avoid the presently occurring errors for HeatIndex-info etc., the lightweight version now ready-to-go as tool to feed the upload to HetWeerActueel.
The dzVents-script for 'HWALight' is using the same bash-file and UserInstructions.
The dzVents-script producing a shorter HWA-file is below in the spoiler:
only deviation relative to the 'full-version' is the shorter template list in the script.
Obviously you have to adjust your file-uploader to get the shorter HWA-file (called HWALight.txt) to it's destination.

Spoiler: show: --[[
-- Line 002
this dzVents-script triggers an offline bashscript to get some meteo data for compilation of an emulated Cumulus file

The bash script will:
Copy the domoticz database to /tmp/cCopy.db as working copy
Collect historic Data
It will then report it's findings back to dzVents, and erase the working copy.

Requires: dzVents > 3.1.0

Tested (the draft-version, by Waaren) for database version 147 = domoticz version V2020.2 (build 12822 - 12850)
Tuned & further tested by Toulon7559. Latest version installed as /home/pi/domoticz/scripts/dzVents/scripts/script_time_CumulusGen_dzVents04.lua
Version with lightweight-list installed as /home/pi/domoticz/scripts/dzVents/scripts/script_time_HWALightGen_dzVents04.lua

History:
20210113: Waaren, Start coding the dzVents part
20210113: Waaren, Start coding the bash script part
20210117: Waaren, first Alfa version for testing, request for comments
20210607: Toulon7559, second Alfa version for testing
20211228: Toulon7559, third Alfa version for testing
20220222: first public Beta release for testing and comments
20220225: Beta release for HWALight

-- Line 025
]]--

local scriptVar = 'buildCumulus'
local checkBashReturn = 'CumulusCheck'

-- Line 031
return {
on = { timer = { "every 10 minutes" }, -- interval for running script
devices = { 'cumulusTrigger'}, -- during test and development [reason for inclusion????]
customEvents = {checkBashReturn}, -- this will be triggered to check if the bash script returned timely
shellCommandResponses = {scriptVar}, -- this will receive the response of the bash script
},
-- Line 038
data = { bashStarted = { initial = 0},
},

--logging = { level = domoticz.LOG_DEBUG, -- switch to LOG_ERROR when all OK
-- marker = scriptVar,
-- },

execute = function(dz, item)

-- Line 048
-- ======================= Settings below this line =================
-- Identify the bashfile by name and location
local bashScript = '/home/pi/domoticz/scripts/dzVents/scripts/buildCumulusData02.sh'

-- this table will be parsed as JSON to the bash script
local tParms =
{
-- Line 056, Controlvars (Insert here the location of files, sources & destinations as valid for your configuration)
callback = scriptVar,
sqlite = '/usr/bin/sqlite3', -- '/usr/local/bin/sqlite3', -- set to location of your sqlite3 binary
targetLocation = '/tmp', -- working directory
domoticzDatabaseDirectory = '/home/pi/domoticz', -- '/opt/domoticz', -- where can the script find your database to copy it ?
output = '/tmp/HWALight.txt', -- '/tmp/Cumulus.txt', -- file location & name for emulation of 'Cumulus-list'

-- Line 063, Weather devices (IDX: insert deviceIDs as valid for your configuration; same deviceID possible for various attributes of 1 IDX)
wind = 97, -- WF_IDX = 97 / WU_IDX = 70
windchill = 97, -- WF_IDX = 97 / WU_IDX = 70
gust = 97, -- WF_IDX = 97 / WU_IDX = 70
heatindex = 69,
insideHeatIndex = 85,
rain = 96, -- WF_IDX = 96 / WU_IDX = 72
rainrate = 96, -- WF_IDX = 96 / WU_IDX = 72
uv = 98, -- WF_IDX = 98 / WU_IDX = 71
solar = 99, -- WF_IDX = 99 / WU_IDX = 73
outsideHumidity = 95, -- WF_IDX = 95 / WU_IDX = 69
insideHumidity = 85,
dewpoint = 95, -- WF_IDX = 95 / WU_IDX = 69
insideDewpoint = 85,
outsideTemperature = 95, -- WF_IDX = 95 / WU_IDX = 69
insideTemperature = 85,
barometer = 95, -- WF_IDX = 95 / WU_IDX = 69
forecast = 74, -- text
}

-- Line 083 ======================= No changes required below this line ==================

local parms = dz.utils.toJSON( tParms) -- Convert Lua table to JSON string
parms = " '" .. parms .. "'" -- sinqle quotes around JSON

local BASH_STARTED = 'Bash Started'
local BASH_RETURNED = 'Bash Returned'

local function getDevices( t)
t.dv = {}
t.dv.wind = dz.devices( t.tParms.wind)
t.dv.windchill = dz.devices( t.tParms.windchill)
t.dv.outsideTemperature = dz.devices( t.tParms.outsideTemperature)
t.dv.insideTemperature = dz.devices( t.tParms.insideTemperature)
t.dv.barometer = dz.devices( t.tParms.barometer)
t.dv.rain = dz.devices( t.tParms.rain)
t.dv.uv = dz.devices( t.tParms.uv)
t.dv.solar = dz.devices( t.tParms.solar)
t.dv.outsideHumidity = dz.devices( t.tParms.outsideHumidity)
t.dv.insideHumidity = dz.devices( t.tParms.insideHumidity)
t.dv.humidity = dz.devices( t.tParms.humidity)
t.dv.dewpoint = dz.devices( t.tParms.dewpoint)
t.dv.insideDewpoint = dz.devices( t.tParms.insideDewpoint)
t.dv.forecast = dz.devices( t.tParms.forecast)
return t
end

local function makeFileTable(t)
t.fileLayout =
{
{'BOF'},
{''},
{'$stationDate = "' .. t['Local Date'] .. '";' },
{'$stationTime = "' .. t['Local time'] .. '";' },
{''},
{'$tempUnit = "°C";' },
{'$humUnit = "%";'},
{'$barUnit = "hPa";'},
{'$rainUnit = "mm";'},
{'$rateUnit = "mm/hr";'},
{'$windUnit = "km/h";' },
{''},
{'$sunriseTime = ' .. '"' .. t.sunrise .. ';"'},
{'$sunsetTime = ' .. '"' .. t.sunset .. ';"'},
{''},
{'outsideTemp','outsideTemperature'},
{'hiOutsideTemp','outsideTemperature_max'},
{'lowOutsideTemp','outsideTemperature_min'},
{'lowOutsideTempTime','outsideTemperature_max','time'},
{'hiOutsideTempTime','outsideTemperature_min','time'},
{''},
{'outsideHumidity','outsideHumidity'},
{'lowHumidity','outsideHumidity_min'},
{'hiHumidity','outsideHumidity_max'},
{'lowHumTime','outsideHumidity_min','time'},
{'hiHumTime','outsideHumidity_max','time'},
{''},
{'outsideDewPt','dewpoint'},
{'hiDewpoint','dewpoint_min'},
{'lowDewpoint','dewpoint_max'},
{'hiDewpointTime','dewpoint_min','time'},
{'lowDewpointTime','dewpoint_max','time'},
{''},
{'windSpeed','windspeed'},
{'wind10Avg','windspeed_avg'},
{'hiWindSpeed','windspeed_max'},
{'hiWindSpeedTime','windspeed_max','time'},
{''},
{'windDir','winddirection_number'},
{'windDirection','winddirection_string'},
{''},
{'windChill','windchill'},
{''},
{'$outsideHeatIndex = "---";'}, --','heatindex'},
{''},
{'barometer','barometer'},
{'barTrend','forecast'}, -- 'barometer_trend'
{''},
{'dailyRain','rain_today'},
{'$stormRain = "---";'}, --t.rain_storm.value
{'monthlyRain','rain_mtd_total'},
{'totalRain','rain_ytd_total'},
{''},
{'rainRate','rainrate_hour'},
{''},
{'solarRad','solarradiation'},
{'hiSolarRad','solarradiation_today_max'},
{'hiSolarRadTime','solarradiation_today_max','time'},
{''},
{'uv','uv'},
{'hiUV','uv_today_max'},
{'hiUVTime','uv_today_max','time'},
{''},
{'$softwareversion = "Domoticz_dzVents, test 04 [HWALight]"'},
{''},
{'EOF'},
{''}
}
return t
end

local function processBashResult(result)
tMeteo = {}
local time,date,mKey
for _, item in ipairs(result) do
for key, value in pairs(item) do
if key == 'Date' then
date = value
elseif key == 'Time' then
time = value
else
tMeteo[key] = {}
tMeteo[key].value = value
mKey = key
end
end
tMeteo[mKey].date = date
tMeteo[mKey].time = time

date, time = nil, nil

end

return tMeteo
end

local function notify(message, priority)
dz.log('Function notify: ' .. tostring(message) .. ' Pri ' .. tostring(priority) , dz.LOG_DEBUG)
return
end

local function makeMeteoEntry( t, device, attribute, key, decimals)
t[key] = {}
t[key].value = t.dv[device][attribute]
if decimals and type( t[key].value) == 'number' then
t[key].value = dz.utils.round( t[key].value, decimals )
end
t[key].time = t.dv[device].lastUpdate.rawTime:sub(1,5)
end

local function minutesToTime(minutes)
local h = math.floor(minutes / 60)
local hh = ('0' .. h):sub(-2)
local mm = ('0' .. minutes - (h * 60)):sub(-2)
return hh .. ':' .. mm
end

local function heatindex( T, H) -- in Celsius, acc. to https://en.wikipedia.org/wiki/Heat_index
local c1 = -8.78469475556
local c2 = 1.61139411
local c3 = 2.33854883889
local c4 = -0.14611605
local c5 = -0.012308094
local c6 = -0.0164248277778
local c7 = 0.002211732
local c8 = 0.00072546
local c9 = -0.000003582

return dz.utils.round( c1 + c2*T + c3*H + c4*T*H + c5*T^2 + c6*H^2 + c7*T^2*H + c8*T*H^2 + c9*T^2*H^2 ,2)
end

local function createCalculatedContent( t)
t.sunrise = minutesToTime(dz.time.sunriseInMinutes)
t.sunset = minutesToTime(dz.time.sunsetInMinutes)

t.heatindex = {}
t.heatindex.value = heatindex( t.outsideTemperature.value, t.outsideHumidity.value)

t.insideHeatindex = {}
t.insideHeatindex.value = heatindex( t.insideTemperature.value, t.insideHumidity.value)

t.barometer_trend = {}
t.barometer_trend = ( ( t.barometer_max.value == t.barometer_min.value ) and 'stable' ) or ( ( t.barometer_max.time > t.barometer_min.time ) and 'rising' ) or 'dropping'
return t
end

local function writeFile( t)
local file = assert(io.open( t.tParms.output, "w"))
file:write( t.file)
file:close()
end

local function makeFile ( t)
local CR = '\n'
local CRS = ';\n'

local function format(value)
if type(value) == 'table' then return ('"---";' .. CR ) end -- protection against nil
local value = value or '---'
if tostring(value):match('%.%d+') and tonumber(value) then value = tostring(value):gsub('%.',','):gsub('%s','') end -- convert decimal point to decimal comma
return '"' .. value .. '"' .. ';' .. CR
end

t.file = ''
for idx, record in ipairs(t.fileLayout) do
-- dz.log(idx .. dz.utils.toStr(record))
if not(record[2]) then
t.file = t.file .. record[1] .. CR
else
t.file = t.file .. '$' .. record[1] .. ' = ' .. format ( t[record[2]][record[3] or 'value' ])
end
end
return t
end

local function getNativeContent( t)
t['Local Date'] = dz.time.rawDate
t['Local time'] = dz.time.rawTime
-- device attribute key decimals
makeMeteoEntry( t, 'wind', 'speedMs', 'windspeed',1)
makeMeteoEntry( t, 'wind', 'direction', 'winddirection_number')
makeMeteoEntry( t, 'wind', 'directionString', 'winddirection_string')
makeMeteoEntry( t, 'windchill', 'chill', 'windchill',1)
makeMeteoEntry( t, 'rain', 'rain', 'rain_today',2)
makeMeteoEntry( t, 'rain', 'rainRate', 'rainrate_hour', 2)
makeMeteoEntry( t, 'uv', 'uv', 'uv', 1)
makeMeteoEntry( t, 'solar', 'radiation', 'solarradiation', 1)
makeMeteoEntry( t, 'outsideTemperature', 'temperature', 'outsideTemperature', 2)
makeMeteoEntry( t, 'insideTemperature', 'temperature', 'insideTemperature', 2)
makeMeteoEntry( t, 'barometer', 'barometer', 'barometer', 0)
makeMeteoEntry( t, 'insideHumidity', 'humidity', 'insideHumidity', 0)
makeMeteoEntry( t, 'outsideHumidity', 'humidity', 'outsideHumidity', 0)
makeMeteoEntry( t, 'dewpoint', 'dewPoint', 'dewpoint', 1)
makeMeteoEntry( t, 'insideDewpoint', 'dewPoint', 'insideDewpoint', 1)
makeMeteoEntry( t, 'forecast', 'text', 'forecast' )

return t
end

local function startBash()
dz.executeShellCommand(
{
command = 'sudo ' .. bashScript .. parms ,
callback = scriptVar,
timeout = 120,
})
end

--main code
if item.isTimer or item.isDevice then
dz.data.bashStarted = BASH_STARTED
startBash()
dz.emitEvent(checkBashReturn).afterSec(120) -- schedule control for Bash return in 2 minutes
elseif item.isCustomEvent then
if item.trigger == checkBashReturn then -- control for bash return
if dz.data.bashStarted ~= BASH_RETURNED then
notify(BASH_FAILED)
handleResult(result)
end
return -- No further action needed
else
dz.data.bashStarted = BASH_RETURNED
end

elseif item.isShellCommandResponse then
dz.data.bashStarted = BASH_RETURNED
if item.json then
local tMeteo = processBashResult(item.json)
tMeteo.tParms = tParms -- Add the parms
-- tMeteo = getDevices( tMeteo) -- add the devices
-- tMeteo = getNativeContent( getDevices( tMeteo) ) --
-- tMeteo = createCalculatedContent( getNativeContent( getDevices( tMeteo) ) )
writeFile( makeFile( makeFileTable( createCalculatedContent( getNativeContent( getDevices( tMeteo) ) ) ) ) )
else
notify('Bash script returned with statusCode ' .. item.statusCode)
dz.log('No valid response from Bash script: ' .. item.statusCode, dz.LOG_ERROR)
dz.log(item, dz.LOG_DEBUG)
end

end
end
}

Toulon7559 · Post by **Toulon7559** » Tuesday 01 March 2022 23:42

As described, the script-set misses a good upload-function to put the resulting file in designated webspace:
the presented lua-script only is a surrogate to copy the upload-file to a better accessible local folder.
Have effective python-scripts for upload to remote webspace, but application would result in 'strange' mix of software-scripts:
integration/extension within the present dzVents-script to be preferred.

Application of HTTP_PUT or HTTP_POST seems viable as hinted in Wiki
Learning-by-example is fastest:
somebody seen a dzVents-script segment uploading with that technique a file to webspace @ internet?

Meteo data in Cumulus uploadfile

Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile

Re: Meteo data in Cumulus uploadfile [uploading in dzVents]

Who is online