Extraction of meteo-data from CWOP-table

Moderator: leecollings

Post Reply
Toulon7559
Posts: 848
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: mixed
Location: Hengelo(Ov)/NL
Contact:

Extraction of meteo-data from CWOP-table

Post by Toulon7559 »

CWOP is a rather 'legacy' meteo-service, just concentrating on the basic meteo-values:
however, it still acts as a worldwide 'feeder' for NWS and NOOA using APRS.
They also redistribute the received data: the 'raw data-table' below is an example (as shown for CWOP-station EW4337).
For some simple meteo-applications that contains sufficient information, so why spending a lot of effort for calling the 'more luxury' servers?
Getting such CWOP-table is simplicity itself, compared to more 'protected' setups like for WUnderground and WOW_UK:
you just need to know the stationcallsign of 2 letters plus 4 digits, and the 2 url-calls mentioned below will provide the data-file.
Example for raw APRS-data: https://www.findu.com/cgi-bin/raw.cgi?call=EW4337
Example for raw Weather-data: https://www.findu.com/cgi-bin/rawwx.cgi?call=EW4337
Extraction of data from the datafile also seems simple, because all values always at same position in the 'basic' dataline and with same format.
To be aware that date&time-value is condensed pack with Day+UTC/Zulu-time and, units of values are 'imperial/american', and therefore for applications with 'Local-time' and SI-units a conversion is needed after extraction.
This document in section 6b explains the setup of the datalines.
.
Who has a script-idea suitable to grab & dissect the actual data of a CWOP-station,
as available in the bottom data-line of the APRS/Weather-data (which holds the most recent data)?

;-) A PHP-script for grab & dissect on the APRS/Weather-data would be ideal,
because it could be put in webspace as a support plugin commmonly serving various users ....
Attachments
CWOP APRS-list
CWOP APRS-list
2022-05-25 (1).png (217.32 KiB) Viewed 1902 times
Last edited by Toulon7559 on Saturday 04 June 2022 17:33, edited 15 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 848
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: mixed
Location: Hengelo(Ov)/NL
Contact:

Re: Extraction of meteo-data from CWOP-table

Post by Toulon7559 »

Direct url-call by browser without any problem yields the table in previous message.
.
As next step applied the below scriptlines as function to capture the contents of an url;
working in other lua-files without problems for urls related to json-files en xml-files.

Code: Select all

function os.capture(cmd, raw)
    local f = assert(io.popen(cmd, 'r'))
    local s = assert(f:read('*a'))
    f:close()
    if raw then return s end
    s = string.gsub(s, '^%s+', '')
    s = string.gsub(s, '%s+$', '')
    s = string.gsub(s, '[\n\r]+', ' ')
    return s
end
Subsequent simple implementation of this function as part of lua-file for grab & print.

Code: Select all

    result=os.capture('curl -s "https://www.findu.com/cgi-bin/rawwx.cgi?call='..CWOP_System_ID..'"')
    if (DEBUG == 1) then 
        print('Table-File CWOP_Weather')  
        print(result)
    end
But such setup produces a print of an apparently empty file ...........
.
Digging a bit deeper (in the source-info of the browser-picture) detect that the output of the url-call looks like a HTML-file,
unexpectedly turning this exercise into webscraping-effort, instead of just reading&storing the datafile after an url-call.
Spoiler: show
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd"><HTML>
<HEAD>
<meta http-equiv="expires" content="-1">
<meta http-equiv="pragma" content="no-cache">
<TITLE>Raw Weather for EW0725</TITLE>
</HEAD>
<BODY alink="#008000" bgcolor="#F5F5DC" link="#0000FF" vlink="#000080">
EW0725>APRS,TCPXX*,qAX,CWOP-6:@290650z5217.78N/00648.33E_345/001g004t051r000p001P001b10149h86L201.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@290700z5217.78N/00648.33E_099/000g001t051r000p001P001b10149h85L212.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@290710z5217.78N/00648.33E_276/001g004t051r000p001P001b10149h83L220.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@290720z5217.78N/00648.33E_073/000g003t052r000p001P001b10149h82L234.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@290730z5217.78N/00648.33E_107/000g003t052r000p001P001b10148h82L239.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@290740z5217.78N/00648.33E_085/000g003t053r000p001P001b10147h79L192.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@290750z5217.78N/00648.33E_167/000g003t052r000p001P001b10147h78L116.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@290800z5217.78N/00648.33E_080/000g003t052r000p001P001b10147h80L148.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@290810z5217.78N/00648.33E_094/000g003t052r000p001P001b10147h82L144.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@290820z5217.78N/00648.33E_260/000g003t052r000p001P001b10147h83L089.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@290830z5217.78N/00648.33E_223/000g003t051r002p003P003b10148h86L058.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@290840z5217.78N/00648.33E_063/000g001t051r002p003P003b10149h88L148.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@290850z5217.78N/00648.33E_056/001g006t050r002p003P003b10149h87L291.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@290900z5217.78N/00648.33E_073/000g001t051r002p003P003b10149h87L349.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@290910z5217.78N/00648.33E_022/000g002t052r002p003P003b10148h84L423.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@290920z5217.78N/00648.33E_170/000g003t053r002p003P003b10148h81L279.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@290930z5217.78N/00648.33E_023/000g004t053r000p003P003b10147h81L305.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@290940z5217.78N/00648.33E_043/001g003t054r000p003P003b10147h80L461.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@290950z5217.78N/00648.33E_278/001g005t054r000p003P003b10147h76L489.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291000z5217.78N/00648.33E_271/001g005t054r000p003P003b10147h73L336.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291010z5217.78N/00648.33E_252/002g005t054r000p003P003b10146h73L416.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291020z5217.78N/00648.33E_339/001g006t054r000p003P003b10146h74L172.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291030z5217.78N/00648.33E_131/003g003t...r000p003P003b.....h...weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@291040z5217.78N/00648.33E_076/000g002t053r000p003P003b10146h78L206.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291050z5217.78N/00648.33E_013/001g004t053r000p003P003b10146h77L342.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291100z5217.78N/00648.33E_261/001g004t054r000p003P003b10146h74L576.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291110z5217.78N/00648.33E_084/001g004t055r000p003P003b10145h71L648.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291120z5217.78N/00648.33E_062/001g006t057r000p003P003b10143h69L490.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@291130z5217.78N/00648.33E_050/001g004t057r000p003P003b10143h70L251.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291140z5217.78N/00648.33E_167/001g004t056r000p003P003b10142h70L391.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291150z5217.78N/00648.33E_104/001g003t056r000p003P003b10142h69L494.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@291200z5217.78N/00648.33E_036/001g007t057r000p003P003b10141h69L458.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291210z5217.78N/00648.33E_114/001g004t057r000p003P003b10140h67L537.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291220z5217.78N/00648.33E_335/001g005t057r000p003P003b10139h67L511.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291230z5217.78N/00648.33E_044/001g007t057r000p003P003b10138h68L617.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291240z5217.78N/00648.33E_030/000g004t058r000p003P003b10138h67L447.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291250z5217.78N/00648.33E_111/001g004t057r000p003P003b10137h68L263.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291300z5217.78N/00648.33E_086/001g006t057r000p003P003b10138h70L258.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291310z5217.78N/00648.33E_262/001g005t056r000p003P003b10138h67L295.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291320z5217.78N/00648.33E_178/001g003t056r000p003P003b10137h66L359.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291330z5217.78N/00648.33E_187/001g003t056r000p003P003b10137h67L353.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291340z5217.78N/00648.33E_271/002g005t056r000p003P003b10137h67L382.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291350z5217.78N/00648.33E_046/001g004t056r000p003P003b10137h69L239.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291400z5217.78N/00648.33E_253/001g004t056r000p003P003b10137h68L242.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291410z5217.78N/00648.33E_267/001g004t056r000p003P003b10137h67L313.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291420z5217.78N/00648.33E_175/001g003t056r000p003P003b10137h70L343.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291430z5217.78N/00648.33E_227/001g004t056r000p003P003b10137h70L258.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291440z5217.78N/00648.33E_202/001g004t055r000p003P003b10136h70L284.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291450z5217.78N/00648.33E_246/000g003t055r000p003P003b10136h69L334.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291500z5217.78N/00648.33E_270/001g004t056r000p003P003b10135h68L333.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@291510z5217.78N/00648.33E_245/001g004t056r000p003P003b10136h68L233.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291520z5217.78N/00648.33E_256/001g004t056r000p003P003b10136h69L145.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@291530z5217.78N/00648.33E_236/000g003t055r000p003P003b10136h71L274.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291540z5217.78N/00648.33E_153/001g005t055r000p003P003b10136h70L236.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291550z5217.78N/00648.33E_211/000g002t055r000p003P003b10135h69L155.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291600z5217.78N/00648.33E_253/001g004t055r000p003P003b10135h71L101.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291610z5217.78N/00648.33E_281/000g003t054r000p003P003b10134h72L106.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291620z5217.78N/00648.33E_264/000g003t054r000p003P003b10134h72L114.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291630z5217.78N/00648.33E_281/000g004t054r000p003P003b10133h73L148.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291640z5217.78N/00648.33E_246/000g003t054r000p003P003b10133h73L113.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291650z5217.78N/00648.33E_069/001g005t054r000p003P003b10134h72L035.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291700z5217.78N/00648.33E_134/000g003t053r000p003P003b10134h75L041.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291710z5217.78N/00648.33E_071/001g004t052r000p003P003b10134h76L065.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291720z5217.78N/00648.33E_052/000g002t052r000p003P003b10134h76L066.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291730z5217.78N/00648.33E_084/000g002t052r000p003P003b10134h75L080.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291740z5217.78N/00648.33E_069/000g001t052r000p003P003b10134h76L077.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291750z5217.78N/00648.33E_102/000g001t052r000p003P003b10134h74L065.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291800z5217.78N/00648.33E_063/001g003t053r000p003P003b10134h73L057.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291810z5217.78N/00648.33E_051/000g003t052r000p003P003b10134h74L043.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291820z5217.78N/00648.33E_041/000g002t052r000p003P003b10134h74L030.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291830z5217.78N/00648.33E_038/001g004t052r000p003P003b10134h73L017.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@291840z5217.78N/00648.33E_032/000g003t052r000p003P003b10135h74L012.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291850z5217.78N/00648.33E_062/000g003t051r000p003P003b10135h73L014.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291900z5217.78N/00648.33E_.../000g000t051r000p003P003b10134h74L014.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291910z5217.78N/00648.33E_232/000g002t051r000p003P003b10135h74L010.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@291920z5217.78N/00648.33E_060/000g002t051r000p003P003b10135h75L006.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@291930z5217.78N/00648.33E_.../000g000t051r000p003P003b10135h76L005.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@291940z5217.78N/00648.33E_.../000g000t050r000p003P003b10135h77L002.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@291950z5217.78N/00648.33E_056/000g001t050r000p003P003b10135h78L001.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@292000z5217.78N/00648.33E_050/000g001t050r000p003P003b10135h78L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@292010z5217.78N/00648.33E_.../000g000t050r000p003P003b10135h78L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@292020z5217.78N/00648.33E_203/000g002t050r000p003P003b10135h78L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292030z5217.78N/00648.33E_.../000g000t050r000p003P003b10134h77L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@292040z5217.78N/00648.33E_343/000g002t050r000p003P003b10135h75L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@292050z5217.78N/00648.33E_246/000g002t050r000p003P003b10135h74L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@292100z5217.78N/00648.33E_.../000g000t050r000p003P003b10134h74L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@292110z5217.78N/00648.33E_020/000g002t050r000p003P003b10134h74L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@292120z5217.78N/00648.33E_.../000g000t049r000p003P003b10134h76L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292130z5217.78N/00648.33E_352/000g002t049r000p003P003b10134h76L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@292140z5217.78N/00648.33E_.../000g000t049r000p003P003b10134h77L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292150z5217.78N/00648.33E_.../000g000t049r000p003P003b10134h78L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@292200z5217.78N/00648.33E_345/000g002t049r000p003P000b10134h78L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@292210z5217.78N/00648.33E_.../000g000t049r000p003P000b10134h79L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@292220z5217.78N/00648.33E_.../000g000t049r000p003P000b10134h79L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@292230z5217.78N/00648.33E_022/000g002t049r000p003P000b10134h80L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@292240z5217.78N/00648.33E_.../000g000t048r000p003P000b10134h81L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292250z5217.78N/00648.33E_.../000g000t048r000p003P000b10134h81L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@292300z5217.78N/00648.33E_.../000g000t048r000p003P000b10133h82L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@292310z5217.78N/00648.33E_.../000g000t048r000p003P000b10134h82L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@292320z5217.78N/00648.33E_.../000g000t047r000p003P000b10133h82L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292330z5217.78N/00648.33E_.../000g000t047r000p003P000b10133h83L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@292340z5217.78N/00648.33E_.../000g000t047r000p003P000b10133h83L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@292350z5217.78N/00648.33E_.../000g000t046r000p003P000b10133h84L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300000z5217.78N/00648.33E_.../000g000t046r000p003P000b10133h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300010z5217.78N/00648.33E_.../000g000t045r000p003P000b10132h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@300020z5217.78N/00648.33E_.../000g000t045r000p003P000b10132h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300030z5217.78N/00648.33E_.../000g000t045r000p003P000b10132h87L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@300040z5217.78N/00648.33E_.../000g000t045r000p003P000b10132h87L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@300050z5217.78N/00648.33E_.../000g000t046r000p003P000b10131h87L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@300100z5217.78N/00648.33E_.../000g000t046r000p003P000b10130h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@300110z5217.78N/00648.33E_.../000g000t046r000p003P000b10131h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300120z5217.78N/00648.33E_.../000g000t046r000p003P000b10131h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@300130z5217.78N/00648.33E_.../000g000t...r000p003P000b.....h...weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300140z5217.78N/00648.33E_047/000g001t046r000p003P000b10130h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@300150z5217.78N/00648.33E_.../000g000t046r000p003P000b10129h86L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300200z5217.78N/00648.33E_.../000g000t046r000p003P000b10129h87L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300210z5217.78N/00648.33E_.../000g000t046r000p003P000b10128h87L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300220z5217.78N/00648.33E_.../000g000t045r000p003P000b10128h88L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300230z5217.78N/00648.33E_.../000g000t045r000p003P000b10127h89L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300240z5217.78N/00648.33E_.../000g000t044r000p003P000b10128h90L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300250z5217.78N/00648.33E_.../000g000t044r000p003P000b10127h91L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300300z5217.78N/00648.33E_.../000g000t044r000p003P000b10127h91L000.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@300310z5217.78N/00648.33E_.../000g000t044r000p003P000b10128h91L001.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300320z5217.78N/00648.33E_276/000g001t044r000p003P000b10128h90L002.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300330z5217.78N/00648.33E_118/000g001t045r000p003P000b10128h89L005.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300340z5217.78N/00648.33E_.../000g000t045r000p003P000b10128h89L010.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300350z5217.78N/00648.33E_.../000g000t045r000p003P000b10129h89L014.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@300400z5217.78N/00648.33E_.../000g000t044r000p003P000b10129h90L020.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300410z5217.78N/00648.33E_.../000g000t044r000p003P000b10129h90L020.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300420z5217.78N/00648.33E_.../000g000t044r000p002P000b10128h91L024.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300430z5217.78N/00648.33E_.../000g000t044r000p002P000b10129h91L037.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300440z5217.78N/00648.33E_.../000g000t044r000p002P000b10129h92L049.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300450z5217.78N/00648.33E_120/000g001t044r000p002P000b10129h91L086.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-1:@300500z5217.78N/00648.33E_312/000g002t045r000p002P000b10130h89L126.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-3:@300510z5217.78N/00648.33E_276/000g001t046r000p002P000b10130h87L131.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300520z5217.78N/00648.33E_316/000g002t046r000p002P000b10131h86L104.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300530z5217.78N/00648.33E_330/000g002t047r000p002P000b10132h85L100.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300540z5217.78N/00648.33E_323/000g002t047r000p002P000b10132h84L123.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300550z5217.78N/00648.33E_285/000g002t048r000p002P000b10132h82L140.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300600z5217.78N/00648.33E_245/000g001t048r000p002P000b10132h82L150.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-6:@300610z5217.78N/00648.33E_333/000g003t049r000p002P000b10132h82L161.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-4:@300620z5217.78N/00648.33E_008/000g003t049r000p002P000b10133h79L222.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-5:@300630z5217.78N/00648.33E_307/000g003t050r000p002P000b10133h77L379.weewx-4.7.0-WeatherFlowUDP <br>
EW0725>APRS,TCPXX*,qAX,CWOP-7:@300640z5217.78N/00648.33E_326/000g003t051r000p002P000b10133h76L240.weewx-4.7.0-WeatherFlowUDP <br>
</BODY>
</HTML>
Means that using a lua-file or python-file we need another way to get the CWOP data-lists:
=> a different plan of attack is required to read the url, convert the result to txt-type-file and subsequently extract the last line and it's contents.

An alternative url-call resulting in nice html-table provides a 'real' html-page with a very simple HTML-table included, covering last 24 hours, with UTC-info to full extend and the meteo-values already translated into metric units. If you reduce the url-call, you get a table with 'imperial' units.
In this setup with the actual values in the first/top dataline of the table.
Easier for web-scraping software?
Principally not in favour of webscraping, but don't see an alternative, simple approach to get to the CWOP-data.

June01, 2022:
2 Latest messages merged, and last deleted
Last edited by Toulon7559 on Wednesday 01 June 2022 22:49, edited 16 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 848
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: mixed
Location: Hengelo(Ov)/NL
Contact:

Re: Extraction of meteo-data from CWOP-table

Post by Toulon7559 »

Applying the url mentioned in the previous message as basis, in this message a basic Python3.x-script which
- calls the url giving a CWOP-datafile in HTML-format
- reads the CWOP-datafile
- extracts the weatherdata
- corrects the extracted data for expected applications for mainland Europe.

To make it running, the settings in this script are linked to an operational personal CWOP-station:
;-) NO responsibility & liability whatever
- that the data from this station is of good quality & continuous, and
- that the info from it is suitable&safe for your applications!!!!!!
Use is at own risk, and data resulting from the datafile should not to be used elsewhere!!!!!!!

.
For YOUR personal use YOU must adjust the geographical location (in line 059) and the CWOP-callsign (in line 081).
Under that condition, further application of the script is left to the ideas/wishes of the user.

However, the script below is only useful as feasability-demonstration,
because it is not clever in the perspective of value-extraction (with manual counting of positions within the dataline).
Based on present experience it certainly needs another method of determining the substring-positions in the extraction-segment.
Argument is that the datafields in the webpage have variable length and all 'separators' are identical:
- the 'separators' cannot be applied for finding specific data-lines and data-types
- when low or negative temperatures occur, it causes a change of length of the temp-value (which unfortunately is in the front of the dataline).
- wind direction and wind speed may have variation in length from 1 digit to 3 digits [because no leading 0's for small values]
- rain-values may be either 4 or 5 characters long for the same reason
=> the search for substringpositions must become capable to deal with variable length of value-fields
etc.
.
Spoiler: show
#!/usr/bin/python3
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2022 script compiled by Toulon7559 from various material from forums for application with Python3.x
# Version 00_220601, reading Info-table from CWOP-account, extracting values and standardizing
# Extraction based on setup as described at https://realpython.com/beautiful-soup-w ... er-python/
# Line 006 Description of setup
#
# For file-extraction for a specific station mandatorily required to have CWOP_CallSign for that specific station,
# CWOP_Documentation = https://weather.gladstonefamily.net/CWOP_Guide.pdf
# Setup for data-extraction, compilation & upload:
# 0. Install module pyEphem to get info on sunrise and sunset, and module requests for http-handling.
# 1. Read & Extract Weather Data from CWOP (= specific url-call)
# 2. Use extracted data for conversion as required to get SI-units,
#
# Overall script running interval to be realised by a cronjob: set running of that cronjob at the interval selected below or shorter.
# Assuming daily quotum of 500 url-calls,
# don't set the general script interval shorter than 5 minutes => 24*12 = 288 calls/24hours, leaving some spare quotum for other CWOP-callers.
#
# ------------------------------------------------------------
# Line 018 = PREPARATION & SETTING
# ------------------------------------------------------------
# Imports for script-operation
import requests # used for handling of http-requests

# Line 023
print ('Start of script CWOP_READER000')
DEBUG = 1 # DEBUG=0 is 'operational running without print'
# the "print"-lines only added for local read-out e.g. during script_tuning

# ------------------------------------------------------------
# Line 029 = PREPARATION FOR HANDLING DATE&TIME
# ------------------------------------------------------------
import calendar
import datetime # used for timestamps & timecheck
import time # used for timestamps & delays
from datetime import datetime, timedelta
now = datetime.now()
now_utc = datetime.utcnow()
dattim1 = datetime.now().strftime('%Y%m%d %H:%M')
dattim2 = datetime.now().strftime('%Y%m%d%H%M%S')
dattim3 = datetime.utcnow().strftime('%Y%m%d%H%M%S')
if DEBUG == 1:
print
print ('Present Date&Time (CET) = ', now)
print ('Present DatTim1 (CET) = ', dattim1)
print ('Present DatTim2 (CET) = ', dattim2)
print ('Present DatTim3 (UTC) = ', dattim3)
print

# ---------------------------------------------------
# Line 049 = EXTRACT SUNRISE & SUNSET
# ---------------------------------------------------
# Imports for script-operation
import ephem
#import datetime # already imported above

# Presetting of parameters

sun = ephem.Sun()
home = ephem.Observer()
home.lat, home.lon = '52.2962647', '6.80555' # YOUR lat, long # ADAPT TO YOUR STATION-LOCATION !!!!!!!!
# lat/long-info must be inserted as decimal degrees
sun.compute(home)
sunrise, sunset = (home.next_rising(sun), home.next_setting(sun))
sunrise_localtime, sunset_localtime = ephem.localtime(sunrise), ephem.localtime(sunset)
daylightminutes = (sunset_localtime - sunrise_localtime) * 1440 # find how many minutes of daylight today
# sunriselessonemin = ephem.date(sunrise + 1*ephem.minute)
if DEBUG == 1:
print ('= Sunrise & Sunset expressed in UTC! =')
print ('Sunrise = ',sunrise)
print ('Sunset = ',sunset)
print ('= Sunrise & Sunset expressed in Local time! =')
print ('Sunrise = ',sunrise_localtime)
print ('Sunset = ',sunset_localtime)

# ------------------------------------------------------------
# Line 075 = READ Weather-Information BY url-calls to the CWOP_Server
# ------------------------------------------------------------
# Login Info for CWOP-account
# CWOP_CallSign = "AB0123" # ADAPT for Station_CallSign => URL1 yet to be adapted for parameterized setup !!!!!!!!

# Example call-string for HTML-reportpage from CWOP = http://www.findu.com/cgi-bin/wx.cgi?cal ... its=metric
URL1 = "http://www.findu.com/cgi-bin/wx.cgi?cal ... its=metric"
page = requests.get(URL1)
a = page.text
if DEBUG == 1:
# print(a)
print
print ('End of reading datafile')
# ------------------------------------------------------------
# Line 089 = EXTRACT Actual Weather-Information from the HTML-file
# ------------------------------------------------------------
# Station_CallSign used as initial reference
# Next positions found by dumb counting of positions in the string
print(a.find("mb"))
print(a.index("mb"))
start_index = a.find("mb")
time_start = start_index + 37
print(time_start)
time_end = start_index + 51
print(time_end)
time_UTC = a[844:858]
print('time_UTC = ', time_UTC)
temp_start = time_end + 26
print(temp_start)
temp_end = temp_start + 3
print(temp_end)
tempC = a[884:887]
print('tempC = ',tempC)
wd_start = temp_end + 27
print(wd_start)
wd_end = wd_start + 4
print(wd_end)
windD = a[914:918]
print('windD = ',windD)
ws_start = wd_end + 26
print(ws_start)
ws_end = ws_start + 4
print(ws_end)
windS = a[943:947]
print('windS = ',windS)
wg_start = ws_end + 25
print(wg_start)
wg_end = wg_start + 4
print(wg_end)
windG = a[973:977]
print('windG = ',windG)
r1h_start = wg_end + 25
print(r1h_start)
r1h_end = r1h_start + 5
print(r1h_end)
rain1h = a[1002:1007]
print('rain1h = ',rain1h)
r24h_start = r1h_end + 25
print(r24h_start)
r24h_end = r24h_start + 5
print(r24h_end)
rain24h = a[1032:1037]
print('rain24h = ',rain24h)
rtoday_start = r24h_end + 25
print(rtoday_start)
rtoday_end = rtoday_start + 5
print(r1h_end)
rain_today = a[1062:1067]
print('rainToday = ',rain_today)
hum_start = rtoday_end + 25
print(hum_start)
hum_end = hum_start + 3
print(hum_end)
relhum = a[1092:1095]
print('relhum = ',relhum)
baro_start = hum_end + 25
print(baro_start)
baro_end = baro_start + 7
print(baro_end)
baro = a[1120:1127]
print('baro = ',baro)
print
print('Summary from extraction:')
print('time_UTC = ',time_UTC)
print('tempC = ',tempC)
print('windD (deg) = ',windD)
print('windS (kmh) = ',windS)
print('windG (kmh) = ',windG)
print('rain1h (cm) = ',rain1h)
print('rain24h (cm) = ',rain24h)
print('rainToday (cm) = ',rain_today)
print('relhum (%) = ',relhum)
print('baro (mb) = ',baro)
print

print ('End of extraction of data')

# -------------------------------------------------
# Line xxx = Collection & Correction of values
# -------------------------------------------------
# Corrections
print
print('Corrected values:')
print('CWOPtime_UTC = ',time_UTC)
print('localtime_CET = ',dattim2)
print('Sunrise = ',sunrise_localtime)
print('Sunset = ',sunset_localtime)
print('tempC = ',tempC)
print('windD (deg) = ',windD)
print('windS (kmh) = ',windS)
print('windG (kmh) = ',windG)
rainrate = 10 * float(rain1h)
print('rain1h (mm) = ',rainrate)
rain24h = 10 * float(rain24h)
print('rain24h (mm) = ',rain24h)
rain_today = 10 * float(rain_today)
# but this is rain since 00:00UTC, not rain since 00:00CET !
print('rainToday (mm) = ',rain_today)
print('relhum (%) = ',relhum)
print('baro (hPa/MSL) = ',baro)
print

# -------------------------------------------------
print ('Here is reserved space for applications')
# -------------------------------------------------
print
# -------------------------------------------------
print ('End of script CWOP_Reader000')
Last edited by Toulon7559 on Friday 10 June 2022 11:19, edited 4 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 848
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: mixed
Location: Hengelo(Ov)/NL
Contact:

Re: Extraction of meteo-data from CWOP-table

Post by Toulon7559 »

Meanwhile built a new Python3.x-script operating on this url having fixed-format data-contents, including a datafeed aimed at HetWeerActueel.
.
This script will be published in this message after clean-up.
In practise the format of the data-table is less fixed than expected, requiring some 'corrective measures' while webscraping.

Addition 2024, 2nd of June:
Due to mods & extensions & detected variations the script meanwhile has grown too big to publish in this forum.
Because of the mutations due to differences at CWOP in APRS_RawData, probably also not easy to put a 'common'version in Github.
Last edited by Toulon7559 on Saturday 27 July 2024 15:40, edited 2 times in total.
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Toulon7559
Posts: 848
Joined: Sunday 23 February 2014 17:56
Target OS: Raspberry Pi / ODroid
Domoticz version: mixed
Location: Hengelo(Ov)/NL
Contact:

Re: Extraction of meteo-data from CWOP-table

Post by Toulon7559 »

Further application & development breeds some 'surprises'.
If feeding a geolocation to PyEphem with too many decimals, the output values for e.g. sunrisetime and sunsettime become 'foolish'.
Further experience with PyEphem also signals that this Python-module is sensitive to the type and to the application of other data-inputs:
some data must be textstrings, other data must be number, but unclear what&when&how ......
This example shows that various inputs are possible, but nowhere a description to be found [?] for setup&application of such inputs.

Any users with knowledge how to deal with setup of PyEphem for generation of sunrise and sunset for a given geolocation and given meteo-setup?

For correct understanding:
Geolocation in this case is NOT my own location, but might be some other place on earth and input data & output data is required for pythonic application.
If own location is applicable and sunrise&sunset required for Domoticz, then much easier to directly ask Domoticz.
Or can Domoticz be tweaked to occasionally provide info on sunrise&sunset for another, remote location?
Set1 = RPI-Zero+RFXCom433+S0PCM+Shield for BMP180/DS18B20/RS485+DDS238-1ZNs
Set2 = RPI-3A++RFLinkGTW+ESP8266s+PWS_WS7000
Common = KAKUs+3*PVLogger+PWS_TFA_Nexus
plus series of 'satellites' for dedicated interfacing, monitoring & control.
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests