Netatmo Public data

[Toulon7559]

Not having a Netatmo-weatherstation myself, but interested to compare online.
Was able to get an extraction via Netatmo's Public API towards a JSON-file, but now next stage is extracting the data of a specific station out of that file.

Spoiler: show

Example json-file

[
{
"land": "DE",
"stad": "Borkum",
"straat": null,
"id": "70:ee:50:16:11:34",
"lat": 53.586141,
"lon": 6.662984,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:47:54",
"measures": {
"temperature": 3.6,
"humidity": 88,
"pressure": 1004.4
}
},
{
"land": "DE",
"stad": "Borkum",
"straat": null,
"id": "70:ee:50:36:f3:40",
"lat": 53.5905,
"lon": 6.6623173,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:46:05",
"measures": {
"temperature": 3.8,
"humidity": 75,
"pressure": 1021.4,
"rain_60min": 0,
"rain_24h": 0.30300000000000005,
"rain_live": 0,
"rain_timeutc": "2022-11-20 17:45:51",
"wind_strength": 5,
"wind_angle": 51,
"gust_strength": 9,
"gust_angle": 66,
"wind_timeutc": "2022-11-20 17:46:04"
}
},
{
"land": "DE",
"stad": "Borkum",
"straat": null,
"id": "70:ee:50:1d:f4:a6",
"lat": 53.5821430898812,
"lon": 6.66388392448425,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:46:16",
"measures": {
"temperature": 3.2,
"humidity": 78,
"pressure": 1004.6,
"rain_60min": 0,
"rain_24h": 0.202,
"rain_live": 0,
"rain_timeutc": "2022-11-20 17:46:07"
}
},
{
"land": "DE",
"stad": "Greetsiel",
"straat": null,
"id": "70:ee:50:01:c5:fc",
"lat": 53.500934,
"lon": 7.095159,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:50:17",
"measures": {
"temperature": 2.5,
"humidity": 80,
"pressure": 1009,
"rain_60min": 0,
"rain_24h": 0.6060000000000001,
"rain_live": 0,
"rain_timeutc": "2022-11-20 17:50:08",
"wind_strength": 1,
"wind_angle": 236,
"gust_strength": 3,
"gust_angle": -1,
"wind_timeutc": "2022-11-20 17:50:15"
}
},
{
"land": "DE",
"stad": "Krummh\u00f6rn",
"straat": null,
"id": "70:ee:50:1b:29:92",
"lat": 53.4990946,
"lon": 7.10417989999996,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:50:38",
"measures": {
"temperature": 3.2,
"humidity": 81,
"pressure": 1007.8
}
},
{
"land": "DE",
"stad": "Landkreis Aurich",
"straat": null,
"id": "70:ee:50:00:51:08",
"lat": 53.6007662523399,
"lon": 7.17545628547668,
"tijdzone": "Europe\/Berlin",
"tijd": "2022-11-20 17:49:21",
"measures": {
"temperature": 2.9,
"humidity": 78,
"pressure": 1010.5,
"rain_60min": 0,
"rain_24h": 0.30300000000000005,
"rain_live": 0,
"rain_timeutc": "2022-11-20 17:49:19"
}
}
]

The difficulty is that the various segments/station have no numbering in the sequence within the file, but are virtually stand-alone.
Have tried with a Python3.x's find-function to determine location with a specific 'ïd"as identifier (which as MAC-address is unique per station), but without success.
Anybody a hint/example for an alternative approach?

[waltervl]

Put the JSON in a table , perform a length of the array with #table
Then you can run through them until you reach a specific field that matches your station eg stad = xxxx Then you can get the results for that station.
For an example see stookalert script on wiki that searches for a provincie https://www.domoticz.com/wiki/StookAlert

[Toulon7559]

Walter,
Biased to using Python instead of dzVents.

But triggered by your hint, I changed the searchkey and found this description:
the example-input for it's solution5 has almost same layout as the json-file which is input for my application.
Fiddled with the various entries of the demonstration-script for solution5, and with the selected 'ïd" the related segment now is extracted!
That cleanly reduces the future effort to dissecting of the selected segment of the json-file.

[waltervl]

No problem How is your memory doing while running python scripts within Domoticz 2022.2 Stable?

[Toulon7559]

Aspect = CPU Capacity
All RPIs run mixture of scripts: Domoticz, Lua, Python and dzVents.
1*RPI_B (with many scripts) running at average 70~76% CPU_Usage, without problems.
1*RPI_3A+ (with many scripts) at this moment running at 20% CPU_Usage, but occasionally flipping out and very slow with webrefresh, apparently with overload [although with 'original' power supply]. Sometimes restart_after_flipping is very slow. Probably need to reduce the 'occupation' of that machine.
Several RPI3Bs running at CPU_Usage of 20% or less.
RPI_Zeroes (as 'satellites') have very low load in scripts and seldomly have problems.
Have impression that load-peaks only occur at 'favourite times' when refreshes are done:
around midnight e.g. now the RPI_B hitting 97% CPU_Usage.
On purpose working with timing-offsets in the scripts and with scheduled execution of scripts.

[waltervl]

You are talking about CPU, I wanted to know your memory usage experience. As there seemed to be some memory leakage with python scripts within Domoticz due to the new python framework.

[Toulon7559]

Aspect = MemoryUsage
Present MemoryUsage as reported by Domoticz for 'Motherboard'
RPI_B at 60% ~ 80%
RPI3A+ at approx. 40%
RPI3Bs at <20%
RPI_Zeroes at 15% ~ 35%

No idea whether of influence, but my running Python-scripts still mostly Python2.x, although transition planned to Python3.x related to shift to Raspian_Bullseye. Functionally no reason to change for most scripts, therefore no push to change for the older software still running on the Buster-RPIs.

[Toulon7559]

Making some progress with extraction of specific station-data from the NetatmoPublic-output, but next hurdle present as shown in the spoiler below, which shows debugging-output from the Python3-script.
The string-search-result shown as Country = 4 etc. comes from the script-lines shown further down this message.

Spoiler: show

Step1 = Check presence of Netatmo_ID
string exists in the file
Step2_basic = find location of ID using specific file-name and specific id-name
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-23 10:42:49', 'measures': {'temperature': 8.4, 'humidity': 96, 'pressure': 1011.2, 'rain_60min': 0, 'rain_24h': 0.202, 'rain_live': 0, 'rain_timeutc': '2022-11-23 10:30:11', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': 12, 'wind_timeutc': '2022-11-23 10:42:47'}}]
Step2_improved, by using parameters from setup-info
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-23 10:42:49', 'measures': {'temperature': 8.4, 'humidity': 96, 'pressure': 1011.2, 'rain_60min': 0, 'rain_24h': 0.202, 'rain_live': 0, 'rain_timeutc': '2022-11-23 10:30:11', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': 12, 'wind_timeutc': '2022-11-23 10:42:47'}}]
Step2_improved, with string_output
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-23 10:42:49', 'measures': {'temperature': 8.4, 'humidity': 96, 'pressure': 1011.2, 'rain_60min': 0, 'rain_24h': 0.202, 'rain_live': 0, 'rain_timeutc': '2022-11-23 10:30:11', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': 12, 'wind_timeutc': '2022-11-23 10:42:47'}}]

Step3 = Find startposition of strings
Country = 4
Place = 18
Street = 18
ID = 52
Lat = 4
Lon = 4
TimeZone = 19
Time = 19
Measures = 31
Temperature = 2
Humidity = 206
Barometer = 129
Rain01Hr = 28
Rain24Hr = 28
RainLive = 28
RainTime = 28
Wind_speed = 333
Wind_angle = 333
Gust_speed = 343
Gust_angle = 343
WindTime = 333

Raw values, actual
DE
Borkum
None
70:ee:50:36:f3:40
53.5905
6.6623173
Europe/Berlin
2022-11-23 10:42:49
8.4
96
1011.2
0
0.20
:
'2022-11-23 10:30:1

To extract data from a textstring, it is required to determine the start and the end of de data-field.
In this case the standard find()-function of Python does not always produce correct info.
As alternative now using the following function in the script.

Code: Select all

# Line 201 = Find location of strings
print
print('Find startposition of strings')
def find_str(s, char):
    index = 0           
    if char in s:
        char = char[0]
        for ch in s:
            if ch in s:
                index += 1
            if ch == char:
                return index

    else:
        return -1

# Line 220 = Find presence and locations
print('Country = ',(find_str(Station_segment, "land")))

IMHO nothing exotic in the text-string.
Result of the string-search is OK for the 1st field, but becomes erratic for all searches further away,
even duplicate reporting for some different string-locations.
Have tried several other string-search-scripts on the strings shown in the spoiler, but not much different.
Also translated back the string into JSON-file and then extract, but no better result.

Pragmatic, simple way out for the erroneuous sections is to manually count positions in the string,
and start with use of a read-out with script-code like below.
For the subsequent field-positons xxStart and xxEnd no string-search as basis, but just application of found offsets.

Code: Select all

# Line xxx = Find actual data for this station
# Readout_command is <string_name>[a,b] with a = startposition and b = endposition within the string
# Ideal is to have a and b filled-in by values from string-search + offset-algorithm
# Values in the next, 1st segment have fixed length within station-messages, but variable per station
# xxStart = 1st character of value, xxEnd = last character of value 
LandPos = find_str(Station_segment, "land")
LandStart = LandPos + 7
LandEnd = LandStart + 2 # Fixed NC for Country-value
Land = Station_segment[LandStart:LandEnd]

As shown in the lines below line 'Raw values, actual', that is stable for static information, but plays havoc for fields with dynamically variable length of the values. From the read-outs it almost looks that the spacing between the strings is variable (which should not be).

Looking for a better function to get correct positions of start and end of dynamic value-fields within a string.
Or alternative search-setup directly operating on the big JSON-file, mentioned/shown in this previous message.

[waltervl]

Not exactly sure what your problem is.
But by converting the json into a python dict it will autmatically link entry with value eg

Code: Select all

# JSON string
employee ='{"id":"09", "name": "Nitin", "department":"Finance"}'
 
# Convert string to Python dict
employee_dict = json.loads(employee)

print(employee_dict['name'])

Will print:
Nitin

https://www.geeksforgeeks.org/read-writ ... ng-python/

[Toulon7559]

Walter,

In general get the impression, that the erratic results may come from missing some subtleties when setting the found scripts and functions in relation to Python3.x.
Have to better study/understand the information.

The route via the dictionary could be extended to make a a json-file which may be queried the usual way:
surely quickest option, therefore primary target for my effort, but (for me) yet with least success.

Although a clumsier approach, quicker results expected with string-search and related value-extraction.
Have now more studied the find()-function and it's brethern
=> have to retry string-search with other layout.

If reason, will adapt my previous message to reflect any findings, or in this message, or in a next message.

Addition 23rd of November, 2022:
Have further tested with string-search using Python's find()-function.
Enclosed as demonstration why looking for other solutions.
.
The 1st spoiler shows the script-segment doing the work on the Datablock:
- the find()-function looks for start-position of designated text
- the boundary-settings for reading require manual counting positions for the offsets (should be stable and once only for the max. layout of messages)
- the read-function is simple/straightforward working on the Datablock with the parameterized boundary-settings.
.

Spoiler: show

# Line xxx = Find actual data for this station
# Result of string-find on purpose integer to suppress any possible jitter due to estimation.
# Readout_command is <string_name>[a,b] with a = startposition and b = endposition within the string
# Ideal is to have a and b filled-in by values from section above
# Formulas for string-search from previous section.
# Line xxx = Values in the next, 1st segment have fixed length within station-messages, but variable per station
# xxStart = position of 1st character of value, xxEnd = position of last character of value
# Offset between the value-fields should be fixed Number of Characters
# Line xxx = Extract positions for start and end of values and extract values
IDStart = int(Datablock.find("id")) + 6
IDEnd = IDStart + 17 # Fixed NC for MAC-address
ID = Datablock[IDStart:IDEnd]
LatStart = int(Datablock.find("lat")) + 5
LatEnd = int(Datablock.find("lon")) - 3
Latitude = Datablock[LatStart:LatEnd]
LonStart = int(Datablock.find("lon")) + 5
LonEnd = int(Datablock.find("tijdzone")) - 3
Longitude = Datablock[LonStart:LonEnd]
TZStart = int(Datablock.find("tijdzone")) + 13
TZEnd = int(Datablock.find("tijd")) - 5
Time_Zone = Datablock[TZStart:TZEnd]
TLStart = int(Datablock.find("tijd")) + 7
TLEnd = TLStart + 19 # Fixed NC for Date&Time
DateTime_Local = Datablock[TLStart:TLEnd]
DateStart = TLStart
DateEnd = DateStart + 10 # Fixed NC for Date
Date = Datablock[DateStart:DateEnd]
TimeStart = DateEnd + 2
TimeEnd = TimeStart + 5 # Fixed NC for Time
Time = Datablock[TimeStart:TimeEnd]
TempStart = int(Datablock.find("temperature")) + 15
TempEnd = int(Datablock.find("humidity")) - 5
TempC = Datablock[TempStart:TempEnd]
RHStart = int(Datablock.find("humidity")) + 13
RHEnd = int(Datablock.find("pressure")) - 13
RelHum = Datablock[RHStart:RHEnd]
PStart = int(Datablock.find("pressure")) + 12
PEnd = PStart + 6 # No dependency on Rain-info, but fixed NC
Baro = Datablock[PStart:PEnd]
# Line xxx = Variable presence of Rain Data
if Raingauge_status > 0:
R1Start = int(Datablock.find("rain_60min")) + 13
R1End = int(Datablock.find("rain_24h")) - 4
Rain01Hr = Datablock[R1Start:R1End]
R2Start = int(Datablock.find("rain_24h")) + 11
R2End = int(Datablock.find("rain_live")) - 4
Rain24Hr = Datablock[R2Start:R2End]
R3Start = int(Datablock.find("rain_live")) + 12
R3End = int(Datablock.find("rain_timeutc")) - 4
RainLive = Datablock[R3Start:R3End]
R4Start = int(Datablock.find("rain_timeutc")) + 15
R4End = R4Start + 19 # Fixed NC for Date and Time
RainTime = Datablock[R4Start:R4End]

else:
Rain01Hr = Invalid
Rain24Hr = Invalid
RainLive = Invalid
RainTime = Invalid

# Line xxx = Variable presence of Wind Data
if Anemometer_status > 0:
W1Start = Datablock.find("wind_strength") + 15
W1End = Datablock.find("wind_angle") - 3
WindS = Datablock[W1Start:W1End]
W2Start = Datablock.find("wind_angle") + 13
W2End = Datablock.find("gust_strength") - 3
WindD = Datablock[W2Start:W2End]
W3Start = Datablock.find("gust_strength") + 15
W3End = Datablock.find("gust_angle") - 3
WindG = Datablock[W3Start:W3End]
W4Start = Datablock.find("gust_angle") + 13
W4End = Datablock.find("wind_timeutc") - 3
WindGD = Datablock[W4Start:W4End]
W5Start = Datablock.find("wind_timeutc") + 16
W5End = W5Start + 19 # Fixed NC for Date and Time
WindTime = Datablock[W5Start:W5End]

else:
WindS = Invalid
WindD = Invalid
WindG = Invalid
WindGD = Invalid
WindTime = Invalid

# Line 0xxx = Check by print
if DEBUG == 1:
print
print('Raw values, actual; from Datablock')
# print(Land)
# print(Stad)
# print(Straat)
print(ID)
print(Latitude)
print(Longitude)
print(Time_Zone)
print(DateTime_Local)
print(Date)
print(Time)
print(TempC)
print(RelHum)
print(Baro)
print(Rain01Hr)
print(Rain24Hr)
print(RainLive)
print(RainTime)
print(WindS)
print(WindD)
print(WindG)
print(WindGD)
print(WindTime)

.
The 2nd spoiler shows 'print' of the text/layout of the Datablock containing the text on which the script operates,
the position-reports by the related print-statements and
the prints from read-out using the position-reports translated to boundary-settings for reading.
.

Spoiler: show

Length of Datablock = 404 characters
Datablock:
'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-23 19:56:49', 'measures': {'temperature': 7.6, 'humidity': 95, 'pressure': 1011.8, 'rain_60min': 0.505, 'rain_24h': 0.808, 'rain_live': 0.101, 'rain_timeutc': '2022-11-23 19:56:37', 'wind_strength': 5, 'wind_angle': -1, 'gust_strength': 15, 'gust_angle': 359, 'wind_timeutc': '2022-11-23 19:56:44'
ID = 1
Lat = 28
Lon = 44
TimeZone = 62
Time = 62
Measures = 122
Temperature = 133
Humidity = 155
Barometer = 171
Rain01Hr = 191
Rain24Hr = 212
RainLive = 231
RainTime = 251
Wind_speed = 290
Wind_angle = 310
Gust_speed = 328
Gust_angle = 349
WindTime = 368
Raw values, actual; from Datablock
70:ee:50:36:f3:40
53.5905
6.6623173

e': 'Europe/Berlin'
e': 'Europ
Berli


011.8,
0.50
0.80
0.10
'2022-11-23 19:56:3
5
-1
15
359
2022-11-23 19:56:44

Observations:
- the text in the Datablock looks OK
- the position read-outs with the find()-function also look OK
- the information from reading in some cases seems usable, but for most other fields the reading bracket is shifting without reason.

[waltervl]

This seems to work to find the temperature of station with id "70:ee:50:36:f3:40"

Code: Select all

import json

test_list = '[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:16:11:34", "lat": 53.586141, "lon": 6.662984, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:47:54", "measures": { "temperature": 3.6, "humidity": 88, "pressure": 1004.4 } }, { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:46:05", "measures": { "temperature": 3.8, "humidity": 75, "pressure": 1021.4, "rain_60min": 0, "rain_24h": 0.30300000000000005, "rain_live": 0, "rain_timeutc": "2022-11-20 17:45:51", "wind_strength": 5, "wind_angle": 51, "gust_strength": 9, "gust_angle": 66, "wind_timeutc": "2022-11-20 17:46:04" } }, { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:1d:f4:a6", "lat": 53.5821430898812, "lon": 6.66388392448425, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:46:16", "measures": { "temperature": 3.2, "humidity": 78, "pressure": 1004.6, "rain_60min": 0, "rain_24h": 0.202, "rain_live": 0, "rain_timeutc": "2022-11-20 17:46:07" } }, { "land": "DE", "stad": "Greetsiel", "straat": null, "id": "70:ee:50:01:c5:fc", "lat": 53.500934, "lon": 7.095159, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:50:17", "measures": { "temperature": 2.5, "humidity": 80, "pressure": 1009, "rain_60min": 0, "rain_24h": 0.6060000000000001, "rain_live": 0, "rain_timeutc": "2022-11-20 17:50:08", "wind_strength": 1, "wind_angle": 236, "gust_strength": 3, "gust_angle": -1, "wind_timeutc": "2022-11-20 17:50:15" } }, { "land": "DE", "stad": "Krummh\u00f6rn", "straat": null, "id": "70:ee:50:1b:29:92", "lat": 53.4990946, "lon": 7.10417989999996, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:50:38", "measures": { "temperature": 3.2, "humidity": 81, "pressure": 1007.8 } }, { "land": "DE", "stad": "Landkreis Aurich", "straat": null, "id": "70:ee:50:00:51:08", "lat": 53.6007662523399, "lon": 7.17545628547668, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:49:21", "measures": { "temperature": 2.9, "humidity": 78, "pressure": 1010.5, "rain_60min": 0, "rain_24h": 0.30300000000000005, "rain_live": 0, "rain_timeutc": "2022-11-20 17:49:19" } } ]'

test_list_dict = json.loads(test_list)
# Find dictionary matching value in list
res = None
for sub in test_list_dict:
    if sub['id'] == "70:ee:50:36:f3:40":
        res = sub
        break
 
# printing result
print("The filtered station data is : " + str(res))
print("The temperature : " + str(res['measures']['temperature']))

Result:

Code: Select all

The filtered station data is : {'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-20 17:46:05', 'measures': {'temperature': 3.8, 'humidity': 75, 'pressure': 1021.4, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-20 17:45:51', 'wind_strength': 5, 'wind_angle': 51, 'gust_strength': 9, 'gust_angle': 66, 'wind_timeutc': '2022-11-20 17:46:04'}}
The temperature : 3.8

[Toulon7559]

Walter,

Thanks!
Work for later this week to implement & try your setup in 'my environment'.

Anton

[Toulon7559]

Walter,

The data of a selected station is extracted from a big JSON-file containing all Netatmo-stations found within the search-window.
The big JSON-file is called /home/pi/netatmo.json
The extract is an outputstring called Station_segment.
The Python-scriptsegment for the extraction is at scriptline 172 till 202 in the 1st spoiler, with results shown in the 1st part of spoiler2.
Have inserted your test-script in my script at script-line 205 and the original runs without problems.
When switching the input from your test_list to outputstring Station_segment,
then the error reports appear as shown at the end of spoiler2.
Apparently a type-missmatch to be solved .......

Spoiler: show

# Line 172 = Step2, Find location of Identifier
print
print('Step2_basic = find location of ID using specific file-name and specific id-name')
# Open the existing JSON file for loading into a variable
with open('/home/pi/netatmo.json') as jsondata:
data = json.load(jsondata)

# Search data based on specific key and specific value using filter and list method
print(list(filter(lambda x:x["id"]=="70:ee:50:36:f3:40",data)))

# Line 182 = Step2a, Find location of Identifier
print
print('Step2_improved, using parameters from setup-info')
# Open the existing JSON file for loading into a variable
with open(Netatmo_file) as jsondata:
data = json.load(jsondata)

# Search data based on specific key and parameterized value using filter and list method
print(list(filter(lambda x:x["id"]==Netatmo_ID,data)))

# Line 192 = Step2b, Find location of Identifier & extract related segment of JSON-file
print
print('Step2_improved, with segment_extract and string_output')
# Open the existing JSON file for loading into a variable
with open(Netatmo_file) as jsondata:
data = json.load(jsondata)

# Search data based on specific key and parameterized value using filter and list method
Station_segment = str(list(filter(lambda x:x["id"]==Netatmo_ID,data)))
print('Station_segment_string = ')
print(Station_segment)

#------------------------------------------------------------
# Line 205 = Read String + Make Dictionary + Extract Data
#------------------------------------------------------------
import json

# test_list = '[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:16:11:34", "lat": 53.586141, "lon": 6.662984, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:47:54", "measures": { "temperature": 3.6, "humidity": 88, "pressure": 1004.4 } }, { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:46:05", "measures": { "temperature": 3.8, "humidity": 75, "pressure": 1021.4, "rain_60min": 0, "rain_24h": 0.30300000000000005, "rain_live": 0, "rain_timeutc": "2022-11-20 17:45:51", "wind_strength": 5, "wind_angle": 51, "gust_strength": 9, "gust_angle": 66, "wind_timeutc": "2022-11-20 17:46:04" } }, { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:1d:f4:a6", "lat": 53.5821430898812, "lon": 6.66388392448425, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:46:16", "measures": { "temperature": 3.2, "humidity": 78, "pressure": 1004.6, "rain_60min": 0, "rain_24h": 0.202, "rain_live": 0, "rain_timeutc": "2022-11-20 17:46:07" } }, { "land": "DE", "stad": "Greetsiel", "straat": null, "id": "70:ee:50:01:c5:fc", "lat": 53.500934, "lon": 7.095159, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:50:17", "measures": { "temperature": 2.5, "humidity": 80, "pressure": 1009, "rain_60min": 0, "rain_24h": 0.6060000000000001, "rain_live": 0, "rain_timeutc": "2022-11-20 17:50:08", "wind_strength": 1, "wind_angle": 236, "gust_strength": 3, "gust_angle": -1, "wind_timeutc": "2022-11-20 17:50:15" } }, { "land": "DE", "stad": "Krummh\u00f6rn", "straat": null, "id": "70:ee:50:1b:29:92", "lat": 53.4990946, "lon": 7.10417989999996, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:50:38", "measures": { "temperature": 3.2, "humidity": 81, "pressure": 1007.8 } }, { "land": "DE", "stad": "Landkreis Aurich", "straat": null, "id": "70:ee:50:00:51:08", "lat": 53.6007662523399, "lon": 7.17545628547668, "tijdzone": "Europe\/Berlin", "tijd": "2022-11-20 17:49:21", "measures": { "temperature": 2.9, "humidity": 78, "pressure": 1010.5, "rain_60min": 0, "rain_24h": 0.30300000000000005, "rain_live": 0, "rain_timeutc": "2022-11-20 17:49:19" } } ]'
test_list = Station_segment

test_list_dict = json.loads(test_list)
print('Dictionary =')
print(test_list_dict)
# Find dictionary matching value in list
res = None
for sub in test_list_dict:
if sub['id'] == "70:ee:50:36:f3:40":
res = sub
break

# printing result
print("The filtered station data is : " + str(res))
print("The temperature : " + str(res['measures']['temperature']))
print

Spoiler: show

End of reading datafile
Step1 = Check presence of Netatmo_ID
string exists in the file
Step2_basic = find location of ID using specific file-name and specific id-name
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 15:18:21', 'measures': {'temperature': 12.5, 'humidity': 79, 'pressure': 1023.4, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 15:18:18', 'wind_strength': 1, 'wind_angle': 359, 'gust_strength': 5, 'gust_angle': 193, 'wind_timeutc': '2022-11-24 15:18:18'}}]
Step2_improved, using parameters from setup-info
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 15:18:21', 'measures': {'temperature': 12.5, 'humidity': 79, 'pressure': 1023.4, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 15:18:18', 'wind_strength': 1, 'wind_angle': 359, 'gust_strength': 5, 'gust_angle': 193, 'wind_timeutc': '2022-11-24 15:18:18'}}]
Step2_improved, with segment_extract and string_output
Station_segment_string =
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 15:18:21', 'measures': {'temperature': 12.5, 'humidity': 79, 'pressure': 1023.4, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 15:18:18', 'wind_strength': 1, 'wind_angle': 359, 'gust_strength': 5, 'gust_angle': 193, 'wind_timeutc': '2022-11-24 15:18:18'}}]
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 212, in <module>
test_list_dict = json.loads(test_list)
File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 3 (char 2)

[waltervl]

Try to change line
test_list_dict = json.loads(test_list)
into
test_list_dict = json.load(jsondata)

[Toulon7559]

Easily tested, but unfortunately without progress:
IMHO some further study needed on the type-missmatch between the extracted text-string Station_segment and the required input-file to make the dictionary.

Spoiler: show

End of reading datafile
Step1 = Check presence of Netatmo_ID
string exists in the file
Step2_basic = find location of ID using specific file-name and specific id-name
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:00:54', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:00:31', 'wind_strength': 3, 'wind_angle': -1, 'gust_strength': 6, 'gust_angle': 270, 'wind_timeutc': '2022-11-24 20:00:50'}}]
Step2_improved, using parameters from setup-info
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:00:54', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:00:31', 'wind_strength': 3, 'wind_angle': -1, 'gust_strength': 6, 'gust_angle': 270, 'wind_timeutc': '2022-11-24 20:00:50'}}]
Step2_improved, with segment_extract and string_output
Station_segment_string =
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:00:54', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:00:31', 'wind_strength': 3, 'wind_angle': -1, 'gust_strength': 6, 'gust_angle': 270, 'wind_timeutc': '2022-11-24 20:00:50'}}]
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 212, in <module>
test_list_dict = json.load(jsondata)
File "/usr/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
ValueError: I/O operation on closed file.

Some quick&easy experiments.
Looking at visual resemblance with your test_list tried the trick to add ' at head and tail by instruction

Code: Select all

Station_segment = "'" + Station_segment + "'"

Produces textstring Station_segment_string2, but no luck for processing: just another error report.

Spoiler: show

End of reading datafile
Step1 = Check presence of Netatmo_ID
string exists in the file
Step2_basic = find location of ID using specific file-name and specific id-name
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:10:59', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:10:59', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': -1, 'wind_timeutc': '2022-11-24 20:10:59'}}]
Step2_improved, using parameters from setup-info
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:10:59', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:10:59', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': -1, 'wind_timeutc': '2022-11-24 20:10:59'}}]
Step2_improved, with segment_extract and string_output
Station_segment_string =
[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:10:59', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:10:59', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': -1, 'wind_timeutc': '2022-11-24 20:10:59'}}]
Station_segment_string2 =
'[{'land': 'DE', 'stad': 'Borkum', 'straat': None, 'id': '70:ee:50:36:f3:40', 'lat': 53.5905, 'lon': 6.6623173, 'tijdzone': 'Europe/Berlin', 'tijd': '2022-11-24 20:10:59', 'measures': {'temperature': 7.6, 'humidity': 96, 'pressure': 1024.3, 'rain_60min': 0, 'rain_24h': 0.30300000000000005, 'rain_live': 0, 'rain_timeutc': '2022-11-24 20:10:59', 'wind_strength': 3, 'wind_angle': 359, 'gust_strength': 10, 'gust_angle': -1, 'wind_timeutc': '2022-11-24 20:10:59'}}]'
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 213, in <module>
test_list_dict = json.loads(test_list)
File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

For more visual resemblance bluntly added the spaces at head and tail of the string around [ and { , respectively around } and ]
and replaced None by null, using 3 extra scriptlines

Code: Select all

string0 = Station_segment.replace("'[{", "'[ { ")
string1 = string0.replace("None", "null")
Station_segment = string1.replace("}]'", " } ]'")

but that makes no difference.

Must be some required subtlety in the translation from text-string into JSON-file.

[waltervl]

The test list string is just a copy from the JSON in your first post. I only removed the new lines so it will become 1 string as the original JSON probably also is. How do you get the JSON from Netatmo? Can't you just download it within the python script?

[Toulon7559]

The big JSON-file giving the collection of Netatmo-station data is output of a PHP-script,
using original Netatmo API-call to get Public data available from the stations within a designated, geo-oriented search-window.
The idea is to have that big JSON-file as data-source for extraction for several selected stations, with separate extracting-script per selected station to extract the detailled data aimed/tuned at an application.
The single PHP-script periodically 'pulls' data from a remote server at internet, while the extracting-scripts as satellites locally perform their functions at 'convenient times', not loading the external interfaces.
Because of this 'demarcation', those extracting-scripts may be in any other language than PHP.
Myself not well-versed in PHP, but having the subsequent user-applications in Python,
is one reason to try Python for the interfacing software.

Meanwhile have pensively been triggered for another aspect of your test_list in which all textual elements enclosed in " ",
because of a statement found at W3Schools

In JSON, string values must be written with double quotes:
JSON
{"name":"John"}

That is applicable in the original JSON-file, but not in the extracted text-file Station_segment:
Is easy to remedy by scriptline

Code: Select all

Station_segment = Station_segment.replace("'", '"')

Although the resulting stringprint looks very similar to your test_list, still the same error report

Spoiler: show

Station_segment_string3 =
'[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe/Berlin", "tijd": "2022-11-24 23:42:49", "measures": {"temperature": 7.1, "humidity": 96, "pressure": 1023.8, "rain_60min": 0, "rain_24h": 0.404, "rain_live": 0, "rain_timeutc": "2022-11-24 23:42:28", "wind_strength": 3, "wind_angle": -1, "gust_strength": 12, "gust_angle": 261, "wind_timeutc": "2022-11-24 23:42:47"} } ]'
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 220, in <module>
test_list_dict = json.loads(test_list)
File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Mimick for subsitute of your test_file has been done, and strange thing is that your testscriptlines completely work if Station_segment_string3 is manually copied from Putty's CLI as substitute for the string in test_list in your proposed testscriptlines.

Next experiments are very rude try&error
It partlially works when I insert a scriptline with json.dumps() to catch the text-string Station_segment as input for making test_list

Code: Select all

test_list = json.dumps(Station_segment)
test_list_dict = json.loads(test_list)
print('Dictionary =')
print(test_list_dict)
# Find dictionary matching value in list
res = None
for sub in test_list_dict:
    if sub["id"] == "70:ee:50:36:f3:40":
        res = sub
        break
 
# printing result
print("The filtered station data is : " + str(res))
print("The temperature : " + str(res['measures']['temperature']))
print

=> getting a dictionary, but stuck at subsequent finding of sub["id"]

Spoiler: show

Station_segment_string3 = change of head & tail
'[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe/Berlin", "tijd": "2022-11-25 08:53:15", "measures": {"temperature": 8.1, "humidity": 97, "pressure": 1024.8, "rain_60min": 0.101, "rain_24h": 2.121, "rain_live": 0, "rain_timeutc": "2022-11-25 08:53:12", "wind_strength": 3, "wind_angle": -1, "gust_strength": 5, "gust_angle": 359, "wind_timeutc": "2022-11-25 08:53:12"} } ]'
Dictionary =
'[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe/Berlin", "tijd": "2022-11-25 08:53:15", "measures": {"temperature": 8.1, "humidity": 97, "pressure": 1024.8, "rain_60min": 0.101, "rain_24h": 2.121, "rain_live": 0, "rain_timeutc": "2022-11-25 08:53:12", "wind_strength": 3, "wind_angle": -1, "gust_strength": 5, "gust_angle": 359, "wind_timeutc": "2022-11-25 08:53:12"} } ]'
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 230, in <module>
if sub["id"] == "70:ee:50:36:f3:40":
TypeError: string indices must be integers

Trying direct reading from that dictionary with scriptline

Code: Select all

test_extract = test_list_dict["land"]

=> no luck, with same kind of error report

Spoiler: show

Dictionary =
'[ { "land": "DE", "stad": "Borkum", "straat": null, "id": "70:ee:50:36:f3:40", "lat": 53.5905, "lon": 6.6623173, "tijdzone": "Europe/Berlin", "tijd": "2022-11-25 09:23:30", "measures": {"temperature": 8.1, "humidity": 97, "pressure": 1025.2, "rain_60min": 0, "rain_24h": 2.121, "rain_live": 0, "rain_timeutc": "2022-11-25 09:23:20", "wind_strength": 3, "wind_angle": 11, "gust_strength": 5, "gust_angle": 216, "wind_timeutc": "2022-11-25 09:23:26"} } ]'
Traceback (most recent call last):
File "/home/pi/Python_scripts/JSON_NetatmoPublic_to_HWA03_xxxx_0198.py", line 228, in <module>
test_extract = test_list_dict["land"]
TypeError: string indices must be integers

We seem to be missing something subtle in filehandling in Python3.x & JSON .......

[waltervl]

There is Python json.load() and json.loads() that behave different.
https://docs.python.org/3/library/json.html#basic-usage

[Toulon7559]

The PHP-script as mentioned in this thread has been changed to additionally provide a json-file per station:
should be easier to read by a recipient-application.

netatmo_70-ee-50-90-ca-8c.txt

Example json_file_per_station

(613 Bytes) Downloaded 66 times

Obviously the original issue with extension .json, but changed to extension .txt to suit inclusion in the forum-message.

Also found very pragmatic way to get a readable json-file:
- upload/write the json-file coming from the PHP-script to webserver
- download/read that json-file from this webserver.
Do not ask 'Why', but then all 'usual' functions for read-out of data are without problem.
Just 'questionmark', why failing if trying to directly read the json-file 'inside' a Raspberry .......