Forwarding sensor measurements to ThingSpeak

[aXion]

Hi,

I want to be able to log the values from specific sensors in Domoticz with high resolution (say 20 secs) for a longer period, maybe days/weeks or even months. So I want to be able to view the same detailed graphs/logs even after a couple of months have passed. What I noticed is that in Domoticz, sensor values are logged with a resolution of 5 minutes and after a maximum of 7 days you're left with only one value a day. I can understand the reason behind this (database growth), but for some sensors I don't want to loose that detailed data and, as I said, want it in a higher resolution than 5 minutes.

That's why I want to forward the measurment values of some sensors to ThingSpeak via a http request (max resolution is 15s) whenever the values are read by Domoticz. I use a MySensors serial Gateway so I can control the update rate of the Mysensor sensor itself.

Can someone tell if this is possible, and if so, how this could be accomplished?

[ThinkPad]

Should be possible with a Lua script i think:

Pseudo code:

Code: Select all

IF devicechanged [NameOfSensor]
os.execute(curl http://thingspeak.com?value=1234)

[aXion]

Thanks, it's a bit unfavorable the Lua script is triggered for each and every device state change, but I've got it working

[ThinkPad]

If you only want to update with 20s interval, maybe you can use parts of my script: http://www.domoticz.com/forum/viewtopic.php?f=23&t=7512 i use it to 'flatten out' the doorbell signal, so only one message is sent if doorbell is pressed multiple times.

[Jumper3126]

Thanks, it's a bit unfavorable the Lua script is triggered for each and every device state change, but I've got it working

Hi Axion
Would it be possible for you to share the exact script you are using for this? Without the API key off course

Regards, Rutger

[parbar]

Also looking for this function. Made some test with lua without success. If someone have an working example please share!
Thanks,
Pär

[KimP]

Hi,

I was also looking for some ways to forward data to another application or something. I wanted to set verbose to 1, loglevel to 0 and write the logs to a named pipe. Most data i wanted is there, except i need to identify devices by name i give them. Output for example is

Code: Select all

subtype       = P1 Smart Meter Power
powerusage1 = 1599.491 kWh
powerusage2 = 1367.110 kWh
powerdeliv1 = 0.000 kWh
powerdeliv2 = 0.000 kWh
current usage = 684 Watt
current deliv = 000 Watt
2016-02-09 20:36:31.086  (P1) P1 Smart Meter (P1 Electricity)

but only way to match is by the name on the last line. So not really an option.

I want get all data and the other app can decide what to save and what not. A pipe or socket would be preferred but simple http request (lots of them, i know) is also an option.

Someone has some suggestions for me how to achieve this?

Grz,
Kim

[Jumper3126]

Hi Par
Could you perhaps share what you have got? Maybe we can make it a team effort.
I'm still eager for this option, but havent gotting to understand the Lua code yet

[parbar]

I'm in to team effort!
First some base information of my system and my goal. Running domoticz on linux ubuntu on a eeepc (not raspeberry).
I have a usb to one-wire adapter connected and use OWFS and and a few wired DS18b20. The temperatures are feed into domoticz and working great.
Recently been playing with ESP8266 and espeasy firmware (http://www.esp8266.nu) and will put up a few around the house to measure temperature and humidity.
I have two main reasons why I want to export data to thingspeak:
1. Keep my domoticz only on local network at home, for security reasons.
2. When outside my local network I can at least have control of temperatures at home, also thingspeak have more options for charts etc to play with.

My first idea was to send single sensor data to thingspeak using lua script every time a new measurement appeared, using device or variable to trigger script. I made a simple script but it didn´t work.
But I realise that in the end I probably end up sending more then one field to thingspeak. A channel can have 8 fields. Considering the 15 second channel update limit at thingspeak made my chance my mind.
I now ( figured this out today) believe use a time script whould be better, gather multiple data and send every minute instead, to avoid hitting the 15 second limit.

A typical url for updating thingspeak channel with two values is:

https://api.thingspeak.com/update?api_k ... d2=[value2]
[APIKEY] = api key
[value1] and [value2] = temperature values to send

The challenge is now how to make a lua script from this!!

First step: what should the lua script file be named?
Something like script_time_xxx.lua What should xxx be in this case?

Second step:
How to gather temperatures (or other values) from multiple sensors so they can be passed to the thingspeak url?


Third step:
How to actually send the data to thingspeak?
Something like: os.execute(curl https://api.thingspeak.com/update?api_k ... d2=[value2])
how to get the values from second step into the os.execute part?

Option to consider when a the base functions above is working:
in step 2 check how old the temperature reading is. If it is lets ay more then 5 minutes since last update then put a total different fixed value so you can easy notice if some sensor is down.

This is my thoughts so far. My time for tinkering with domoticz is limited (family you know) but I hope to get some pointers and something to start with.

Looking forward for any help!

/Pär

[parbar]

I made an bash script instead. For now only one measurement is forwarded to Thingspeak, but with some development I hope it will forward multiple values. Maybe not so advanced and very basic but it works. Using crontab to run the script.

https://thingspeak.com/channels/21739

I can share the script later and welcome comments and improvements!

[Jumper3126]

Hi Parbar,
The reason for me to focus on thingspeak is the low resolution in Domoticz. an average every 5 min is nice for the temp of a room, but I also want to monitor (and tweak) my central heating and therefore I would like realtime (every 15-30 sec) a datapoint in a graph. In addition, I like your idea of keeping Domoticz on the local network, and still being able to monitor the data from outside.

Similar to you, I'm limited in my tinkering time (recent father, and planning some home reconstruction). But I'am not in a tremendous hurry for this project, so hopefully with the help of the communinity we can come to something nice.

Untill now, I have not really attempted a script, as my knownledge on Lue is limited. I do have a python script however that can post temp values it requests from my central heating controller (evohome) and post them in thingspeak. Perhaps we can use elements from this file.

Could you share your bash script?


the python evohome to thingspeak scripts (step 4 is the important part):

Code: Select all

# Load required libraries
import requests
import json
import time

from datetime import datetime

### STEP 1: connect to the EvoHome and get user credentials
# Set your EvoHome login details in the 2 fields below
USERNAME = 'my user name'
PASSWORD = 'my password'
#seconds_between_logpoints = 600
current_temp = [0,0,0,0]
current_setpoint = [0,0,0,0]

# Initial JSON POST to the website to return your userdata
url = 'https://rs.alarmnet.com/TotalConnectComfort/WebAPI/api/Session'
postdata = {'Username':USERNAME, 'Password':PASSWORD,'ApplicationId':'91db1612-73fd-4500-91b2-e63b069b185c'}
headers = {'content-type':'application/json'}
response = requests.post(url,data=json.dumps(postdata),headers =headers)
userinfo = json.loads((response.content).decode(encoding="utf-8"))

# DV Now, we have downloaded the userinfo from the first HTTP request.
# Note that in python3 we have to use decode as bytes are handled differently 
# within python3 compared to python2. After the decoding, all works
# 
# The object "userinfo" is a dictionary and contains as one item the 
# key "userInfo"-- which holds all our userinfo.
# To access this:
# myUserInformation = userinfo["userInfo"]
# myUserInfoDetails = myUserInformation["streetAddress"]
#
# to print all information, do 
print("\nThe userinfo that is returned: ", userinfo)
print("\nFinished the initial JSON request")

### STEP 2: Connect to the EvoHome website for a single time, to get initial data 
###         and to figure out how many devices there are present
# Extract the sessionId and your userid from the response
userid = userinfo['userInfo']['userID']
sessionId = userinfo['sessionId']
print("The userid and sessionId are:", userid, sessionId)

# Next, using your userid, get all the data back about your site
url = 'https://rs.alarmnet.com/TotalConnectComfort/WebAPI/api/locations?userId=%s&allData=True' % userid
headers['sessionId'] = sessionId
response = requests.get(url,data=json.dumps(postdata),headers =headers)
fullData = json.loads((response.content).decode(encoding="utf-8"))[0]

### STEP 3: Connect to the EvoHome website for every defined number of seconds
###         and send the data to Thingspeak
# We need to store the data for all our devices, and send them in one go to Thingspeak
# The data is stored as a list (I assume here that there are 4 devices)

# Next, using your userid, get all the EvoHome data back about your site
url = 'https://rs.alarmnet.com/TotalConnectComfort/WebAPI/api/locations?userId=%s&allData=True' % userid
headers['sessionId'] = sessionId
response = requests.get(url,data=json.dumps(postdata),headers =headers)
fullData = json.loads((response.content).decode(encoding="utf-8"))[0]

# Get current time and temperature information
device_nr = 0
for device in fullData['devices']:
	tijd = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
	temperatuur = device['thermostat']['indoorTemperature']
	setpoint = device['thermostat']['changeableValues']['heatSetpoint']['value']
	naam = device["name"]
	current_temp[device_nr] = temperatuur
	current_setpoint[device_nr] = setpoint
	print("Device='{0}'\tTemp={1}\tSetpoint={2}\tTime={3}".format(naam, temperatuur, setpoint, tijd))
	device_nr = device_nr + 1

### STEP 4: now sending to Thingspeak:
payload = {"field1":current_temp[0], "field2":current_setpoint[0], "field3":current_temp[1], "field4":current_setpoint[1],"field5":current_temp[2], "field6":current_setpoint[2], "field7":current_temp[3], "field8":current_setpoint[3]}
url_TS = "https://api.thingspeak.com/update?key=JZRZ6MFJNXY70VHQ"
requests.get(url_TS, params=payload)

[Jumper3126]

Alright, I got some time to study some lua scripts and combine them to the code below. Unfortunately it doesnt work yet as I'm getting the following error:

2016-03-09 10:10:50.067 Error: EventSystem: /home/pi/domoticz/scripts/lua/script_device_thingspeak.lua:8: attempt to index global 'net' (a nil value)

Any idea what might be going wrong and how I can resolve it?

Code: Select all

local aanvoer_dv = 'CV Boiler Water Temperature'

commandArray = {}

if devicechanged[aanvoer_dv] then
        aanvoer = otherdevices_svalues[aanvoer_dv]
		print('send to thingspeak' .. aanvoer)
		conn=net.createConnection(net.TCP, 0)
		conn:on("receive", function(conn, payload) print(payload) end)
		conn:connect(80,'184.106.153.149')
		conn:send("GET /update?key=<1Z2ZVxxxxxxxxVAY>&field1="..aanvoer.." HTTP/1.1\r\n")
		conn:send("Host: api.thingspeak.com\r\n")
		conn:send("Accept: */*\r\n")
		conn:send("User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n")
		conn:send("\r\n")
		conn:on("sent",function(conn) print("Closing connection") conn:close() end)
		conn:on("disconnection", function(conn) print("Got disconnection...") end)
		conn = nil
end
return commandArray

[ThinkPad]

Your Lua script seems a bit complicated. Have a look at my script that i used to send the outside temperature to my thermostat: https://bitbucket.org/frankiepankie/dom ... ew-default