Getting WebSockets working in plugin framework Domoticz.Connection

Python and python framework

Moderator: leecollings

Post Reply
EscApe
Posts: 540
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2026.3
Location: The Netherlands
Contact:

Getting WebSockets working in plugin framework Domoticz.Connection

Post by EscApe »

Can anyone offer some insights in the code below? Maybe @Dnpwwo?

I am trying to use WebSockets for the Harmony Hub remote, but got stuck on incoming messages. I know Logitech has revived the old api, but lets call my effort 'future proofing' or 'a learning experience' :-P

The little proof of concept below will get the configuration from the hub and on every heartbeat the current status. The information is dumped to the log. The code is only usable to get some knowledge about the inner workings of the plugin framework and Harmony api. The hubid has to be set in the plugin configuration. No auto detection (yet).

If I run this 'plugin' both request work. The hub returns the configuration and the current status. I still have to figure out how to properly proces the incoming messages (found the earthquake example). However, the configuration gets broken up into separate messages. As far as I can tell it is broken up on a lower lever than the WebSockets protocol itself, because all following messages are missing the WebSockets header (it's only on the first message).

It would be great if websockers were to be implemented in the Domoticz framework! Thats way beyond my skills though.

Code: Select all

<plugin key="harmonytest" name="harmonytest" author="EScape" version="0.1" externallink="https://www.google.com">
	<description>
		<h2>Harmony testing</h2><br/>
		Can i use the plugin?
	</description>
	<params>
		<param field="Address" label="IP Address" width="200px" required="true" default="192.168.1.111"/>
		<param field="Mode5" label="hubId" width="200px" required="true" default="0000"/>
		<param field="Mode6" label="Debug" width="150px">
			<options>
				<option label="None" value="0"	default="true" />
				<option label="Python Only" value="2"/>
				<option label="Basic Debugging" value="62"/>
				<option label="Basic+Messages" value="126"/>
				<option label="Connections Only" value="16"/>
				<option label="Connections+Python" value="18"/>
				<option label="Connections+Queue" value="144"/>
				<option label="All" value="-1"/>
			</options>
		</param>
	</params>
</plugin>
"""
import Domoticz
import json
from struct import Struct
import random

class BasePlugin:
	apiConn = None
	runAgain = 2
	disconnectCount = 0
	PACK_LEN1 = Struct('!BB').pack
	PACK_LEN2 = Struct('!BBH').pack
	PACK_LEN3 = Struct('!BBQ').pack
   
	def __init__(self):
		return

	def send_frame(self, message, opcode=0x01):
		#Adapted from https://gemfury.com/agriconnect/python:aiohttp/-/content/http_websocket.py

		MSG_SIZE = 2 ** 14
		rsv = 0
		msg_length = len(message)
		mask_bit = 0

		if msg_length < 126:
			header = self.PACK_LEN1(0x80 | rsv | opcode, msg_length | mask_bit)
		elif msg_length < (1 << 16):
			header = self.PACK_LEN2(0x80 | rsv | opcode, 126 | mask_bit, msg_length)
		else:
			header = self.PACK_LEN3(0x80 | rsv | opcode, 127 | mask_bit, msg_length)
		if len(message) > MSG_SIZE:
			self.apiConn.Send(header)
			self.apiConn.Send(message)
		else:
			self.apiConn.Send(header + message)
	
# Description of frame
#		  HEADER, = struct.unpack("!H", Data[:2])
#		  wsData = Data[2:]
#		  FIN	 = (HEADER >> 15) & 0x01
#		  RSV1	 = (HEADER >> 14) & 0x01
#		  RSV2	 = (HEADER >> 13) & 0x01
#		  RSV3	 = (HEADER >> 12) & 0x01
#		  OPCODE = (HEADER >>  8) & 0x0F
#		  MASKED = (HEADER >>  7) & 0x01
#		  LEN	 = (HEADER >>  0) & 0x7F
#	Opcode:
#	0x00: this frame continues the payload from the last.
#	0x01: this frame includes utf-8 text data.
#	0x02: this frame includes binary data.
#	0x08: this frame terminates the connection.
#	0x09: this frame is a ping.
#	0x0a: this frame is a pong.
	def request_config(self):
		payload={}
		#payload['hubId']=self.hub_id #Doesn't even need the hubid?
		payload['timeout']=self.timeout
		payload['hbus']={}
		payload['hbus']['cmd']='vnd.logitech.harmony/vnd.logitech.harmony.engine?config'
		payload['hbus']['id']='1'
		payload['hbus']['params']='{"verb":"get","format":"json"}'
		sendData=json.dumps(payload)
		self.send_frame(sendData.encode('utf-8'))
		Domoticz.Status('request config!')
		
	def request_state(self):
		payload={}
#		payload['hubId']=self.hub_id
		payload['timeout']=self.timeout
		payload['hbus']={}
		payload['hbus']['cmd']='vnd.logitech.connect/vnd.logitech.statedigest?get'
		payload['hbus']['id']='22'
		payload['hbus']['params']='{"verb":"get","format":"json"}'
		payloadstr=b'{"hbus": {"cmd": "vnd.logitech.connect/vnd.logitech.statedigest?get", "params": "{\"verb\":\"get\",\"format\":\"json\"}", "id": "0"}, "timeout": 30}'
		sendData=json.dumps(payload)
		self.send_frame(sendData.encode('utf-8'))
		Domoticz.Status('request state!')
		return

	def hex2bin(self,bytestr):
		return "{0:0{1}b}".format(bytestr, 4)
		
	def onStart(self):
		runAgain = 2
		self.timeout=30
		self.Connected=False
		self.Initialized=False
		if Parameters["Mode6"] != "0":
			Domoticz.Statusging(int(Parameters["Mode6"]))
			DumpConfigToLog()
		self.hub_ip=Parameters["Address"]
		self.hub_id=Parameters["Mode5"]
		self.apiConn = Domoticz.Connection(Name="HarmonySocks", Transport="TCP/IP", Protocol="None", Address=self.hub_ip, Port="8088")
		self.apiConn.Connect()
		Domoticz.Status("onStart finished")

	def onStop(self):
		Domoticz.Log("onStop - Plugin is stopping.")
		self.apiConn.Disconnect()

	def onConnect(self, Connection, Status, Description):
		sendData='GET /?domain=svcs.myharmony.com&hubId='+self.hub_id+' HTTP/1.1\r\n' \
				'Origin: http://'+self.hub_ip+':8088\r\n' \
				'Connection: Upgrade\r\n' \
				'Host: '+self.hub_ip+':8088\r\n' \
				'Sec-WebSocket-Key: qcLfmlZLMw3jqHxKDHGV6A==\r\n' \
				'Sec-WebSocket-Version: 13\r\n' \
				'Upgrade: websocket\r\n\r\n' \
				
		Domoticz.Status("onConnect called")
		if (Status == 0):
			Domoticz.Status("Hub connected successfully.")
			Domoticz.Status(sendData)
			Connection.Send(sendData)
		else:
			Domoticz.Log("Failed to connect ("+str(Status)+") to: "+self.hub_ip+":8088 with error: "+Description)

	def onMessage(self, Connection, Data):
		Domoticz.Status('Message!')
#		Domoticz.Status('firstbyte received:' + str(Data[:1]))
		Domoticz.Status(str(Data))
		if not self.Initialized:
			self.request_config()
			self.Initialized=True

	def onCommand(self, Unit, Command, Level, Hue):
		Domoticz.Status("onCommand called for Unit " + str(Unit) + ": Parameter '" + str(Command) + "', Level: " + str(Level))

	def onDisconnect(self, Connection):
		Domoticz.Log("onDisconnect called for connection to: "+Connection.Address+":"+Connection.Port)

	def onHeartbeat(self):
		if (self.apiConn != None and (self.apiConn.Connecting() or self.apiConn.Connected())):
			Domoticz.Status("onHeartbeat called, Connection is alive.")
			self.request_state()
		else:
			self.runAgain = self.runAgain - 1
			if self.runAgain <= 0:
				if (self.apiConn == None):
					self.apiConn = Domoticz.Connection(Name='HarmonySocks', Transport="TCP/IP", Protocol="None", Address=self.hub_ip, Port="8088")
				self.apiConn.Connect()
				self.runAgain = 2
			else:
				Domoticz.Status("onHeartbeat called, run again in "+str(self.runAgain)+" heartbeats.")

global _plugin
_plugin = BasePlugin()

def onStart():
	global _plugin
	_plugin.onStart()

def onStop():
	global _plugin
	_plugin.onStop()

def onConnect(Connection, Status, Description):
	global _plugin
	_plugin.onConnect(Connection, Status, Description)

def onMessage(Connection, Data):
	global _plugin
	_plugin.onMessage(Connection, Data)

def onCommand(Unit, Command, Level, Hue):
	global _plugin
	_plugin.onCommand(Unit, Command, Level, Hue)

def onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile):
	global _plugin
	_plugin.onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile)

def onDisconnect(Connection):
	global _plugin
	_plugin.onDisconnect(Connection)

def onHeartbeat():
	global _plugin
	_plugin.onHeartbeat()

# Generic helper functions
def LogMessage(Message):
	if Parameters["Mode6"] == "File":
		f = open(Parameters["HomeFolder"]+"http.html","w")
		f.write(Message)
		f.close()
		Domoticz.Log("File written")

def DumpConfigToLog():
	for x in Parameters:
		if Parameters[x] != "":
			Domoticz.Status( "'" + x + "':'" + str(Parameters[x]) + "'")
	Domoticz.Status("Device count: " + str(len(Devices)))
	for x in Devices:
		Domoticz.Status("Device:		   " + str(x) + " - " + str(Devices[x]))
		Domoticz.Status("Device ID:		  '" + str(Devices[x].ID) + "'")
		Domoticz.Status("Device Name:	  '" + Devices[x].Name + "'")
		Domoticz.Status("Device nValue:	   " + str(Devices[x].nValue))
		Domoticz.Status("Device sValue:	  '" + Devices[x].sValue + "'")
		Domoticz.Status("Device LastLevel: " + str(Devices[x].LastLevel))
	return

def DumpHTTPResponseToLog(httpDict):
	if isinstance(httpDict, dict):
		Domoticz.Status("HTTP Details ("+str(len(httpDict))+"):")
		for x in httpDict:
			if isinstance(httpDict[x], dict):
				Domoticz.Status("--->'"+x+" ("+str(len(httpDict[x]))+"):")
				for y in httpDict[x]:
					Domoticz.Status("------->'" + y + "':'" + str(httpDict[x][y]) + "'")
			else:
				Domoticz.Status("--->'" + x + "':'" + str(httpDict[x]) + "'")
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

@EscApe,

The WebSocket protocol seems pretty straight forward, I will have a look at adding support for it.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
febalci
Posts: 331
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by febalci »

I remember exploding my brains out doing the websocket connection as a domoticz plugin for the EMSC eartquake db. It never worked stable, since the EMSC db and connecton were not stable at all. But on intranet should be stable enough. I also remember using Wireshark to understand the packets structure a lot and it was really useful. I highly suggest using wireshark.

It'a been some time but as far as i remember from the websockets definiton, there was 2 types of message which depends on the size of the message. The first one was only 1 packet (as in earthquake example), the second one was the message broken into parts if it exceeds some certain size. And also, if it is broken, the subsequent messages will not have the header, because the header was sent in the first message. I didn't bother to solve multi packet messages since EMSC was sending only 1 packet size messages. Yet again, it's been some time and i might not be remembering correctly; i am an old guy :D
EscApe
Posts: 540
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2026.3
Location: The Netherlands
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by EscApe »

@Dnpwwo

That would be great! If i can help by testing stuff (also supplying tcpdumps) let me know. Looking forward to tinkering with it.
Thorgal789
Posts: 919
Joined: Wednesday 15 August 2018 14:38
Target OS: -
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Thorgal789 »

I have a working Websocket in my plugin https://github.com/Smanar/Domoticz-deCO ... /plugin.py and using framework fonctions (with the febalci code too)
But yes, I have too so much problem with it. Realy so much problems
1 - The only working protocol is none, other make freeze, because the framework wait for missing data.
2 - I have random error message like "Error: (deCONZ): Socket Shutdown Error: 9, Bad file descriptor" because synchronisation problem.I need to force the socket to close and sometime, domoticz close it before me (and If I don't force it, it don't close it)
3 - I have too sometime "Error: EventSystem: Malformed action sequence!" and ofc, if I use another application for websocket, all will be fine.
4 - With protocol = "None" the body can exceed 4096 byte (as said by felbaci) so I m using Connection.BytesTransferred() fonction to make "magic" (not documented fonction).

The connexion library is bugged as hell, but this code try to make with its bugs.

There is too a fonction to extract JSON on body according to header in the fonction.py file, But i m just using string.find() python fonction to parse body in my code.

And I m not sure, but on last beta version, you can use python request library, will be usefull to avoid build-in connexion library.
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

Just an update on this.

I've been working on it for the last week and hope to have an initial version of WebSocket support built into the Domoticz beta in the next few days.

Just need to get the data masking right and finish a sample plugin that uses it
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
febalci
Posts: 331
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by febalci »

Dnpwwo wrote: Friday 11 January 2019 0:33 Just an update on this.

I've been working on it for the last week and hope to have an initial version of WebSocket support built into the Domoticz beta in the next few days.

Just need to get the data masking right and finish a sample plugin that uses it
Hi @dnpwwo,

Is there any update on this? Thanks for your hard work and the excellent plugin system. ..
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

Sadly not.

I'm slightly stumped. I've had the websocket upgrade working for a long time and the messages seem perfect when compared to a test web site using wireshark but the server keeps closing the connection rather than responding which is drivig me nuts :?

I can push the code I have if you are interested in helping work out where the issue it.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
febalci
Posts: 331
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by febalci »

Dnpwwo wrote: Sunday 03 March 2019 11:34 I can push the code I have if you are interested in helping work out where the issue it.
Dear @Dnpwwo,thanks a lot for your efforts. I will do my best to help if i can; whether to find out if it is related to keepalive function or with a problem with the site you are testing it with..
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

I pushed the update yesterday so if you build from source you can get it now otherwise it will (should?) be in the latest beta.

My test plugin looks like this:

Code: Select all

# WebSocket client example
#
# Author: Dnpwwo, 2019
#
#   Demonstrates WS:// connectivity using http://demos.kaazing.com/echo/
#
"""
<plugin key="WSClient" name="WebSocket client example" author="Dnpwwo" version="1.0.0" externallink="https://en.wikipedia.org/wiki/WebSocket">
    <description>
        <h2>WebSocket client example</h2><br/>
        Establishes an initial HTTP connection to the server, then upgrades to WebSockets.
    </description>
    <params>
        <param field="Address" label="IP Address" width="200px" required="true" default="demos.kaazing.com"/>
        <param field="Mode1" label="Protocol" width="75px">
            <options>
                <option label="WSS" value="443"/>
                <option label="WS" value="80"  default="true" />
            </options>
        </param>
        <param field="Mode6" label="Debug" width="150px">
            <options>
                <option label="None" value="0"  default="true" />
                <option label="Python Only" value="2"/>
                <option label="Basic Debugging" value="62"/>
                <option label="Basic+Messages" value="126"/>
                <option label="Connections Only" value="16"/>
                <option label="Connections+Python" value="18"/>
                <option label="Connections+Queue" value="144"/>
                <option label="All" value="-1"/>
            </options>
        </param>
    </params>
</plugin>
"""
import Domoticz
from random import randint

class BasePlugin:
    webSockConn = None
    runAgain = 6
    sProtocol = "WS"
   
    def __init__(this):
        return

    def onStart(this):
        if Parameters["Mode6"] != "0":
            Domoticz.Debugging(int(Parameters["Mode6"]))
            DumpConfigToLog()
        if (Parameters["Mode1"] == "443"): this.sProtocol = "WSS"
        this.webSockConn = Domoticz.Connection(Name=this.sProtocol+" Test", Transport="TCP/IP", Protocol=this.sProtocol, Address=Parameters["Address"], Port=Parameters["Mode1"])
        this.webSockConn.Connect()

    def onStop(this):
        Domoticz.Log("onStop - Plugin is stopping.")

    def onConnect(this, Connection, Status, Description):
        if (Status == 0):
            Domoticz.Debug("Server connected successfully.")
            sendData = { 'URL'  : '/echo',
                         'Verb': 'GET',
                         'Headers' : {  'Host': Parameters["Address"],
                                        'User-Agent': 'Domoticz/1.0',
                                        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                                        'Accept-Language': 'en-GB,en;q=0.5',
                                        'Accept-Encoding': 'gzip, deflate',
                                        'Origin': 'http://'+Parameters["Address"],
                                        'Sec-WebSocket-Protocol': 'x-kaazing-handshake',
                                        'Sec-WebSocket-Extensions': 'permessage-deflate', 
                                        'Sec-WebSocket-Key': 'ha4Syz4tl11pVe+6DP8eAQ==', 
                                        'Sec-WebSocket-Version': '13', 
                                        'Connection': 'keep-alive, Upgrade',
                                        'Pragma': 'no-cache',
                                        'Cache-Control': 'no-cache',
                                        'Upgrade': 'websocket'
                                        }
                       }
            Connection.Send(sendData)
        else:
            Domoticz.Log("Failed to connect ("+str(Status)+") to: "+Parameters["Address"]+":"+Parameters["Mode1"]+" with error: "+Description)

    def onMessage(this, Connection, Data):
        DumpWSResponseToLog(Data)
        
        if "Status" in Data:
            Status = int(Data["Status"])
            if (Status == 101):
                Domoticz.Log("Good Response received from server, WebSocket upgrade complete.")
            elif (Status == 200):
                Domoticz.Log("Unexpected good Response received from server, Dropping connection.")
                this.webSockConn = None
            elif (Status == 302):
                Domoticz.Log("server returned a Page Moved Error.")
                sendData = { 'Verb' : 'GET',
                             'URL'  : Data["Headers"]["Location"],
                             'Headers' : { 'Content-Type': 'text/xml; charset=utf-8', \
                                           'Connection': 'keep-alive', \
                                           'Accept': 'Content-Type: text/html; charset=UTF-8', \
                                           'Host': Parameters["Address"]+":"+Parameters["Mode1"], \
                                           'User-Agent':'Domoticz/1.0' },
                            }
                Connection.Send(sendData)
            elif (Status == 400):
                Domoticz.Error("server returned a Bad Request Error.")
                this.webSockConn = None
            elif (Status == 500):
                Domoticz.Error("server returned a Server Error.")
                this.webSockConn = None
            else:
                Domoticz.Error("server returned a status: "+str(Data["Status"]))
                this.webSockConn = None

        elif "Operation" in Data:
            if Data["Operation"] == "Ping":
                rand = randint(0,2)
                if (rand == 0):
                    this.webSockConn.Send({'Operation':'Pong', 'Mask':939139389, 'Payload':'Hello'})
                elif (rand == 1):
                    this.webSockConn.Send({'Operation':'Pong', 'Mask':939139389, 'Payload':b'Hello'})
                else:
                    this.webSockConn.Send({'Operation':'Pong', 'Mask':939139389, 'Payload':bytearray([0x48, 0x65, 0x6c, 0x6c, 0x6f])})
                
                
    def onCommand(this, Unit, Command, Level, Hue):
        Domoticz.Debug("onCommand called for Unit " + str(Unit) + ": Parameter '" + str(Command) + "', Level: " + str(Level))

    def onDisconnect(this, Connection):
        Domoticz.Log("onDisconnect called for connection to: "+Connection.Address+":"+Connection.Port)

    def onHeartbeat(this):
        if (this.webSockConn != None and (this.webSockConn.Connecting() or this.webSockConn.Connected())):
            rand = randint(0,2)
            #if (rand == 0):
            #this.webSockConn.Send({'Mask':939139389, 'Payload':'Sent Text Message!'})
            this.webSockConn.Send({'Mask':0x19e32ee2, 'Payload':'Sent Text Message!'})
            Domoticz.Debug("[onHeartbeat] Sent Text Message!")
            #elif (rand == 1):
            #    this.webSockConn.Send({'Mask':939139389, 'Payload':b'Sent Bytes Message (binary)'})
            #    Domoticz.Debug("[onHeartbeat] Sent Bytes Message (binary)")
            #else:
            #    this.webSockConn.Send({'Mask':939139389, 'Payload':bytearray([0x48, 0x65, 0x6c, 0x6c, 0x6f])}) # Hello
            #    Domoticz.Debug("[onHeartbeat] Sent bytearray Message!")
        else:
            this.runAgain = this.runAgain - 1
            if this.runAgain <= 0:
                if (this.webSockConn == None):
                    this.webSockConn = Domoticz.Connection(Name=this.sProtocol+" Test", Transport="TCP/IP", Protocol=this.sProtocol, Address=Parameters["Address"], Port=Parameters["Mode1"])
                this.webSockConn.Connect()
                this.runAgain = 6
            else:
                Domoticz.Debug("onHeartbeat called, run again in "+str(this.runAgain)+" heartbeats.")

global _plugin
_plugin = BasePlugin()

def onStart():
    global _plugin
    _plugin.onStart()

def onStop():
    global _plugin
    _plugin.onStop()

def onConnect(Connection, Status, Description):
    global _plugin
    _plugin.onConnect(Connection, Status, Description)

def onMessage(Connection, Data):
    global _plugin
    _plugin.onMessage(Connection, Data)

def onCommand(Unit, Command, Level, Hue):
    global _plugin
    _plugin.onCommand(Unit, Command, Level, Hue)

def onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile):
    global _plugin
    _plugin.onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile)

def onDisconnect(Connection):
    global _plugin
    _plugin.onDisconnect(Connection)

def onHeartbeat():
    global _plugin
    _plugin.onHeartbeat()

# Generic helper functions
def LogMessage(Message):
    if Parameters["Mode6"] == "File":
        f = open(Parameters["HomeFolder"]+"http.html","w")
        f.write(Message)
        f.close()
        Domoticz.Log("File written")

def DumpConfigToLog():
    for x in Parameters:
        if Parameters[x] != "":
            Domoticz.Debug( "'" + x + "':'" + str(Parameters[x]) + "'")
    Domoticz.Debug("Device count: " + str(len(Devices)))
    for x in Devices:
        Domoticz.Debug("Device:           " + str(x) + " - " + str(Devices[x]))
        Domoticz.Debug("Device ID:       '" + str(Devices[x].ID) + "'")
        Domoticz.Debug("Device Name:     '" + Devices[x].Name + "'")
        Domoticz.Debug("Device nValue:    " + str(Devices[x].nValue))
        Domoticz.Debug("Device sValue:   '" + Devices[x].sValue + "'")
        Domoticz.Debug("Device LastLevel: " + str(Devices[x].LastLevel))
    return

def DumpWSResponseToLog(httpDict):
    if isinstance(httpDict, dict):
        Domoticz.Debug("WebSocket Details ("+str(len(httpDict))+"):")
        for x in httpDict:
            if isinstance(httpDict[x], dict):
                Domoticz.Debug("--->'"+x+" ("+str(len(httpDict[x]))+"):")
                for y in httpDict[x]:
                    Domoticz.Debug("------->'" + y + "':'" + str(httpDict[x][y]) + "'")
            else:
                Domoticz.Debug("--->'" + x + "':'" + str(httpDict[x]) + "'")
and it seems to create connect and upgrade to websockets but the connection is gracefully closed by the server after a message is sent (that should be echoed).

Any thoughts appreciated :oops:
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
febalci
Posts: 331
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by febalci »

Dnpwwo wrote: Tuesday 05 March 2019 5:21 I pushed the update yesterday so if you build from source you can get it now otherwise it will (should?) be in the latest beta.
and it seems to create connect and upgrade to websockets but the connection is gracefully closed by the server after a message is sent (that should be echoed).

Any thoughts appreciated :oops:
Dear @Dnpwwo,

Specific to this echo server (Don't know really why), the websocket upgrade headers should have these:

Code: Select all

    def onConnect(this, Connection, Status, Description):
        if (Status == 0):
            Domoticz.Debug("Server connected successfully.")
            sendData = { 'URL'  : '/echo',
                         'Verb': 'GET',
                         'Headers' : {  'Host': Parameters["Address"],
                                        'User-Agent': 'Domoticz/1.0',
                                        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                                        'Accept-Language': 'en-GB,en;q=0.5',
                                        'Accept-Encoding': 'gzip, deflate',
                                        'Origin': 'http://'+Parameters["Address"],
                                        'Websocket-Protocol': '',
                                        'Sec-WebSocket-Protocol': '',
                                        'Sec-WebSocket-Extensions': 'x-kaazing-idle-timeout, x-kaazing-ping-pong, x-kaazing-http-revalidate', 
                                        'Sec-WebSocket-Key': 'ha4Syz4tl11pVe+6DP8eAQ==', 
                                        'Sec-WebSocket-Version': '13', 
                                        'Connection': 'keep-alive, Upgrade',
                                        'Pragma': 'no-cache',
                                        'Cache-Control': 'no-cache',
                                        'Upgrade': 'websocket'
                                        }
                       }
            Connection.Send(sendData)
        else:
            Domoticz.Log("Failed to connect ("+str(Status)+") to: "+Parameters["Address"]+":"+Parameters["Mode1"]+" with error: "+Description)
As seen above, the 'Websocket-Protocol' and 'Sec-Websocket-Protocol' should be '' and 'Sec-Websocket-Extensions' should be 'x-kaazing-idle-timeout, x-kaazing-ping-pong, x-kaazing-http-revalidate'. I realized it in the Wireshark log.

So now, the echo server works... I am gonna try other websocket servers like the earthquake one; if any bugs comes out i will inform you. Btw, the Mask parameter in:

Code: Select all

this.webSockConn.Send({'Mask':0x19e32ee2, 'Payload':'Sent Text Message!'})
is not accepting higher integers like 0xe7b23f07; converts them to 0xffffffff directly....

UPDATE:
I tried the connection with the EMSC Earthquake near real notification web site: http://www.seismicportal.eu/realtime.html That is a websocket connection with keepalive function (I keep sending ping every heartbeat to maintain this), and whenever an earthquake occurs, the server sends Json data regarding the details. The connection stays alive but whenever i get a json message, the plugin terminates itself:

Code: Select all

2019-03-05 14:23:25.624 (test) Sending 6 bytes of data
2019-03-05 14:23:25.625 (test) 89 80 37 fa 21 3d .. .. .. .. .. .. .. .. .. .. .. .. .. .. ��7�!=
2019-03-05 14:23:25.714 (test) Pushing 'ReadEvent' on to queue
2019-03-05 14:23:25.726 (test) Processing 'ReadEvent' message
2019-03-05 14:23:25.726 (test) Received 2 bytes of data
2019-03-05 14:23:25.726 (test) 8a 00 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. �.
2019-03-05 14:23:25.726 (test) Pushing 'onMessageCallback' on to queue
2019-03-05 14:23:25.726 (test) Processing 'onMessageCallback' message
2019-03-05 14:23:25.727 (test) Calling message handler 'onMessage'.
2019-03-05 14:23:25.727 (test) WebSocket Details (1):
2019-03-05 14:23:25.727 (test) --->'Operation':'Pong'
2019-03-05 14:23:27.185 (test) Pushing 'ReadEvent' on to queue
2019-03-05 14:23:27.192 (test) Processing 'ReadEvent' message
2019-03-05 14:23:27.192 (test) Received 325 bytes of data
2019-03-05 14:23:27.192 (test) c1 7e 01 41 7c 92 d1 6a 83 30 14 86 ef fb 14 e1 5c 6b 88 4d �~.A|��j�0.���.�\k�M
2019-03-05 14:23:27.194 (test) 6b ad 77 a5 73 43 68 bb a1 42 61 63 94 a0 59 27 53 23 69 1c k�w�sCh��Bac��Y'S#i.
2019-03-05 14:23:27.195 (test) 94 d2 77 5f a2 a6 ac bb 58 2e 42 fe ef 9c 9c 9c 3f c9 05 58 ��w_����X.B��?�.X
2019-03-05 14:23:27.197 (test) ae 4a d1 40 08 b9 e4 4c 71 70 a0 60 8a 41 78 99 20 04 47 2e �J�@.��Lqp�`�Ax�..G.
2019-03-05 14:23:27.198 (test) 6a ae e4 19 42 64 b4 26 ea dc 72 ad e0 45 94 8d 02 67 80 b9 j��.Bd�&��r��E��.g��
2019-03-05 14:23:27.199 (test) 10 b2 28 1b bd fd a4 63 6f 3d 44 c8 f5 29 f6 7c 67 54 5e 80 .�(.���co=D��)�|gT^�
2019-03-05 14:23:27.200 (test) c9 dc 0a 37 58 62 d2 af df f5 7c 35 f8 56 f9 51 b7 d1 49 de ��.7Xbү��|5�V�Q��I�
2019-03-05 14:23:27.201 (test) d7 86 b2 30 68 4a bc 25 a1 84 1e 88 1e 1e 5d 0e b1 56 8a 96 ׆�0hJ�%��.�..].�V��
2019-03-05 14:23:27.202 (test) 4b 55 f6 87 8e ed 55 ec a4 ba b6 30 46 c6 7d 2e a1 2e 99 67 KU����U줺�0F�}.�.�g
2019-03-05 14:23:27.203 (test) 9e 17 4e 69 48 08 26 af b6 eb 9a 1d ed 99 75 65 21 ff b6 ec �.NiH.&���.�ue!���
2019-03-05 14:23:27.204 (test) 8b 5b 56 99 fb b9 b3 03 ac 53 9f 26 69 17 c5 eb 5b 1a 53 1a �[V����.�S�&i.��[.S.
2019-03-05 14:23:27.204 (test) fd f2 09 05 6f fb 34 e3 76 44 5d f3 8f a5 a1 29 1d a6 38 18 ��..o�4�vD]󏥡).�8.
2019-03-05 14:23:27.205 (test) b5 2a eb 7b 2b 34 23 7e e8 05 da 0d 0e 6e 56 4e a2 93 39 3f �*�{+4#~�.�..nVN��9?
2019-03-05 14:23:27.206 (test) 0c a5 17 b3 60 31 f3 ff 84 72 fd a8 95 30 b5 21 da a6 6b 37 .�.�`1���r���0�!ڦk7
2019-03-05 14:23:27.207 (test) c9 52 9b f1 51 9d 9b e6 20 f9 b1 ff 07 08 d2 0c a3 ed 2a c9 �R��Q���.���..�.��*�
2019-03-05 14:23:27.209 (test) e2 1d 4a a2 a7 f8 79 e7 a0 4d 14 ed 57 c9 03 8a d3 0d 06 f3 �.J���y�M.�W�.��..�
2019-03-05 14:23:27.209 (test) 62 93 eb f5 07 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. b���.
2019-03-05 14:23:27.209 Error: PluginSystem: Exception processing message. 
2019-03-05 14:24:30.239 Error: test hardware (2) thread seems to have ended unexpectedly
The incoming message is this:

Code: Select all

Line-based text data (27 lines)
    {"action":"create","data":{\n
      "geometry": {\n
        "type": "Point",\n
        "coordinates": [\n
          -97.55,\n
          38.87,\n
          -5.0\n
        ]\n
      },\n
      "type": "Feature",\n
      "id": "20190303_0000140",\n
      "properties": {\n
        "lastupdate": "2019-03-05T11:30:00.0Z",\n
        "magtype": "mb",\n
        "evtype": "ke",\n
        "lon": -97.55,\n
        "auth": "NEIC",\n
        "lat": 38.87,\n
        "depth": 5.0,\n
        "unid": "20190303_0000140",\n
        "mag": 2.5,\n
        "time": "2019-03-03T22:30:48.9Z",\n
        "source_id": "748749",\n
        "source_catalog": "EMSC-RTS",\n
        "flynn_region": "KANSAS"\n
      }\n
    }}
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

@febalci,

I tried a couple of HTTP header combinations because I was suspicious that it was the problem but after looking at Wireshark didn't manage to crack it. Pretty weird behavior that it reports the upgrade successful but then shuts the connection when you try and use it. Anyway, at least we know now. I will will update my test plugin and try some of the other message types particularly around binary data.

I'll add some exception handling to the code and try and reproduce the crash.

Thanks for your help with this :!:

Edit:
Looked at the message and derived the following:
  • the 325 bytes received make up a complete message
  • the message contains a text frame
  • the payload is not masked
  • the text payload is 321 bytes long
what is interesting is that although the header says it is not masked and the payload length (and actual size) would agree that it is not masked the data in you post clearly looks masked (certainly does not look like text)
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
febalci
Posts: 331
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by febalci »

Dnpwwo wrote: Wednesday 06 March 2019 3:50
Looked at the message and derived the following:
  • the 325 bytes received make up a complete message
  • the message contains a text frame
  • the payload is not masked
  • the text payload is 321 bytes long
what is interesting is that although the header says it is not masked and the payload length (and actual size) would agree that it is not masked the data in you post clearly looks masked (certainly does not look like text)
Dear @Dnpwwo,

actually it is unmasked, however in compressed gzip format. I really don't know in which phase the packet should be deflated altough it uses gzip,deflate in the http header. Wireshark can show the payload as decompressed payload and also i can decompress it using:

Code: Select all

#!/usr/bin/env python3
import zlib

websocket_payload_packet_1 = bytes.fromhex("""
7c92cb6a84301486f7f31472d61aa251bcec86610aa57428d5554b9154532b5523311664f0dd9ba8914e17cd22e47ce7fa27b9022d64cd3b48a0108c4a063694545248ae07cb828af19649314162695b1139f54c59f0c4eb4e82bdc2827351d69d4a1f94ef758196e5c4210a027bb34884a2d0184e80f0727c53fbace95ef84e4d310ab69486bad4c8c36e8c09263956cbf5f1eaeb05ef9990f5d2739baea1831cfb52ebd8f21c4c1c1c64ae9b109c608cf08b19baa595e9d9be1bc8be0dfb628635fa7a6ed4001de5a70eba9cef4f7b18950afd920925eb973025762363f78fa27526e5f690e923ebf65609c93c4f2bf12314ef4a063e8a82e56be9d08f423ffee32ad493365cd786f3637a729eb3d4447c3453d7e58255cb2fb0e0e178498f29e87739ccf30f00
""".replace("\n", ""))


# Data from frame 1
data = websocket_payload_packet_1



z = zlib.decompressobj(wbits=-15)
out = z.decompress(data)
print(out)
User avatar
Dnpwwo
Posts: 820
Joined: Sunday 23 March 2014 9:00
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by Dnpwwo »

@febalci,

Can you remove the gzip header? My test script still works without it.

Code: Select all

'Accept-Encoding': 'gzip, deflate'
that might help work out the problem.

I suspect it is the string conversion in Python that is aborting.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw
pipiche
Posts: 2035
Joined: Monday 02 April 2018 20:33
Target OS: Linux
Domoticz version:
Contact:

Re: Getting WebSockets working in plugin framework Domoticz.Connection

Post by pipiche »

@Dnpwwo, interested about the great work you are doing here, if I'm not mistaken this is the implementation of the client side, where basically the plugin will use the websocket to exchange with a Server , correct ?

If so, do you hae any plan as you have the HTTP server, to get also the WebSocket Server side supported ? In other term, the plugin would be able to share async messages with a browser
Post Reply