Python Plugin - listening on HTTP

[pipiche]

I have some trouble while listening on HTTP and serving files.
While using the loopback via tunneling ( 127.0.0.1 ) everything works fine, but as soon as I'm using the IP address, some files seems to be not correctly serve. It seems that it is related to the size of the file ( 500KBytes) .

Is there any limitation on the HTTP protocol in the Python plugin framework ?

CC: @Dnpwwo

I have been able to create a small plugin with 3.9M of pages ( htm, css, json, and other stuff generated by Angular )

[Dnpwwo]

Nothing explicit but either Python or C++ may not like huge bits of data.

How are you sending it? If it is big you should chunk the file and stagger the data transmit other wise you can flood the net work. The HTTP protocol support HTTP chunking (documented on the wiki). All modern browsers support it as well.

If you post a code snippet I can take a look

[pipiche]

The code is pretty simple

1) receive the GET request with a path
2) opening the path, reading the file
3) sending back the content of the file to the client

Code: Select all

    
   def onMessage(self, Connection, Data):
        Domoticz.Log("onMessage called for connection: "+Connection.Address+":"+Connection.Port)
        DumpHTTPResponseToLog(Data)

        homedirectory = Parameters["HomeFolder"]
        # Incoming Requests
        headerCode = "200 OK"
        if (not 'Verb' in Data):
            Domoticz.Error("Invalid web request received, no Verb present")
            headerCode = "400 Bad Request"
        elif (Data['Verb'] not in ( 'GET', 'PUT', 'POST', 'DELETE')):
            Domoticz.Error("Invalid web request received, only GET requests allowed ("+Data['Verb']+")")
            headerCode = "405 Method Not Allowed"
        elif (not 'URL' in Data):
            Domoticz.Error("Invalid web request received, no URL present")
            headerCode = "400 Bad Request"
        elif not os.path.exists(  homedirectory + 'www' + Data['URL']):
            Domoticz.Error("Invalid web request received, file '"+ homedirectory + 'www' +Data['URL'] + "' does not exist")
            headerCode = "404 File Not Found"
        if (headerCode != "200 OK"):
            DumpHTTPResponseToLog(Data)
            Connection.Send({"Status": headerCode})
            return
        # We are ready to send the response
        _response = {}
        _response["Headers"] = {}
        _response["Headers"]["Connection"] = "Keealive"
        _response["Headers"]["Cache-Control"] = "no-cache, no-store, must-revalidate, max-age=0"
        _response["Headers"]["Pragma"] = "no-cache"
        _response["Headers"]["Expires"] = "0"
        _response["Headers"]["User-Agent"] = "Plugin-Zigate"
        _response["Headers"]["Server"] = "Domoticz"
        _response["Headers"]["Accept-Range"] = "none"

        #webFilename = self.homedirectory +'www'+Data['URL']
        webFilename = homedirectory +'www'+ Data['URL']
        with open(webFilename , mode ='rb') as webFile:
            _response["Data"] = webFile.read()
        Domoticz.Log("Reading file: %.40s len: %s" %(_response["Data"], len(_response["Data"])))

        _contentType, _contentEncoding = guess_type( Data['URL'] )
        Domoticz.Log("MimeType: %s, Content-Encoding: %s " %(_contentType, _contentEncoding))

        if _contentType:
            _response["Headers"]["Content-Type"] = _contentType
        if _contentEncoding:
            _response["Headers"]["Content-Encoding"] = _contentEncoding

        _response["Status"] = "200 OK"
        Connection.Send( _response )
  

[pipiche]

I'll have a look to what you do with Google-Plugin

[pipiche]

NOt sure that we can split pages into range. I have implemented what you have done for the Google-Plugin, but when client is requesting a JS file, it looks like it expects to get it in once, as the ffirst chunk is receive but I'm not getting any incoming request for the next range

[Dnpwwo]

@pipiche,

There are two ways of breaking large files down into smaller pieces in HTTP, one controlled by the client and one by the server.

The Google plugin uses 206 messages to send partial responses. That technique relies on the client to then control the flow by responding with subsequent requests that include the 'Range' header to indicate what to send next. I've only used it for MP3 files and it seems pretty robust.

The other technique is to use 'chunked transfer encoding' which is driven 100% by the server because all the data is return in a single response but that response is broken into 'chunks'. Domoticz also supports this and it is documented on the wiki: https://www.domoticz.com/wiki/Developin ... ol_Details. I strongly suspect you will be the first person to use this outside my testing so you may need to play around with it.

[pipiche]

@pipiche,

There are two ways of breaking large files down into smaller pieces in HTTP, one controlled by the client and one by the server.

The Google plugin uses 206 messages to send partial responses. That technique relies on the client to then control the flow by responding with subsequent requests that include the 'Range' header to indicate what to send next. I've only used it for MP3 files and it seems pretty robust.

The other technique is to use 'chunked transfer encoding' which is driven 100% by the server because all the data is return in a single response but that response is broken into 'chunks'. Domoticz also supports this and it is documented on the wiki: https://www.domoticz.com/wiki/Developin ... ol_Details. I strongly suspect you will be the first person to use this outside my testing so you may need to play around with it.

I'm came to the conclusion that Range indeed rely on the client asking for partial sent. So Chunk is probably the best. On the other, I'm not sure this is my issue, as when I try to load the large file directly it works.
Will continue to investigate, and for sure will try the Chunk stuff

[zak45]

I'm came to the conclusion that Range indeed rely on the client asking for partial sent. So Chunk is probably the best. On the other, I'm not sure this is my issue, as when I try to load the large file directly it works.
Will continue to investigate, and for sure will try the Chunk stuff

Have seen also this problem now on my linux test box. Not have same limitations as in Windows ...

One possible way to investigate could be :
"Content-Length": "xxxxx"
if you put a number smaller than your page, you will see only a partial view...
if you let the framework do the job, looks like in some cases (not able to find when this happend for the moment), this lenght should not have right value ..?? !!!

Take that only as supposition ...

[pipiche]

@Dnpwwo what is your view on gzip compression ? Should it be done in the plugin, or inside the Framework ? That would save a lot! as for instance my 500KB file is a pure Java Script

[pipiche]

gzip and chunk implemented ! Will see tonight if it fixed my initial issue, but for sure it save bandwith

[pipiche]

@Dnpwwo, I do confirm that it is solving the issue. Chunk is the right approach, gzip is improving also.

[zak45]

this is my test result:

listener created OK:

Code: Select all

        httpServerConn = Domoticz.Connection(Name="EZJWebServer", Transport="TCP/IP", Protocol="HTTP", Port=Parameters["Mode1"])
        httpServerConn.Listen()
        Domoticz.Log(_("Listen on EZJWebserver - Port: {}").format(Parameters['Mode1']))

command to send data to the client OK:

Code: Select all

                    Connection.Send({"Status":status, 
                                    "Headers": {"Connection": "keep-alive", 
                                                "Accept": "Content-Type: text/html; charset=UTF-8",
                                                "Access-Control-Allow-Origin":"http://" + Parameters['Address'] + ":" + Parameters['Port'] + "",
                                                "Cache-Control": "no-cache, no-store, must-revalidate",
                                                "Content-Type": "text/html; charset=UTF-8",
                                                "Pragma": "no-cache",
                                                "Expires": "0"},
                                    "Data": htmldata})

No "Content-Length" Key, so framework would calculate it.

Generated HTML code to send to the client with the variable OK:

Code: Select all

                            bit =("e"*6)
                            Domoticz.Log('Bit len: ' + str(len(bit)))

                            htmldata = '''<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"/>
<link rel="icon" href="data:,"/>
<title>My test Editor</title>
</head>
<body>
<div>'''+bit+'''</div>
</body>
</html>
'''

if we put 'bit =("e"*6)', client receive all informations and this is the source code of the received HTML page, all is OK:

Code: Select all

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"/>
<link rel="icon" href="data:,"/>
<title>My test Editor</title>
</head>
<body>
<div>eeeeee</div>
</body>
</html>

if we put 'bit =("é"*6)', client receive all informations minus 6 characters and this is the source code of the received HTML page, page on Error:

Code: Select all

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"/>
<link rel="icon" href="data:,"/>
<title>My test Editor</title>
</head>
<body>
<div>éééééé</div>
</body>
</

Problem should be related to French special characters, these ones need more than one byte and probably made wrong calculation of "Content-Length".
This made some trouble when want to dynamicaly generate HTML page with French special characters. Depend of how much they are, HTML page is truncated and can become unusable.

Let me know if you come to same result.

Version: 4.10731
Build Hash: 5f335818
Compile Date: 2019-05-11 10:44:53
dzVents Version: 2.4.19
Python Version: 3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 15:51:26) [MSC v.1900 32 bit (Intel)]

[Dnpwwo]

@zak45,

Multibyte characters are probably the problem. The logic to determine content length is:

Code: Select all

	Py_ssize_t iLength = 0;
	if (PyUnicode_Check(pData))
		iLength = PyUnicode_GetLength(pData);
	else if (pData->ob_type->tp_name == std::string("bytearray"))
		iLength = PyByteArray_Size(pData);
	else if (PyBytes_Check(pData))
		iLength = PyBytes_Size(pData);
	sHttp += "Content-Length: " + std::to_string(iLength) + "\r\n";

the Python Unicode type doesn't have a 'Size' function (it has some deprecated macros but that is all). GetLength is correctly returning the number characters but that is not what we need here.

To keep going you could force your HTML to be a ByteArray and it should work.

What does it do if you inject the content length header with the correct number? Does it work then?

I'll need to do some more research on this.

[zak45]

Multibyte characters are probably the problem.
To keep going you could force your HTML to be a ByteArray and it should work.

What does it do if you inject the content length header with the correct number? Does it work then?

This is what I have done to have it working:

calculate the length of the HTML page:

Code: Select all

lenhtml = len(htmldata.encode('utf-8'))

Use the result into headers:

Code: Select all

Connection.Send({"Status":status, 
                                    "Headers": {"Connection": "keep-alive", 
                                                "Accept": "Content-Type: text/html; charset=UTF-8",
                                                "Access-Control-Allow-Origin":"http://" + Parameters['Address'] + ":" + Parameters['Port'] + "",
                                                "Cache-Control": "no-cache, no-store, must-revalidate",
                                                "Content-Length": ""+str(lenhtml)+"",
                                                "Content-Type": "text/html; charset=UTF-8",
                                                "Pragma": "no-cache",
                                                "Expires": "0"},
                                    "Data": htmldata})

Hope this help.

[pipiche]

I'm getting a very strange and bad behaviour since I've move to the latest beta:

(1) Chunk doesn't work and I'm getting connection closed , without any request from my end, and Keep-alive header is well positioned.

Seems that the "Queued asyncronous read aborted " is triggering it!!!!

Code: Select all

May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) Queued asyncronous read aborted (10.0.0.15:59087).
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data

Code: Select all

May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) Disconnect event received for '10.0.0.15:59087'.
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) onDisconnect: Name: '10.0.0.15:59087', Transport: 'TCP/IP', Protocol: 'HTTP', Address: '10.0.0.15', Port: '59087', Baud: 0, Connected: False, Parent: 'Zigate Server Connection'
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) onDisconnect Name: '10.0.0.15:59087', Transport: 'TCP/IP', Protocol: 'HTTP', Address: '10.0.0.15', Port: '59087', Baud: 0, Connected: False, Parent: 'Zigate Server Connection'
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59067'.
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59074'.
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59087'.
May 20 12:04:03 rasp domoticz[17717]: (Zigate-DEV) Deallocating connection object '10.0.0.15:59087' (10.0.0.15:59087).

And other case.

Code: Select all

May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Queued asyncronous read aborted (10.0.0.15:59088).
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Sending 1031 bytes of data

Code: Select all

May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Disconnect event received for '10.0.0.15:59088'.
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Received 463 bytes of data
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) onDisconnect: Name: '10.0.0.15:59088', Transport: 'TCP/IP', Protocol: 'HTTP', Address: '10.0.0.15', Port: '59088', Baud: 0, Connected: False, Parent: 'Zigate Server Connection'
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) onDisconnect Name: '10.0.0.15:59088', Transport: 'TCP/IP', Protocol: 'HTTP', Address: '10.0.0.15', Port: '59088', Baud: 0, Connected: False, Parent: 'Zigate Server Connection'
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59067'.
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59074'.
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59087'.
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) --> 10.0.0.15:59088'.
May 20 12:04:02 rasp domoticz[17717]: (Zigate-DEV) Deallocating connection object '10.0.0.15:59088' (10.0.0.15:59088).

(2) I still feel that something is wrong with large files and multiple connections.
If I GET each large file one by one , I get them
If then I try to load the page (which is then calling those large file), it hangs ! In this case, I'm chucking in small piece, but this is adding a load on the queuing mechanism , which might be the case.



When using loopback, everything works well, so I would consider the plugin logic correct

[Dnpwwo]

@pipiche,

I haven't changed anything in the protocols or transports recently so the lastest beta shouldn't have changed anything.

Is there a version of your plugin on github I can have a look at?

In my (limited) experience read aborts seem to happen if the client is flooded and slowing down data can help. Perhaps I need to make the Delay parameter accept floats to throttle the sends a little.

[pipiche]

I’ll create one so you will have only the interesting part


Envoyé de mon iPhone en utilisant Tapatalk

[pipiche]

@pipiche,

I haven't changed anything in the protocols or transports recently so the lastest beta shouldn't have changed anything.

Is there a version of your plugin on github I can have a look at?

In my (limited) experience read aborts seem to happen if the client is flooded and slowing down data can help. Perhaps I need to make the Delay parameter accept floats to throttle the sends a little.

Really appreciate your support: Here is the gitHub
https://github.com/pipiche38/HTTPserver

You start the plugin and it will listen on port 9440 for http request.
Then from the browser select http://host:port/index.html

That is it !

[pipiche]

@pipiche,

I haven't changed anything in the protocols or transports recently so the lastest beta shouldn't have changed anything.

Is there a version of your plugin on github I can have a look at?

In my (limited) experience read aborts seem to happen if the client is flooded and slowing down data can help. Perhaps I need to make the Delay parameter accept floats to throttle the sends a little.

It could be related to latest Domoticz with new Boost !

[pipiche]

@Dnpwwo, I saw in the PluginTransports.h a m_Buffer of 4096 bytes, would that mean that there is a limitation to send not more than 4K per Send request ?
If saw, that is ok, but we should get an erreor back if we are trying to seen a bigger size, no ?