Python Plugin - listening on HTTP

[pipiche]

While invetsigating my issue arround Connection aborted from the Framework, I have the feeling this is due to multiple connection at the same time, as indeed if I try to download an large JavaScript file alone, I have no issue, but as soon as the browser is loading a pages calling doing paralell loading I have that issue.

So it might me a mixed between chunk an pipeline connections. I don't know how you do manage it inside the framework.

Last I'm pretty convinced that the issue is coming since I have move to latest domoticz with boost 1.70

[Dnpwwo]

@pipiche,

The 4K buffer is used for incoming data not outgoing so this is not the problem.

I had a look at your HTTPServer code, the 'Delay=1' probably not working as you expect. It will delay a minimum of '1 second' from when you queu the response. Because you are queuing them in a tight loop they will be processed after 1 second pretty much one after the other and can still flood the link.

Not sure about boost version, I saw something similar when sending files to a Google Home Mini. I just didn't have enough info to work out if the Mini was actually aborting the connection or not.

[pipiche]

@Dnpwwo, I saw in the PluginTransports.h a m_Buffer of 4096 bytes, would that mean that there is a limitation to send not more than 4K per Send request ?
If saw, that is ok, but we should get an erreor back if we are trying to seen a bigger size, no ?

Did test on my prod environment with an older version of Domoticz/Boots and it works. No flooding, no queue abort!!

Code: Select all

DomoticzVersion: 4.10678
DomoticzHash: 52de7fec
boost version 1.68

So either boost 1.70 is more efficient and then we get in a race condition which was not happening on 1.68, or something has been introduced in Domoticz or Boost 1.68 which create that issue.


However, I need to break the chunk to 2 Kbytes in order to get it working, otherwise it stucks. So I still feel that there is something wrong when sending multiple session (which is the case when you are opening the index.html) and which is not the case when you download the file alone and here there is no chunk required and you can get large file !

[Dnpwwo]

@pipiche,

With a lot of debugging on it worked for me which also points to a flooding issue.

I've got a few ideas to make it better in Python in the short term, I'll play with it tomorrow

[pipiche]

@pipiche,

With a lot of debugging on it worked for me which also points to a flooding issue.

I've got a few ideas to make it better in Python in the short term, I'll play with it tomorrow

Excellent, let me know if I can help.

However I believe the flooding is triggered by the chunk mode, if I do not use chunk mode large file doesn’t go through if there are multi-sessions


Envoyé de mon iPhone en utilisant Tapatalk

[Dnpwwo]

@pipiche,

My Python ideas made things better but did not solve it completely.

I've looked at the connections overall by updating your plugin's debugging to show 'Connections Only'

Code: Select all

        <param field="Mode6" label="Debug" width="150px">
            <options>
                <option label="None" value="0"  default="true" />
                <option label="Python Only" value="2"/>
                <option label="Basic Debugging" value="62"/>
                <option label="Basic+Messages" value="126"/>
                <option label="Connections Only" value="16"/>
                <option label="Connections+Queue" value="144"/>
                <option label="All" value="-1"/>
            </options>
        </param>

and can see that the Connection gets an End-of-File notification from Boost. The queued read interprets that as the other end of the connection disconnecting (which is normally the case) which triggers the server side cleanup (disconnection). Subsequent write operations therefore have nowhere to write to and you get

2019-05-25 13:06:25.026 Error: (Test Website) No transport, write directive to '127.0.0.1:64483' ignored.

as a defensive measure by the Plugin Framework. So far, so good. I understand the symptoms but what causes the spurious 'End-of-File' error in the first place?

The smoking gun potentially comes from the Boost documentation for the 'write_some' function the Plugin Framework uses:

The write_some operation may not transmit all of the data to the peer.

and more directly from the description

The function call will block until one or more bytes of the data has been written successfully, or until an error occurs.

which could make it open to flooding although nothing states that an 'End-of-File' will be thrown explicitly if it is flooded.

I don't want to revert to fully synchronous writing because it will be slower in most circumstances but I can try using a per connection mutex to stop a second write operation commencing prior to the previous one completing.

[pipiche]

@pipiche,

My Python ideas made things better but did not solve it completely.

I've looked at the connections overall by updating your plugin's debugging to show 'Connections Only'

Code: Select all

        <param field="Mode6" label="Debug" width="150px">
            <options>
                <option label="None" value="0"  default="true" />
                <option label="Python Only" value="2"/>
                <option label="Basic Debugging" value="62"/>
                <option label="Basic+Messages" value="126"/>
                <option label="Connections Only" value="16"/>
                <option label="Connections+Queue" value="144"/>
                <option label="All" value="-1"/>
            </options>
        </param>

and can see that the Connection gets an End-of-File notification from Boost. The queued read interprets that as the other end of the connection disconnecting (which is normally the case) which triggers the server side cleanup (disconnection). Subsequent write operations therefore have nowhere to write to and you get

2019-05-25 13:06:25.026 Error: (Test Website) No transport, write directive to '127.0.0.1:64483' ignored.

as a defensive measure by the Plugin Framework. So far, so good. I understand the symptoms but what causes the spurious 'End-of-File' error in the first place?

The smoking gun potentially comes from the Boost documentation for the 'write_some' function the Plugin Framework uses:

The write_some operation may not transmit all of the data to the peer.

and more directly from the description

The function call will block until one or more bytes of the data has been written successfully, or until an error occurs.

which could make it open to flooding although nothing states that an 'End-of-File' will be thrown explicitly if it is flooded.

I don't want to revert to fully synchronous writing because it will be slower in most circumstances but I can try using a per connection mutex to stop a second write operation commencing prior to the previous one completing.

Cannot really say more than "wahooo..." Pretty good work. and indeed the 'boost' statement are somehow weird.
One comment from my end, is Domoticz Web Server using something else than the boost library ? How do they deal with that ? Are they 100% synchronous ?

Maybe it is stupid, but could we have a Connection option to ask for Synchronous instead of the current mechanism ? Will it impact the ALL plugin framework behaviour ?

[pipiche]

I did a further test on my end, in order to reduce the load. So I made the plugin responding without Keep-alive, so Connection is close after the response is sent.

Looks like the Connection is closed by the plugin, even if the all informations are not completely sent ! Probably link to the Boost library I guess.

[Dnpwwo]

Ideally there shouldn't be a compromise between synchronous and asynchronous.

I've added a thread pool to boost::asio to help it scale and now track completion of writes to stop flooding and control memory usage. Your website now serves in around 2 seconds but I'm seeing differences between chrome and firefox.

Firefox seems to terminate connections after a chunked response while chrome does not. Both seem to render the site correctly which is good.

I've changed your test plugn to be multi-threaded and it now trys to ram 'send' requests into the framework as fast a possible with mostly successful results.

Still a bit more debugging to do though.

[pipiche]

Ideally there shouldn't be a compromise between synchronous and asynchronous.

I've added a thread pool to boost::asio to help it scale and now track completion of writes to stop flooding and control memory usage. Your website now serves in around 2 seconds but I'm seeing differences between chrome and firefox.

Firefox seems to terminate connections after a chunked response while chrome does not. Both seem to render the site correctly which is good.

I've changed your test plugn to be multi-threaded and it now trys to ram 'send' requests into the framework as fast a possible with mostly successful results.

Still a bit more debugging to do though.

Sound good. and looking forward as I'll have to integrate it into the real plugin

On the other side, the real plugin is doing Compression and allowing caching, so for me getting served in 2 seconds the first time, is ok, as we do not expect big Chunk after start ...

Feel free if you want me to test the changes inside the Framework before pushing to Domoticz beta.

[pipiche]

@Dnpwwo anything I can do to help ?

[neha31]

Excellent thread..!!!

[pipiche]

And excellent implementation. However the HTTPS is still missing

[pipiche]

Any plan to support listening on HTTPS ?