Domoticz need restart every 2 weeks - too many open files > Kodi UDP issue

[gbonny]

Version: 2024.7
Platform: Ubuntu 22.04 LTS VM on ESXi 7
Plugin/Hardware:
SolarEdge Web API, P1 meter (USB), RFXCOM433E (USB), OpenZWave via ZMEEUZB1 (disabled), MQTT Auto Discovery Client Gateway ZWave-JS-UI, Philips Hue bridge, Kodi Media server, Panasonic TV, OWM (disabled), BuienRadar, two Dahua camera's and HTTP/HTTPS poller Watermeter.

Description:
The problem is that I need to restart every 2 weeks. Symptoms are the errors in the log, see below and the GUI isn't accessible. I just found out that the following has probably a relation:

It starts here for example:

Code: Select all

2024-11-11 21:41:40.859  Error: EventSystem: in /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: cannot open /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: Too many open files
2024-11-11 21:46:42.786  Error: dzVents: /home/domoticz/domoticz/dzVents/runtime/EventHelpers.lua:370: attempt to index a nil value (local 'pfile')
2024-11-11 21:46:42.788  Error: EventSystem: in /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: /home/domoticz/domoticz/dzVents/runtime/EventHelpers.lua:580: attempt to index a nil value (local 'externalNames')
2024-11-11 21:51:48.933  Error: Watermeter_poller: Error getting data from url "http://192.168.150.33/api/v1/data"
2024-11-11 21:56:49.573  Error: EventSystem: in /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: cannot open /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: Too many open files
2024-11-11 21:56:49.577  Error: EventSystem: in /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: cannot open /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: Too many open files
2024-11-11 21:56:49.581  Error: EventSystem: in /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: cannot open /home/domoticz/domoticz/dzVents/runtime/dzVents.lua: Too many open files
2024-11-11 22:03:11.672  Error: Philips-Hue: Error getting Light States, (Check IPAddress/Username)

Some commands found here: viewtopic.php?t=28298

Code: Select all

sudo lsof > /tmp/lsof.log
sudo cat /tmp/lsof.log | awk '{ print $2 " " $1 ; }' | sort -rn | uniq -c | sort -rn  > /tmp/ordered_lsof.log
echo Top 20 processes with open files 
head -20 /tmp/ordered_lsof.log
echo
echo Total open files $(awk '{n += $1}; END{print n}' /tmp/ordered_lsof.log)
  34240 844 domoticz
    440 868 udisksd
    319 860 snapd
    273 907 ModemMana
    264 6020 packageki
    259 557 multipath
    228 656 vmtoolsd
    215 846 fail2ban-
    168 854 rsyslogd
    147 851 polkitd
    132 1196 master
    124 956 unattende
    114 1 systemd
    108 146860 upowerd
     84 645 systemd-t
     72 520 systemd-j
     59 866 systemd-l
     59 3463600 sshd
     57 3463475 sshd
     56 3463491 (sd-pam)

Total open files 38880

Code: Select all

lsof -p $(pid domoticz)
shows a long list of open UDP ports:
domoticz 844 domoticz   35u     IPv4            4285222       0t0    UDP *:54668
domoticz 844 domoticz   36u     IPv4              39784       0t0    UDP *:56033
domoticz 844 domoticz   37u     IPv4             193359       0t0    UDP *:39321
domoticz 844 domoticz   38u     IPv4              44199       0t0    UDP *:41881
domoticz 844 domoticz   39u     IPv4             118405       0t0    UDP *:32795
domoticz 844 domoticz   40u     IPv4             165369       0t0    UDP *:39694
domoticz 844 domoticz   41u     IPv4             119876       0t0    UDP *:35772
domoticz 844 domoticz   42u     IPv4             157346       0t0    UDP *:36578
.. <cut>

So the issue seems to be that Domoticz keeps a thousand UDP ports open for a long period of time so the process has too many 'files' open.

Why do these connections stay open and what is causing this?

When I restart "sudo service domoticz restart" the issue is solved. There are no UDP connections from Domticz at that moment.
I have this problem for a couple of years now.

[gbonny]

Ha! I think I've found the issue, every time a notification (or test message via settings) is sent to Kodi Media Player (whether online or offline) a new UDP entry appears:

Code: Select all

sudo lsof -p 3467223 | grep UDP
domoticz 3467223 domoticz   33u     IPv4           21305033      0t0      UDP all-systems.mcast.net:9777
domoticz 3467223 domoticz   34u     IPv4           21311120      0t0      UDP *:37499
domoticz 3467223 domoticz   38u     IPv4           21311745      0t0      UDP *:60016
domoticz 3467223 domoticz   39u     IPv4           21311237      0t0      UDP *:48635
domoticz 3467223 domoticz   41u     IPv4           21312086      0t0      UDP *:37318

Kodi Notification settings:
IP 224.0.0.1 or Kodi IP 192.168.150.123
Port 9777
Hops 5


@Gizmocuz can you look into this in the code?

[gizmocuz]

Sorry, can't help you with this as I'm not the author of this class, and don't use Kodi (and this hardware)

I had a small look at the code, and it seems there is only one place where a new socket is made: "CKodiNode::handleConnect()"

All looks ok... but I believe you found an issue!

Maybe that is why at the top of the class I wrote some long time ago:

//Giz: To Author, please try to rebuild this class using ASyncTCP
//Giz: Are those 'new'/thread lines not a memory leak ? Maybe use a shared_ptr for them ?

[gbonny]

Sorry, can't help you with this as I'm not the author of this class, and don't use Kodi (and this hardware)

I had a small look at the code, and it seems there is only one place where a new socket is made: "CKodiNode::handleConnect()"

All looks ok... but I believe you found an issue!

Maybe that is why at the top of the class I wrote some long time ago:

//Giz: To Author, please try to rebuild this class using ASyncTCP
//Giz: Are those 'new'/thread lines not a memory leak ? Maybe use a shared_ptr for them ?

Thanks for looking into this. However I think you might be looking in the wrong spot.
You're looking at /hardware/Kodi.cpp right? This file contains TCP connections and I was talking about UDP connections

The issue might be in this file: /notifications/NotificationKodi.cpp instead.
It seems on line 253 a socket is created when it's a destination is a multicast address.
On line 259 a socket is created when it's a regular IP address. No socket options are set, unlike for multicast addresses, not sure it's needed for regular IP address as well?!
On line 279 the message is send when the socket was created successfully.
When the message was successfully send, shouldn't we close the socket then? This might be the main issue of persistent UDP sessions?

I'm not an expert in C++ neither a socket expert but I tried to understand the code and did lookup "c++ sockets" on Google and got to this page:
https://www.geeksforgeeks.org/udp-serve ... ntation-c/ they're using sendto instead of Send. Not sure what's the difference, however they close(socketfd) at the end. Same socket like in NotifcationKodi.cpp

/hardware/KMTronicUDP.cpp and /hardware/Limitless.cpp also contain a closesocket().

What do you think? Thanks!

[gizmocuz]

I think you're absolutely right! I patched the code and a new beta 1(6330) is (shortly) available
Hope this solves your issue... but you are sending lots of notifications if this is causing issues...

[gizmocuz]

It should be fixed in beta #16333

[gbonny]

I think you're absolutely right! I patched the code and a new beta 1(6330) is (shortly) available
Hope this solves your issue... but you are sending lots of notifications if this is causing issues...

Thank you very much! It happens after a while so it's not that many notifications in 2 weeks total, I suppose?! idk =)

However I think more ppl could be having issues, because with every new notification you create: every/all notification systems is selected by default. I wonder who will deselect every notification system one isn't using

I'll test it asap!