Cannot open ../dzVents/runtime/dzVents.lua: Too many open files

[Freemann]

Since a few weeks my domoticz "randomly" isn't reachable on its webserver and all LUA scripts aren't working.
Looking at the Domoticz Logfile i see this message constantly showing up:
2019-06-10 11:47:43.548 Error: EventSystem: in /home/XXXX/dev-domoticz/dzVents/runtime/dzVents.lua: cannot open /home/XXXX/dev-domoticz/dzVents/runtime/dzVents.lua: Too many open files

After a "domoticz restart" it works again, but I have no idea whats going on. Especcialy because I didn't change much.

Currently I'm on version 4.10732, the version before the upgrade to the new OpenZwave. Don't want to update because of the bug/faults I see on git relating to this new OpenZwave version.

Any Idea what can cause this error message?

[waaren]

Since a few weeks my domoticz "randomly" isn't reachable on its webserver and all LUA scripts aren't working.
Looking at the Domoticz Logfile i see this message constantly showing up:
2019-06-10 11:47:43.548 Error: EventSystem: in /home/XXXX/dev-domoticz/dzVents/runtime/dzVents.lua: cannot open /home/XXXX/dev-domoticz/dzVents/runtime/dzVents.lua: Too many open files
Any Idea what can cause this error message?

Can you check your limits and processes with open files when you see this error? You can use these commands.

Code: Select all

ulimit -Hn  # system hard limit max 4096 for standard user 1048576 for root on my RPI
ulimit -n # soft (user) limit for all users (1024 on my system)

sudo lsof > /tmp/lsof.log
sudo cat /tmp/lsof.log | awk '{ print $2 " " $1 ; }' | sort -rn | uniq -c | sort -rn  > /tmp/ordered_lsof.log
echo Top 20 processes with open files 
head -20 /tmp/ordered_lsof.log
echo 
echo Total open files $(awk '{n += $1}; END{print n}' /tmp/ordered_lsof.log)
echo 

[Freemann]

ok will do!
you having this prepare/ready, means it happens more often?

monitoring the top20 for about 10mins and don't see any changing figures.

What are the figures saying?

Here's a first report;

Code: Select all

  30666 2910 java
  17425 1844 python
   7067 3141 mongod
   3696 1150 dockerd
   3625 27237 domoticz
   2635 1738 mysqld
   1440 961 pcmanfm-q
   1295 964 lxqt-pane
   1030 965 lxqt-poli
    968 8141 update-ma
    925 1016 update-no
    910 807 container
    884 963 lxqt-noti
    808 962 lxqt-glob
    711 880 lxqt-sess
    640 854 Xorg
    621 969 lxqt-runn
    597 1140 qlipper
    597 1138 lxqt-powe
    592 993 pulseaudi

[edit]
O wait... stupid... was only running the last command, but this it then running on the previous not changed file.

Did run the command all again and here's a second set of figures;

Code: Select all

  31768 2910 java
  15416 1844 python
   7067 3141 mongod
   3696 1150 dockerd
   3683 27237 domoticz
   2635 1738 mysqld
   1440 961 pcmanfm-q
   1295 964 lxqt-pane
   1030 965 lxqt-poli
    968 8141 update-ma
    925 1016 update-no
    910 807 container
    884 963 lxqt-noti
    808 962 lxqt-glob
    711 880 lxqt-sess
    688 854 Xorg
    630 1519 docker-pr
    621 969 lxqt-runn
    597 1140 qlipper
    597 1138 lxqt-powe

So java is growing, python is getting smaller... is there a way to find out where this is coming from?

[waaren]

What are the figures saying?

The first column tells you ho many file(descriptor)s the process from the second column have opened at the moment the lsof command was given. So it would be interresting to see how many are open when you see the error. There is a (soft)limit per process and a hard limit system wide.

you having this prepare/ready, means it happens more often?
monitoring the top20 for about 10mins and don't see any changing figures.

I had it once and that turned out to be caused by a bash job that rescheduled itself and did not stop. So too many jobs on my system
Luckily Google has a far better long term memory than I do.
If I compare your top 20 with mine

Code: Select all

   4872 4030 mysqld
    361 22571 MainWorke
    360 1720 llvmpipe-
    280 1326 node
    260 502 node
    242 22571 WebServer
    241 22571 Webem_ssn
    241 22571 RFXCOM
    240 22571 RFLink
    240 22571 PluginMgr
    240 22571 OZW-drive
    240 22571 EventSyst
    240 22571 Domoticz_
    202 1251 smbd
    194 1313 lpqd
    174 1253 cleanupd
    171 1252 smbd-noti
    167 1239 winbindd
    166 1236 winbindd
    162 1275 winbindd

I see some high numbers in the Java and Python area in your top 20. Maybe that's an area to investigate first?

[Freemann]

OK, had Home-Assistant for testing running in a Docker container. Stopped HA and Python is of the list.
That keeps Java all alone on top, thinking that the Ubiquiti Unifi Controller container is the next potential point to look at.

Code: Select all

  31046 2910 java
   9546 3141 mongod
   4002 27237 domoticz
   3432 1150 dockerd
   2635 1738 mysqld
   1440 961 pcmanfm-q
   1295 964 lxqt-pane
   1030 965 lxqt-poli
    968 8141 update-ma
    925 1016 update-no
    884 963 lxqt-noti
    858 807 container
    808 962 lxqt-glob
    711 880 lxqt-sess
    664 854 Xorg
    621 969 lxqt-runn
    597 1140 qlipper
    597 1138 lxqt-powe
    592 993 pulseaudi
    579 1142 nm-tray

[Freemann]

Ok, this evening another "to many files..":

Code: Select all

2019-06-14 22:03:35.260  Error: EventSystem: in /home/marco/dev-domoticz/dzVents/runtime/dzVents.lua: cannot open /home/marco/dev-domoticz/dzVents/runtime/dzVents.lua: Too many open files

Here the output of the above commands;

Code: Select all

xx@dxx:~$ echo Top 20 processes with open files
Top 20 processes with open files
xx@xx:~$ head -20 /tmp/ordered_lsof.log
  31378 27237 domoticz
  30430 31924 java
  19728 1403 python
   6623 32007 mongod
   4818 1150 dockerd
   2666 910 mysqld
   1440 961 pcmanfm-q
   1295 964 lxqt-pane
   1290 807 container
   1168 5153 influxd
   1030 965 lxqt-poli
    988 8141 update-ma
    925 1016 update-no
    884 963 lxqt-noti
    808 962 lxqt-glob
    798 639 docker-pr
    756 5126 docker-pr
    711 880 lxqt-sess
    704 854 Xorg
    621 969 lxqt-runn
xx@xx:~$ echo

xx@xx:~$ echo Total open files $(awk '{n += $1}; END{print n}' /tmp/ordered_lsof.log)
Total open files 133180

[waaren]

Ok, this evening another "too many files..":

Code: Select all

2019-06-14 22:03:35.260  Error: EventSystem: in /home/marco/dev-domoticz/dzVents/runtime/dzVents.lua: cannot open /home/marco/dev-domoticz/dzVents/runtime/dzVents.lua: Too many open files

Here the output of the above commands;

Code: Select all

Total open files 133180

Are you sure you are running on a RPI ? These figures do look more as if you are trying to compete with the Google datacenters
On my production Raspberry with over 300 active devices, many scripts, some python plugins, mysql databases and MQTT my total number
is 17666 and even on my NAS (Synology 916+) with many active packages including a domoticz instance I only have 67000.

Having said that I find the number for domoticz on your system in itself already quite high.
what do you see with

Code: Select all

lsof -p $(pidof domoticz)

?

Maybe other forum members can share the figures they count on their systems so you have more numbers to compare yours with.

[Freemann]

I'm running LUbuntu 19.04 with 12 docker containers on a NUC with 256g ssd and 8g mem.
Performance is totally no issue.

The high Python number is caused by Home-Assistant.
The high Java number is caused by the Ubiquiti Unifi-Controller.
The high Domoticz number is caused by....

Code: Select all

xx@xx:~$ [b]sudo lsof -p $(pidof domoticz)[/b]
[sudo] password for xx:
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND    PID USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
domoticz 23468 root  cwd       DIR                8,3     4096     262145 /xxxx/xxxx/dev-domoticz
domoticz 23468 root  rtd       DIR                8,3     4096          2 /
domoticz 23468 root  txt       REG                8,3 13500496     267226 /xxxx/xxxx/dev-domoticz/domoticz
domoticz 23468 root  mem       REG                8,3    14256     798023 /usr/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
domoticz 23468 root  mem       REG                8,3    51672     798019 /usr/lib/x86_64-linux-gnu/libnss_files-2.29.so
domoticz 23468 root  mem       REG                8,3  5698832     795947 /usr/lib/locale/locale-archive
domoticz 23468 root  mem       REG                8,3    14688     798249 /usr/lib/x86_64-linux-gnu/libutil-2.29.so
domoticz 23468 root  mem       REG                8,3   243840     797708 /usr/lib/x86_64-linux-gnu/libexpat.so.1.6.8
domoticz 23468 root  mem       REG                8,3  5080176     798120 /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0
domoticz 23468 root  mem       REG                8,3    43384     797646 /usr/lib/x86_64-linux-gnu/libcrypt-2.29.so
domoticz 23468 root  mem       REG                8,3  1175000     798197 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
domoticz 23468 root  mem       REG                8,3   304984     797854 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
domoticz 23468 root  mem       REG                8,3    64496     797843 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
domoticz 23468 root  mem       REG                8,3   165880     798289 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
domoticz 23468 root  mem       REG                8,3    96872     798145 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
domoticz 23468 root  mem       REG                8,3   217560     797842 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
domoticz 23468 root  mem       REG                8,3   669888     797555 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
domoticz 23468 root  mem       REG                8,3   577560     797912 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
domoticz 23468 root  mem       REG                8,3    43552     797844 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
domoticz 23468 root  mem       REG                8,3    22448     797909 /usr/lib/x86_64-linux-gnu/libkeyutils.so.1.8
domoticz 23468 root  mem       REG                8,3    35128     797723 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
domoticz 23468 root  mem       REG                8,3   269808     797811 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
domoticz 23468 root  mem       REG                8,3   113392     798162 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
domoticz 23468 root  mem       REG                8,3    97072     798139 /usr/lib/x86_64-linux-gnu/libresolv-2.29.so
domoticz 23468 root  mem       REG                8,3    51840     797914 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
domoticz 23468 root  mem       REG                8,3    18344     797642 /usr/lib/x86_64-linux-gnu/libcom_err.so.2.1
domoticz 23468 root  mem       REG                8,3   203200     797901 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
domoticz 23468 root  mem       REG                8,3   893664     797913 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
domoticz 23468 root  mem       REG                8,3    75776     798217 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.5
domoticz 23468 root  mem       REG                8,3  1236720     798050 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
domoticz 23468 root  mem       REG                8,3   522592     797794 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
domoticz 23468 root  mem       REG                8,3   224072     797848 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.5
domoticz 23468 root  mem       REG                8,3  1566760     798239 /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
domoticz 23468 root  mem       REG                8,3    63736     797919 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.10
domoticz 23468 root  mem       REG                8,3   331120     797921 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.10
domoticz 23468 root  mem       REG                8,3   309552     797812 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
domoticz 23468 root  mem       REG                8,3  1707392     790062 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.23.0
domoticz 23468 root  mem       REG                8,3   223400     798002 /usr/lib/x86_64-linux-gnu/libnettle.so.6.5
domoticz 23468 root  mem       REG                8,3    67424     798111 /usr/lib/x86_64-linux-gnu/libpsl.so.5.3.1
domoticz 23468 root  mem       REG                8,3   522824     798201 /usr/lib/x86_64-linux-gnu/libssh.so.4.7.3
domoticz 23468 root  mem       REG                8,3   113512     798151 /usr/lib/x86_64-linux-gnu/librtmp.so.1
domoticz 23468 root  mem       REG                8,3   120752     797870 /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.4
domoticz 23468 root  mem       REG                8,3   157904     798006 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.17.1
domoticz 23468 root  mem       REG                8,3  2000480     797609 /usr/lib/x86_64-linux-gnu/libc-2.29.so
domoticz 23468 root  mem       REG                8,3  1365096     797943 /usr/lib/x86_64-linux-gnu/libm-2.29.so
domoticz 23468 root  mem       REG                8,3    18656     797677 /usr/lib/x86_64-linux-gnu/libdl-2.29.so
domoticz 23468 root  mem       REG                8,3   149840     798113 /usr/lib/x86_64-linux-gnu/libpthread-2.29.so
domoticz 23468 root  mem       REG                8,3   567736     790438 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.5.0
domoticz 23468 root  mem       REG                8,3   108768     798349 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
domoticz 23468 root  mem       REG                8,3    39880     798149 /usr/lib/x86_64-linux-gnu/librt-2.29.so
domoticz 23468 root  mem       REG                8,3   149704     798236 /usr/lib/x86_64-linux-gnu/libudev.so.1.6.12
domoticz 23468 root  mem       REG                8,3    30880     790636 /usr/lib/x86_64-linux-gnu/libusb-0.1.so.4.4.4
domoticz 23468 root  mem       REG                8,3  2925408     797648 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
domoticz 23468 root  mem       REG                8,3   585504     798203 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
domoticz 23468 root  mem       REG                8,3    31032     798018 /usr/lib/x86_64-linux-gnu/libnss_dns-2.29.so
domoticz 23468 root  mem       REG                8,3    26402    1314274 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
domoticz 23468 root  mem-r     REG                8,3    32768     267165 /xxxx/xxxx/dev-domoticz/domoticz.db-shm
domoticz 23468 root  mem       REG                8,3   179032     797344 /usr/lib/x86_64-linux-gnu/ld-2.29.so
domoticz 23468 root    0u      CHR                1,3      0t0          6 /dev/null
domoticz 23468 root    1u      CHR                1,3      0t0          6 /dev/null
domoticz 23468 root    2u      CHR                1,3      0t0          6 /dev/null
domoticz 23468 root    3u  a_inode               0,13        0      12924 [eventfd]
domoticz 23468 root    4u  a_inode               0,13        0      12924 [eventpoll]
domoticz 23468 root    5u  a_inode               0,13        0      12924 [timerfd]
domoticz 23468 root    6w      REG               0,48 62102490         24 /tmp/domoticz.txt
domoticz 23468 root    7r      REG                0,4        0      13008 /proc/sys/kernel/osrelease
domoticz 23468 root    8r      REG                0,4        0 4026532044 /proc/version
domoticz 23468 root    9u     unix 0xffff8db9a6327c00      0t0   12808280 type=DGRAM
domoticz 23468 root   10uW     REG               0,23        6       1012 /run/domoticz.pid
domoticz 23468 root   11ur     REG                8,3 30771200     266026 /xxxx/xxxx/dev-domoticz/domoticz.db
domoticz 23468 root   12u      REG                8,3  1050128     265881 /xxxx/xxxx/dev-domoticz/domoticz.db-wal
domoticz 23468 root   13ur     REG                8,3    32768     267165 /xxxx/xxxx/dev-domoticz/domoticz.db-shm
domoticz 23468 root   14u  a_inode               0,13        0      12924 [eventfd]
domoticz 23468 root   15u  a_inode               0,13        0      12924 [eventpoll]
domoticz 23468 root   16u  a_inode               0,13        0      12924 [timerfd]
domoticz 23468 root   17u     IPv6           12808291      0t0        TCP *:XXXX (LISTEN)
domoticz 23468 root   18u  a_inode               0,13        0      12924 [eventfd]
domoticz 23468 root   19u  a_inode               0,13        0      12924 [eventpoll]
domoticz 23468 root   20u  a_inode               0,13        0      12924 [timerfd]
domoticz 23468 root   21u  a_inode               0,13        0      12924 [eventfd]
domoticz 23468 root   22u  a_inode               0,13        0      12924 [eventpoll]
domoticz 23468 root   23u  a_inode               0,13        0      12924 [timerfd]
domoticz 23468 root   24u     IPv6           12808300      0t0        TCP *:6144 (LISTEN)
domoticz 23468 root   25u     sock                0,9      0t0   12808310 protocol: TCP
domoticz 23468 root   26u     IPv4           12807756      0t0        UDP *:52270
domoticz 23468 root   27w      REG                8,3    69292     267201 /xxxx/xxxx/dev-domoticz/Config/OZW_Log.txt
domoticz 23468 root   28uW     CHR              166,0      0t0        461 /dev/ttyXXXX
domoticz 23468 root   29u     unix 0xffff8db8e3ee9400      0t0   12843304 type=STREAM
domoticz 23468 root   30u     unix 0xffff8db8e3eea800      0t0   12843305 type=STREAM
domoticz 23468 root   31u     sock                0,9      0t0   12811519 protocol: TCP
domoticz 23468 root   32u     sock                0,9      0t0   12815626 protocol: TCP
domoticz 23468 root   33u     sock                0,9      0t0   12818987 protocol: TCP
domoticz 23468 root   34u     sock                0,9      0t0   12824354 protocol: TCP
domoticz 23468 root   35u     sock                0,9      0t0   12833227 protocol: TCP
domoticz 23468 root   36u     sock                0,9      0t0   12838816 protocol: TCP

htop shows:

Code: Select all

  1  [|||||||||||||||||                                                    22.5%]   Tasks: 157, 700 thr; 1 running
  2  [|||||||||||||||                                                      19.0%]   Load average: 0.30 0.56 0.52
  3  [|||||||||||||||||||||||                                              30.9%]   Uptime: 9 days, 00:33:45
  4  [||||||||||                                                           12.1%]
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||2.47G/7.66G]
  Swp[|||                                                             27.8M/977M]

[Snowtiger]

My system is very low on numbers.
Top 20 and total

Code: Select all

pi@raspberrypi:~ $ head -20 /tmp/ordered_lsof.log
    376 1086 llvmpipe-
    264 19849 MainWorke
    176 19849 EventSyst
    175 19849 PluginMgr
    175 19849 OZW-drive
    175 1355 menu-cach
    175 1355 lxpanel
    175 1355 gmain
    175 1355 gdbus
    175 1292 menu-cach
    175 1292 lxpanel
    175 1292 gmain
    175 1292 gdbus
    174 19849 WebServer
    174 19849 Webem_ssn
    132 1247 gvfsd-fus
    128 1141 master
    127 3323 gdbus
    127 3323 deja-dup
    127 3323 dconf

Total open files 12699

[waaren]

I'm running LUbuntu 19.04 with 12 docker containers on a NUC with 256g ssd and 8g mem.
Performance is totally no issue.

There is no direct relation to the numbers from lsof and CPU / memory usage, This is about open file (descriptor) s.
If possible you should raise the soft (per user) and / or hard (system wide) limits of max open files.
In this situation it will be very worthwhile to search for the many possibilities the lsof command has, to investigate what the cause of the many open files could be.
There are countless examples and very easy to find. The same is true for adjusting the hard and soft max open file limits.

[limar]

I'm having the same issue. It might be noteworthy to mention that I'm also running the unifi controller on the PI. Were you able to track down the issue?