Fail2Ban for Domoticz fails behind reverse proxy

[janpep]

I hope someone has a similar setup and got this to work.
I spent quite some time searching the internet. I see many have this problem. I tried several hints, but I did not find the solution.

Situation 1 is working fine.
Internetport forwarded to --> Virtual machine (Ubuntu 22.04) with Domoticz and Fail2Ban installed.
Wrong login is recognized. Ban of the remote IP address is added to iptables.
Iptables blocks the next connection.
Also after the time set, the unban takes place. This shows the fail2ban configuration is correct for this situation.

Situation 2 is not working. (same fail2ban configuration)
Internetport forwarded to --> ReverseProxy on Synology --> Virtual machine (Ubuntu 22.-04) with Domoticz and Fail2Ban installed.
Wrong login is recognized. Ban of the remote IP address is added to iptables.
That is, it is in the same place in iptables, but now it does NOT blocks the next connection.

In both situations the IP address to block is added in iptables to the same chain.

Code: Select all

Chain f2b-domoticz (1 references)
target     prot opt source               destination
REJECT     all  --  IP-address-situation1  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  IP-address-situation2  anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

This is the first called in the INPUT chain.
At the moment I am out of options and ideas and configured my forwarding to the working situation 1 without the proxy.

[Kedi]

I use (almost) the same setup as Situation 2, but have Auto block, with 1 failed attempt, on my Synology and for Domoticz I use 2FA and a totally different https port.
I have no security issues. (knock knock)

[janpep]

I use (almost) the same setup as Situation 2, but have Auto block, with 1 failed attempt, on my Synology and for Domoticz I use 2FA and a totally different https port.
I have no security issues. (knock knock)

Thanks for your response.
- Yes, for the Synology and some Syno packages itself, I also use auto blocking in combination with 2FA, forced https, changed portsnumbers etc. and get notifications on attempts. Advantage there is to have also some firewall rules in addition. Some ports (-personal use only-) for the country where I am. Countries with most noticed attacking (based on my logfiles) are blocked for all trafic. Some ports (for website) open, Finally all other ports blocked.
- Access to a number of higher risk items (like SSH), can only be achieved via VPN.
- For the websites on the Synology, I have a script that blocks IP adresses that are denied once by extended .htaccess rules and I also have created my own "visitors plugin", that forbids acces for who is known and not whitelisted in abuseipdb.com. (Currently offline for maintenance.) This has a lot of daily hits for all kind of curious or evil attempts to the websites.
- For Domoticz I also use another portnumber and 2FA. But the 2FA is only active on the login form and i belieave not on the json access, I use from the macrodroid on my mobile phone? Assumed, but now I cannot remember if I have really tried/tested it.
Next to that, I had the Fail2Ban already running in the Ubuntu VM, because access is not catched by the autoblock or firewall of the Synology itself (when forwarded directly from the router). So I included Domoticz and Zwave-js-ui in Fail2ban as an additional layer. Better save than sorry.

Then, I did not wanted to add overhead for extra firewall rules in iptables of the Ubuntu machine (with the country blocking etc.), when they are already set on the Synology itself, So I thought to move the access to Domoticz and zwave-js-ui from direct router forwarding to reverse proxy. Then it has to pass the Synology firewall first.
Everything is working just fine, including the Fail2ban banning, but not the blocking.

It is more that I can not stand not being able to find the cause than that I am worried about security. Both Domoticz and Fail2ban send a notification for failed login attempts and that has only happened a few times in years.

When I cannot solve it (and after some days of searching, try and error), this post here is my last attempt , then I rather keep using Fail2ban in direct access to Domoticz on the virtual machine, than to choose for the country blocking by Synology firewall rules that come with the reverse proxy. I think that the country blockade is easier to bypass than the Fail2ban.

[PierreT]

Obviously can't work because it is not the remote machine making the TCP connection to the Domoticz server but your HTTP proxy machine. There is nothing reverse that the proxy can do btw and should you have instructed it to do so anyway you're just wasting CPU time.

[janpep]

Obviously can't work because it is not the remote machine making the TCP connection to the Domoticz server but your HTTP proxy machine.

Yes, I can understand that. Banning and blocking takes place based on the domoticz log, which does see the remote address. Blocking does not really happen, because the firewall sees the connection coming from the reverse proxy.
It would be nice when iftables can be instructed to look at the remote address for the connections coming from the reverse proxy.

[Kedi]

If I want to do:

Code: Select all

https://<domoticz domain name>:<portnumber>/json.htm?type=devices&rid=1220

I get a 401 Unauthorized error.
I have a certificate so domain name should be correct, portnumber should be correct, 2FA should be correct.
I almost always use (Synology) VPN with Domoticz internal IP-address, with backup VPN from my router.
I scan logfile for 'unwanted' IP-address connection and get an PushOver Alert if so.
Until to day I never had an unwanted login.

[PierreT]

Obviously can't work because it is not the remote machine making the TCP connection to the Domoticz server but your HTTP proxy machine.

Yes, I can understand that. Banning and blocking takes place based on the domoticz log, which does see the remote address. Blocking does not really happen, because the firewall sees the connection coming from the reverse proxy.
It would be nice when iftables can be instructed to look at the remote address for the connections coming from the reverse proxy.

Exactly, so what you need to do is not change the firewall on the Domoticz machine but on the internet connected proxy machine. Plenty of ways to do that. Simplest thing to setup is to pipe the blocking command through ssh using key authentication. You could also add a vhost to that web server or a dedicated TCP listener instance to execute the blocking commands and then have fail2ban use curl on the Domoticz machine to report the offending addresses to the internet connected machine.

[janpep]

Exactly, so what you need to do is not change the firewall on the Domoticz machine but on the internet connected proxy machine. Plenty of ways to do that.

Thank you!
I will look into these options.
I still have some reservations about this, because I deliberately run Domoticz in a 'separated' (VM) environment. I do not want to mix that too much with the Synology itself. I do not want to have the risk that when something might go wrong, the firewall of the Synology will get into trouble, nor do I want to make the whole setup too complicated. I will see.

[PierreT]

I suggest you take a look at ipsets then. Rather than change the iptables rules themselves on the fly this filter allows you to only alter the IP lookup table that is referenced by a static rule in the chain. To be fair, I would never allow dynamic updates to be applied to any of the main chains in the first place.

[janpep]

I suggest you take a look at ipsets then. Rather than change the iptables rules themselves on the fly this filter allows you to only alter the IP lookup table that is referenced by a static rule in the chain. To be fair, I would never allow dynamic updates to be applied to any of the main chains in the first place.

Thanks again. I will look into that too.
The other option I thought of is to simply add geoIP also to the virtual machine. Then forwarding can continue to run directly to the VM. Only allowed country has access, fail2ban already works, my domain SSL certificate already works, 2FA already works and I do not have to mess around with the Synology.

[janpep]

I get a 401 Unauthorized error.

Just tested. I can confirm this. With wrong passwd, you get the 401.
But ... I did not find this attempt in the Domoticz log and no Domoticz notification!
Then this will also not be registered by Fail2ban and not blocked and no Fail2ban notification.
I think you can try this for as long as you want.

[PierreT]

The other option I thought of is to simply add geoIP also to the virtual machine.

My favourite filter. Stops 99% of all the non-intended traffic right at the doorstep, except of course if you happen to live in China. Do make sure to follow the monthly updates to the database and don't forget to temporarily add your destination country to the filter when going on holidays.

[janpep]

My favourite filter.

I have looked at it from all sides. In the end (in my situation) the only added value I saw in using the reverse proxy is that it also uses the Synology firewall. Furthermore, in combination with Fail2ban, it only leads to a more complex setup with (by me) unwanted actions between the virtual machine and the Synology.

Then I choose to keep it more separate and strengthen the own firewall for the virtual machine with geoip.
I have installed the geoip in my test environment and tested via VPN from abroad. Today I still have to figure out how I can best update it via crontab.

Then I have:
- Direct port forwarding from router to Domoticz and Zwave-js-ui on VM. (I still want to see how I can put Zwave-js-ui on a different port. Every little bit helps.)
- VM with its own firewall rules in iptables with only the relevant ports accessible from permitted countries.
- Fail2ban, which monitors incoming traffic on these ports and blocks them if necessary.
- Immediate notification when that happens.
- Also 2FA added for Domoticz.

When I go on vacation abroad, I already adjust the country blocking of the Synology, so it is a small effort to do that for the virtual machine as well.

One point of attention is perhaps that attempts can be made by accessing the domoticz API. If they were logged (which I have not been able to determine in quick test), then those attempts can also be blocked.

Thanks for the thoughts and tips.

[janpep]

Update on the result so far:
I have to say I am quite happy with it.
- Fail2ban works without any further hassle on the open ports (,where only https is allowed).
- By not using the reverse proxy and adding geoip to iptables, I have revised and improved its setup based on a template for both ipv4 and ipv6.
- With geoip the possibility to make a direct connection to the virtual machine is severely limited.
- Geoip is updated automatically by crontab.
- In addition to Domoticz I was also able to change the port for zwave-js-ui, so that it is also not set to the default anymore.

Adjusting the allowed countries in iptables is now done per script from Domoticz, which changes the rule for geoip and is shown in the name of my VacationMode device.

• A script retrieves the start and end times of vacation from my agenda (ical) and uses it to set the timer of my Domoticz VacationMode.

• When VacationMode switches ON it triggers my vacationmode script that changes a number of devices (boiler, EVObhome, changed schedules for lights etc.).

• The vacationmode script now also calls my bash script for iptables, which changes the ipv4 and ipv6 rule for the allowed countries.

• At the end this script renames my VacationMode device to show the allowed countries. (see image as an example).

• And of course all actions in the opposite direction when vacationMode is switched off by the timer and the vacationmode script is triggered again.

VacationMode-1-JanPep.png (13.3 KiB) Viewed 998 times

VacationMode-2-JanPep.png (12.74 KiB) Viewed 998 times

At the moment I still have to adjust the script in time to change the code(s) for holiday destination countries.
The last step, to fill in the desired countries somewhere, I still have to think about.

[Kedi]

I have installed the geoip in my test environment and tested via VPN from abroad

Nice solution, a little bit to complicated for me, I like KISS solutions.
I wonder if hackers know how to 'mask' there IP-address by VPN (or other methode) too.
I think that a hacker that uses his/her own 'home' IP-address is stupid an not capable of hacking an other system.
Are you using DMZ or portforwarding?

[janpep]

I have installed the geoip in my test environment and tested via VPN from abroad

I think that a hacker that uses his/her own 'home' IP-address is stupid an not capable of hacking an other system.
Are you using DMZ or portforwarding?

1. At my websites I saw frequently connections to e.g. https://domain.com/?author=1 from one IP address, directly followed with https://domain.com/?author=2 from another IP address and so on. Must be the same guy . So apparently it is possible to spoof IP addresses, or work from multiple (hacked?) locations.
So you may be able to work around blocked country or Fail2ban, but all together I think the measures reduce the chance of trouble.

2. To do a quick test from another country, I created a VPN connection to a friend abroad and accessed my system from that route. So it looked as if I connected from another country and I could see whether or not I was blocked when I replaced the iptables rule for the configured source-counties.

3. My setup is now as follows:
- Domoticz and Zwave-js-ui are running with my custom portnumbers under Ubuntu virtual machine on my Synology.
- Domain SSL certificate installed for both applications.
- Portforwarding for these ports from my router for ipv4 and ipv6.
- Strict firewall rules in iptables are set for ipv4 and ipv6.
- Now completed with the additional installation of geoip on the virtual machine. Not difficult to set up.
- Remaining ports allowed only from home country, which rule is automatically replaced when my 'Vakantiemode' goes ON or OFF.
- Fail2ban is running on the virtual machine to monitor the failed login attempt and sends me a notification when a ban is activated.
- When you pass this, you still have to try and guess the very strong passwords.
- When you pass this, you still need to have the 2FA.
- Additionally: When you are denied by Domoticz with false login attempts, my already existing script on the Synology sees the log and adds your IP address to the blockinglist (included in apache) for my websites, to prevent further visits from you. Same when you do an illegal action on my website. And in addition to extensive rules in .htaccess my home made wordpress plug-in blocks you when you are known in the abuseIPDB. Logging in to the websites (for maintanance) is only possible from the IP address that I transmit from my notebook or telephone and that is processed via a script.
- The Synology itself also has its own firewall, geoIP, auto blocking and 2FA.
I guess for a private system this seems like a pretty good start to security.

- When I am away from home, for other more risky activities like SSH or VNC, I first make a VPN connection to my home. These connections are also monitored and can be seen (from which device the connection is made) in Domoticz when they are online and give me a notification.

[Kedi]

Nice and robust. Now hope that you don't accidently lock your self out when you are on holiday.
I had it once, years ago. I was lucky to have a certificate controled SSH backdoor, and could fix my lockout.
That made me redesign my whole setup to be KISS compliant.

[janpep]

Nice and robust. Now hope that you don't accidently lock your self out when you are on holiday.

That will not happen so fast.
The VPN (server) is on my router and not on the Synology, so I will always have access to my home network and be able to fix what is needed.
The whole setup was not made in 1 day, but grew over the years.
For most devices (including the VPN connections) I use fixed IP addresses.
First firewall rule is always to have access from a few special IP addresses.
Also the VPN IP addresses (in a separate range) are trusted by NAS and virtual machines.
From Domoticz I can Start/Stop/Restart/Reboot/Wake on LAN the most important devices and programs.

Or course, in case of a powerfail everything stops for a while (no UPS yet ), but will then automatically start in last state.
When needed I can power on the NAS with DECT (AVM FRITZ!DECT 200) from the router (because then Domoticz/Zwave/MQTT are not running).
The more 'risky' changes, like this iptables renewal, I do first (from home) on virtual test NAS or on virtual test Ubuntu and never while I'm on vacation. Even then I first make a backup or snapshot of the starting situation. You must always make sure you can go back.

[janpep]

And the finishing touch is in a selector device with travel icon .

I have added some combinations of common destinations with transit countries in the selector.
Here I can make a selection for my next vacation.
I now can adjust the script to obtain the active selection when the VacationMode switches ON.
Then it can use this selection to set the iptables geoip rule.
And when the VacationMode switches OFF, the script can set both selector and iptables geoip rule back to the home country.

VacationMode-3-JanPep.png (14.73 KiB) Viewed 930 times

[Kedi]

Just talking like the devil's advocate: What happens when your DS718+ dies?