Fail2Ban for Domoticz fails behind reverse proxy

[janpep]

Just talking like the devil's advocate: What happens when your DS718+ dies?

Of course that is always possible, but my thought is that I am protected and armed against the devil as much as possible.

• A power failure is usually short. Then websites and domoticz are down for a while. That is not of vital importance.

• In case of a problem with the disk, I am covered because it is mirrored.

• In case of other corruption (virus, ransomware?) I am covered, because a backup is made daily via a site2site VPN connection to my old NAS, which is located elsewhere. My websites are already synchronized with it, so it can be switched over fairly quickly if necessary.

• This permanent VPN connection is even firewalled to allow only backup and synchronisation of the websites, from a special backupuser and no other use! But even if this is also corrupt, then I still have the most recent backup of the whole thing on an external disk, which I will take with me on vacation. So I can always access all my important documents if necessary.

• Domoticz does not work for a while, but hey.. I am on vacation. I am going swimming! I will see about that when I get home. And under the parasol I will think of the solution!

• I can still log in with VPN to see via my camera whether lights are on or off and possibly ask one of my children to turn them off. (In principle, everything that is switched automatically can also be operated manually.)

• When I get home, I can temporarily put Domoticz back on my Raspberry, but I have stopped fiddling with SD cards, because I found it unreliable.

• And in the event of a total defect, you have one big advantage. Fail2ban (to stay on topic) is no longer necessary at that point.

So only thing that remains is when I die. Then my wife has no way to understand everything I have build, but the solution is simple.
I guess I write a script that can export all important data to be accessible on an external disk. Then reset the NAS to factory settings and from that on switch lights, airco, EVOhome et cetera with their own hardware switches, remote controls or apps.
Because one thing is for sure. Domoticz is very powerful in all its possibilities and I like it very much, but it is absolutely not user friendly in its setup and absolutely not suitable when you do not have some system administration skills.

[Kedi]

I am impressed.

[janpep]

I am impressed.

And when this devil is a burglar who comes for my PC, Notebook, NAS, old backup NAS or external backup disk.
- PC and notebooks are fully encrypted.
- Weekly full image backup of these devices on the NAS are encrypted.
- Backup to the old NAS (with rotation scheme) is encrypted.
- External backup disk is fully encrypted.
- And when he starts my NAS at another location and succeeds in getting access to is, then most important data is in an encrypted shared folder aswell.
- Keys and very strong passwords in an encrypted wallet on an encrypted usb stick.

I believe I am well protected from the devil both spiritually and materially. As far as it depends on me, I have done my best.

[PierreT]

1. At my websites I saw frequently connections to e.g. https://domain.com/?author=1 from one IP address, directly followed with https://domain.com/?author=2 from another IP address and so on. Must be the same guy . So apparently it is possible to spoof IP addresses, or work from multiple (hacked?) locations.

That is pretty much why I think Fail2ban is fairly useless today. Maybe 20-30 years ago you'd still see someone actually working a keyboard to try get into some internet exposed service, but today these are all worms working from a large pool of corrupted machines all of which will hit your machine just once and then move on to the next possible target to return after maybe an hour for the next single hit. There is no sane protocol to stop that and if you'd want to block all of them with individual rules the only likely thing to happen is your machine running out of memory. Should be needless to say that instantly blocking an IP on a failed login can lock you out as well after a simple mistype or accidental caps lock activation. I understand you have a backup entry through VPN but that is somewhat of a gamble too as some internet access points restrict outgoing ports and it can become very expensive very quickly if you need to use the phone network outside EU.

PS could you please stop writing reverse proxy? It is hurtful to my eyes and sanity.

[janpep]

That is pretty much why I think Fail2ban is fairly useless today. Maybe 20-30 years ago you'd still see someone actually working a keyboard to try get into some internet exposed service, but today these are all worms working from a large pool of corrupted machines all of which will hit your machine just once and then move on to the next possible target to return after maybe an hour for the next single hit.

Just a guess, but I thing you are lucky that you do not have to feed all persons that run scripts from one machine and not via VPN through other countries, but from their own country. At least they are blocked. Also then number of attempts blocked for my websites decreased very strong after blocking some countries. And on a daily bases I stop guys with (proven) bad intentions, using the same IP address that is already registered at abuseIPDB.
That doesn't prevent everything, but it does prevent a lot.

There is no sane protocol to stop that and if you'd want to block all of them with individual rules the only likely thing to happen is your machine running out of memory. Should be needless to say that instantly blocking an IP on a failed login can lock you out as well after a simple mistype or accidental caps lock activation. I understand you have a backup entry through VPN but that is somewhat of a gamble too as some internet access points restrict outgoing ports and it can become very expensive very quickly if you need to use the phone network outside EU.

Of course, the list is cleaned up from time to time and I took care to prevent to block my own IP adresses.
I hardly ever use public 'internet access points'. Then for security always with VPN on.
When it might give a problem I can use mobile internet. I can not remember to have had a problem to open the VPN (which is not on my NAS, but on the router), when it was not my own mistake.

PS could you please stop writing reverse proxy? It is hurtful to my eyes and sanity.

Synology calls it "Reverse Proxy" and that is what I had set up. I do not see the reason to call it anything else.
When you search for it, I think your sanity will be blown away by the number of results

[PierreT]

Just a guess, but I thing you are lucky that you do not have to feed all persons that run scripts from one machine and not via VPN through other countries, but from their own country.

Yeah well, you can't possibly tell which machines are corrupted and how they connect to the internet and therefore you. I've tracked multiple of these worms hitting on a honeypot I set up and although interesting to see what they do, there is no way to track back the origin as each is capable of passing commands onto other machines. In effect you are merely blocking people that were hacked who are unknowingly aiming to hack someone else to become your new attacker.

Of course, the list is cleaned up from time to time and I took care to prevent to block my own IP adresses.
I hardly ever use public 'internet access points'. Then for security always with VPN on.
When it might give a problem I can use mobile internet. I can not remember to have had a problem to open the VPN (which is not on my NAS, but on the router), when it was not my own mistake.

How do you know what your IP address is when you are in "Vacation Mode"? Why even grant access to the whole of NL when you are not on vacation? Just to be clear, your phone does not actually have a public IP. I found that out the hard way when road works cut my public line and I was forced to use a 4G connection for several months. Me and the rest of the world were completely shut off from my home services. In other words, if you get kicked off for entering a wrong password there is no point in trying your wife's phone unless she uses a different provider.

Synology calls it "Reverse Proxy" and that is what I had set up. I do not see the reason to call it anything else.
When you search for it, I think your sanity will be blown away by the number of results

Yes most people are stupid and blindly copy what other stupid people before them wrote. This is usually referred to as "high education". A proxy is something (or someone) that will forward a request on behalf of you. Generally speaking this will not cause the reply to be directed directly at you and in computing this is in fact prohibited/not possible at all. The proxy will receive the answer and direct this back to you, the original requester. This does not make the proxy a reverse proxy, it is just what the proxy is meant to do.

So you may ask, what is a reverse proxy? A reverse proxy is something that aims to manipulate the content of the original response such that it corresponds to what the web browser appears to look at. This may sound complicated, but consider your browser to be pointing at myproxy.com/somedir which is a reference to originalhost.local/someotherdir. Just give that a thought, what is supposed to happen when originalhost returns a page containing a link to myproxy.com/someotherdir/someotherpage.html? That's actually the easy comprehensible thing that a reverse proxy is really able to do, it will horribly fail however on anything java and of course everything taken from outside the proxied path. Long story short, reverse proxy doesn't work, except maybe with plain HTML web pages that were created prior to 1994.

[janpep]

Yes most people are stupid and blindly copy what other stupid people before them wrote.

You can have a strong opinion, but I think this goes off topic and with a tone I do not appreciate.

[PierreT]

Yes most people are stupid and blindly copy what other stupid people before them wrote.

You can have a strong opinion, but I think this goes off topic and with a tone I do not appreciate.

The tone, the tone... Does it probit you to learn or does it make you believe stronger in your version of the absolute truth?

[gizmocuz]

My two cents (And I do travel a lot!)

First, do not expose Domoticz (or any service) to the outside world.
(So no port forwarding!)

Next, install wireguard (if you happen to use a Fritzbox, this can be done natively in the Fritzbox, else install it as a service somewhere on another device, like the machine you run Domoticz on)
Forward the port you use on your router (51820 for example) to this machine (not needed if you use a Fritzbox)

Add a few wireguard clients (like for your mobiles, your laptop, tablet, office etc)

Install the native wireguard application on your tables/devices and connect to your house with this

Now you can access your services as you are using locally (http://192.168.0.20:8080 for example)

This is very safe too

You can also use OpenVPN, but wireguard is much faster and natively supported by most hardware devices now (kernel mode)

If you have a dynamic IP address at home, you can use services like dyndns to get a fixed hostname (not needed when using a Fritzbox, it already has a fixed hostname)

[janpep]

First, do not expose Domoticz (or any service) to the outside world.
(So no port forwarding!)

Thank you for thinking with me.
Of course in general you can say: "No door is saver than a closed door." And that is always true.

I am curious to hear what risc you still see, when Domoticz is on different port, access is limited to few countries, authentication is on, password is very strong, wrong authentication is guarded by Fail2Ban, the 2FA is used and wrong authentication on api gives an unauthorized error 401.
I understand very well that the other port, the fail2ban or the geoip on itself are not 100%. But all together I think that all these measures lowers the risk to a minimum. I have seen only a few attempts over the course of a number of years.

I use Wireguard (indeed one entry for every device as you suggest, and monitor their connections), for more risky access such as ssh, vnc or other targets.That is short term use. For continuous use (I tried for a while) sometimes this is not always stable when changing providers, have to turn it off and on after network change, or can be blocked as PierreT mentioned.
That is how I came up with the current method.

[gizmocuz]

With one wireguard connection to your house you can reach your entire house.
There is no need to have a separate wireguard connection for each device you want to reach.

The wireguard client on your (mobile/ipad) device can stay enabled. I have not met many conditions where I needed to turn it off, unless the hotel network was using the same subnet, or a public internet hotspot (like a plane) only supports port 80/443
(Don't use 192.168.0.xx or 192.168.1.x or 172.16.0.x, there are common subnets you find on default installed routers, use something like 192.168.48.x)

With one click you have your connection made to your house and you can connect/work with all your local devices.

There is also no need to install/use fail2ban anymore, as the connection can be trusted. (It is setup via wireguard)

It is also not (very!) easy to copy/paste the configuration settings of the client, and most clients use a camera to get all parameters, I consider it very safe.
If for whatever reason you think someone got your wireguard details, just delete that user in your local wireguard server, but I suspect this won't ever happen.

Also, we call this 'a single point of failure', as it is just:

[your house] <----> [wireguard] <----> [your remote device]

I have read this thread and noticed you use different rules for EU/Outside EU and more very complicated stuff... this all is not necessary

Keep it simple.... keep it safe!

[janpep]

Keep it simple.... keep it safe!

I do not consider the current setup as very complicated. Fail2ban and 2FA were already installed.
The only thing added is the geoip which limits greatest part of the world for direct access.
This because using the Synology 'reverse proxy' for that (my initial idea), turned out to become quite complicated (in my opinion).
At the moment a script does a simple replace for other countries in the geoip rule. Only when I go on vacation. (BTW. No difference betweeen EU or Outside EU. I have never mentioned that).

I explained that I was experiencing instability with continuous VPN connection in some situations.

I am very familiar with Wireguard. Also used for a permament site2site VPN connection, firewalled with host and application restrictons.
And indeed it makes no sense to have a connection for every device in house. What I ment and do have is a different client setup for every (outside used) mobile device/notebook. Each gets its own IP address. So based on the IP address it (internally) receives, I see what device is connecting.

For the internal network I have indeed changed the subnet to not commonly used as default by known routers.
Same for changing common ports etc. But that is in itself also not a watertight measure. However, all these things together ensure that additional thresholds are raised, which arm against a "standard approach".

As far as I am concerned, I think we can close this topic. It became clear why Fail2Ban failed behind the Synology 'reverse proxy'. Also some alternative approaches with their pros and cons have been discussed.

[gizmocuz]

Well, I probably missed the part where you have instability issues when using VPN...
As mentioned, I travel a lot, and with some very bad internet connections... Sometimes I have to press the refresh button, but 95% of the time while abroad, WireGuard connections are extreme stable

Good that you know how to use WireGuard, this eliminates the installation and usage of fail2ban/geo-blocks etc.

And as most of the time you will be in Nederland, your wireguard connection will be very stable

Keep it simple.... and secure

[lost]

With one wireguard connection to your house you can reach your entire house.

This depends on the use-case, but I'm not so fan of the idea to trade exposing a single service (could escalate on host to the whole LAN, but that's indirect and even basic user rights management will already make this task not so straightforward) vs directly the whole LAN if there is a flaw. Must also be very confident with all the devices that'll need to get the (full) access setup!

Of course, if many devices/services on your LAN may be accessed from the outside that's a better bargain to only have one "door" to monitor...

As said by others, https is not so targeted: Most unintended connection attempts looks to result from web indexing robots that immediately give up. Another story for ssh: I do not expose it anymore full time to the outside. A virtual push-button switch in Domoticz opens the port for 1mn if needed, so that's https needed to open ssh just the time to make a connection. Used port-knock daemon in the past but looks to be no more updated & now lacks IPv6 support.

I really try to keep accessibility possible with, would say, empty pockets & only my head contents needed! So not even 2FA for now, even if this may change in the future.

Fail2ban limit attempts even if some built-in https server side configuration (as sshd_config MaxStartups) would be more reactive with heavy burst attempts from many IPs than can now reach your home thanks to fiber capacity & make a log based monitoring solution alone much less effective than in the ADSL era.

[gizmocuz]

This is the exact reason for the existence of VPN's.... to have 1 backdoor
Crazy to have multiple VPN's in your system for different devices. Sure, this is all possible, not necessary.
Wireguard is damn secure, well encrypted.
Absolutely no need for multiple instances (for the same user), sure you could use another instance with limited rights, but even this can be configured

So, to recap, exposing ports to the outside world (http/s,ssh, whatever) is in general not a good idea.
Having a backdoor via VPN (Wireguard) is secure (you can run this on any port you like that 99% of the hackers won't find as nobody is scanning 65K ports), is a safe way to connect to your internal services.

As we can now assume it is secure, there is also no need to run fail2ban for your Domoticz instance, 'If' you want to use fail2ban, you could listen to authentication logs of Wireguard.

[PierreT]

Fail2ban limit attempts even if some built-in https server side configuration (as sshd_config MaxStartups) would be more reactive with heavy burst attempts from many IPs than can now reach your home thanks to fiber capacity & make a log based monitoring solution alone much less effective than in the ADSL era.

Fail2ban is dinosaur technology. Having it trigger on a single wrong password entry is plain silly. So the next "sane" thing to do is to accept possible mistypes twice, and then block. Except this doesn't work with modern day hacking methods because each instance of the hacking network will hit your machine only once every several hours. You can easily have a million attempts before seeing the same IP return for its next attempt and fail2ban will have lost track of that IP long before that.

[lost]

You can easily have a million attempts before seeing the same IP return for its next attempt and fail2ban will have lost track of that IP long before that.

Yes, I know that and that's the reason why I now don't expose a ssh server directly. But as I said on my own system https is not much a target, so I keep fail2ban for now. But if, in the future, situation worsen up to observed ssh bruteforce flooding from the whole world I would quickly build another dike before someone could guess a user/password combination (as I also never use default usernames for anything remote accessible).

Anyway, fail2ban is configured to block early & not keep bans for long so at least should not become a DoS target by accumulating too many FW rules...

Just fear I may not be able to keep my current goal to only need my "head contents with empty pockets" to access my system remotely if I must raise the dike someday: A 2FA looks no more a problem as now everyone have a smartphone in the pocket (and an authenticator already mandatory for work/banks...), but when I master both ends I really try hard to avoid this.

Would really prefer to have login that would mix fixed and one-time password with, for instance, some fixed prefix and an added suffix that may be head computed from current date/time and/or, for instance, some innocent looking info from the login page for a web site using a user chosen formula!