This is the second part of this troubleshooting story and it’s probably missing quite a bit of context. Context can be found in the first part.
Admittedly, memories of my high-school biology classes are a bit fuzzy. The birds and the bees, the tortoise and the hare, the cats and the butterflies, all sorts of fascinating things happening on the open plains, on the other side of the great firewall.
When the butterfly caught a lion or a leopard, its egg would start growing in the apple, and the apple would start to rot from the inside. To be fair, that does also happen to all apples in due course, but the worm certainly does not help. Rotten apples and open windows. The perfect combination for the fast worms and the evil butterflies have a field day.
Firewalls in general do a reasonably good job keeping them out. Where there’s a firewall however, there is always a crack, and the older the firewall, the bigger the crack is likely to be. It becomes a numbers game and the more firewalls they can check, the more likely they are to find a crack.
One way to slow them down, is apparently tarpits. I don’t quite see how tarpits would slow down butterflies, but it does make sense for worms. I also don’t quite see how tarpits don’t outright kill the worms but it seems they have developed the ability to wiggle their way free.
Can I do enterprise tricks with entry level gear?
For the greater good, a few years back, I also set up a tarpit on my firewall. Every second those wiggly things wasted in my tarpit waiting for an answer, would be a second they didn’t spend searching for cracks elsewhere. I could spare a few CPU cycles.
Until one day a particularly smart, or particularly stupid, kind of worm happened upon my firewall. “I got a crack!” it called out to its mates, before it even realised it was in a tarpit.
That didn’t really matter. What did matter was that a whole blooming swarm answered the war cry and within minutes, I had thousands of connections open and tarpitted. My firewall’s poor little 600MHz MIPS processor was overwhelmed and unable to keep up.
The problem
I value the greater good and I like to think I do a lot of things towards it. There are limits however. When the greater good causes my better half’s internet connection to fail, that interferes with my “personal good”, and we can’t have that. As a compromise, I set up a blacklist on the back of the tarpit.
On the first attempt: “You would like to access a closed port? Let me make a note of your IP address and come right through to the pit here”. On subsequent attempts: “You’re on the list already. I’m not wasting any more CPU cycles on you. Talk to the hand.”
This worked great for years, or so I thought. In hindsight, it’s been giving me minor trouble for some time, as I was already suspecting. Did something change? Why was the problem so much more obvious all of a sudden?
I want to say QUIC, the UDP based protocol that is becoming more and more prevalent these days. That however would imply that QUIC is to blame, which is absolutely wrong.
The problem was my simplistic blacklist handling. A single packet, on a closed port, without an existing connection, would land an IP address on the blacklist for 15 minutes and all traffic from that IP would be dropped during that time.
The theory, maybe
Here are some assumptions on why this was happening. I have not done any extensive testing.
TCP is more resilient to rogue packets arriving late. The TCP connection is, after all, established and terminated using well defined handshakes that firewalls recognise and use to keep track of open connections.
UDP on the other hand is stateless and firewalls the world over, try to guess what constitutes an open connection. The vast majority of them go by time since last traffic. Mine considers the connection closed 10 seconds after the last received packet.
Ironically, cutting down on the time spend on handshakes is the biggest touted advantage of QUIC.
Because of this, I assume TCP connections, while certainly not immune to the problem, are far less likely to be affected than UDP ones. QUIC is also becoming more popular and that could explain the rise in occurrences.
I still have no explanation why the outages would come in batches but the fact that they were not constant, leads me to believe that and some kind of service rollout process is causing delayed packets.
Leave it to me to amplify a late packet to a full blown 15 minute service outage. How’s that for five nines?
OK, now what?
With the problem identified, the remaining question was what to do about it. There were three options that I could think of:
Disable the blacklist. I didn’t want to do that because of the warm and fuzzy feeling it provides, knowing that each IP address can make one effective attempt at my firewall every 15 minutes. It would also mean that I my tarpit would be susceptible to flooding again.
Implement an n-layer blacklist. Don’t blacklist IPs on the first packet, blacklist them on the nth. This would reduce the problem but it would not eliminate it. n+1 rogue packets would still land a perfectly good IP on the blacklist. Every larger n value would reduce the chance of that but would also diminish the purpose of the blacklist.
Implement a whitelist or to be more precise, a blacklist-bypass list. If a device on the LAN side tries to connect to an IP on the blacklist, the same IP is added to a whitelist. This does not mean that all firewall rules are bypassed, it only means that the blacklist entry for that IP address is ignored and all packets and have to go through the normal rules.
The verdict
The whitelist seems to work well so far. I got a scare a few days back when the firewall got overwhelmed by traffic, and should perhaps run a few tests to make sure I didn’t mess up the firewall’s hardware offloading. Those can wait until I get another scare sometime in the future, probably far enough, not to remember what I changed to break it.
All in all, an interesting puzzle to solve. Who would have thought that a blacklist implemented to slow down attackers would somehow catch websites whose packets were sent the long way around.