A long time ago, The Daily WTF posted a brillant article called the complicator’s gloves. If you see a spelling mistake in that first sentence, it’s actually an age captcha and you are clearly too young to proceed. Go back to The Daily WTF, spend at least 3 days catching up and then come back to continue reading.
I think I would have fit right in that team. I would look at that poster and I’d be thinking “gloves are for… losers” as I proceeded to make my home lab even more complicated.
A couple of years ago I realised that making things complicated comes at a cost. Things break, they take too long to maintain, and they become generally too hard to work with.
The technical debt became so high that I would have a new cool idea but before I could start, I would remind myself of the two dozen things that I should really do before I put together another half cocked solution.
Three side projects later and I would be both out of will and energy to do anything else, and the cool idea would go on the ToDo pile, quite often never to be touched again.
A new approach
It was time for bold moves. I didn’t need a disk-less docker swarm that PXE-booted on iSCSI from a dedicated ZFS storage server. Neither did I need the ability to PXE-boot repurposed laptops for random experiments.
Now that ZoL was offering a much more civilised version of ZFS than the mid-0.6 version that I started with, the 16GB of RAM on that file server seemed like a waste for my needs. So I went out and got another 16GB. Anyone remember the good old days you could do that without having to sell a kidney?
Before I go on and to make sure I don’t sound ungrateful, a big shout out to all the ZoL people who have been saving my… sanity for over a decade now, since those 0.6x days. Snapshots are a godsend in themselves but with the exception of the odd lock-up back in the early days, I never had any issues or suffered any data loss.
So the plan was to turn that storage server into an all-in-one storage/virtualisation server. That’s where Proxmox comes in. Again. Now at v8, it had also come a long way from the v4 days, when I last run a HA cluster at home.
With ZFS support out of the box, the decision was relatively easy. Replace the Ubuntu installation with Proxmox, import the old pool, export the old ZVOLs over iSCSI for the swarm, the rest of the shares over NFS and nobody would ever notice we were here.
To the TrueNAS fans out there who I can hear screaming, yes, one day maybe, when I have a dedicated HBA to pass through and a more uniform disk collection. I wouldn’t mind the convenience of a full blown NAS host.
In the end there can be only one
After a short burn-in period, I created a VM with enough resources for all my running containers, fought the bloody raft one last time to get the swarm to accept the new VM as a new master and removed the bare metal hosts from the swarm.
I don’t remember the specifics but as it had often done in the past, the raft broke. More than once, and I had to roll back to the snapshot and start over.
Did I promote the new master first or did I demote one of the old ones? I still have the same question I had back then. Is 2 closer to even or 4? Is it just a hit and miss that you get a majority vote and things can actually happen?
It was finally all done! I now had a lone “legacy” docker swarm master running in a VM on Proxmox. It has been surprisingly stable ever since.
I suppose if you don’t have too many members pushing, pulling and generally rocking the raft, the raft has no reason to break. It also probably has little reason to break if the members are stable.
Who’s to say that PXE-booted, single core, Raspberry Pis with iSCSI storage served from a questionable file server were the reason for any instability. We’ll never know… Gloves, remember?
OK, maybe two, but only briefly
The next step? While the Jenkins based CI/CD pipeline I had in place did the job, it was a bit too heavy as well as overkill for my little setup.
With the fear of the raft also fresh in my mind, I had decided that I would replace that last swarm master with a plain old stand-alone docker instance. The existing stacks could then be moved gradually across.
The mail stack
You’re still reading and wondering when are we gonna get to the mail server bit? Well, at this stage, the mail server was a stack of Postfix, Dovecot, PostfixAdmin, RoundCube and of course, a MariaDB back end. It lived on the legacy docker server.
It was put together at the time that a correct SPF record was enough for messages to get through to the big boy mail servers. DMARC and DKIM were being introduced but were not strictly necessary yet.
Some time later, messages started landing in spam and eventually Yahoo stopped accepting the messages altogether. I didn’t really care. I was hardly ever sending any mails from that server.
Its main use was system messages for the lab itself and a catch-all mailbox for handing out one-off addresses to questionable registration forms. Sub-addressing (plus-addressing) is the modern way of doing it.
Now it was its turn to be moved to the new docker server and as usual I looked around to see if anyone more competent has put together a better stack than mine. Plenty of such people out there. There are also plenty of choices too.
The most popular names seemed to be docker-mailserver, Mailu and Mailcow. All of them are a much more polished variation of what I already had, with Dovecot and Postfix at the core. Nice, familiar, trusted, what’s not to like?
The new kids on the block
As I was trying to find out more about each of them, which one would have the smallest footprint and which one would be capable of handling my near-zero load, another name kept popping up.
Stalwart is a new, all-in-one mail server written in Rust. To paraphrase Sheldon, everything is better in Rust! Better yet, there is also Bulwark, a companion webmail client for Stalwart.
When I say new, I mean new new, as in v0.16. Many positive reviews out there, with the only red flag being about upgrade trouble. It’s in such an early development stage that the data structures are not guaranteed to remain the same in future versions. In fact the current recommendation is to stick with whatever version you have and avoid upgrades until v1.0 comes out.
I decided to take the giant leap of faith and potentially risk losing my precious cron and monit notifications. Just kidding. I back those up on a vanilla Dovecot instance using ImapSync. Did you think I’d be that reckless?
So far they both have been extremely stable and Bulwark is getting very frequent updates. You can tell that they are young projects with the odd basic feature missing here and there but If they continue like this, I’ll be very happy with the choice to switch.
In terms of features, go read the marketing brochure on one of their sites. For me, the most impressive part was the ease of setup. All web based and all fully integrated. I may regret that web based part when the time comes to migrate, but I guess I’ll burn that bridge when I get to it.
Set up a domain, give it a token to the DNS server and it goes off and configures everything. It fetches TLS certificates from Let’s Encrypt and it creates all the necessary SPF, DMARC and DKIM records. Even Yahoo was almost happy to accept my emails again. Almost.
Nearly there
The final problem with having my mail reliably delivered, was my IP address. My ISP, as any responsible ISP should do, registers residential IP addresses with Spamhaus, and Yahoo was not happy about that.
I didn’t have too many alternatives. Either fill in the form at Spamhaus and request an exemption for a year or use my ISP’s SMTP relay. I didn’t like the idea of forgetting to renew the exemption and have my mail quietly stop working, so relay it is.
That’s what I thought, but my ISP had stopped offering the SMTP relay service over a decade ago. What about AWS? Could SES be used with a self-hosted mail server? Well, it does have an SMTP interface so… maybe?
DNS verification that I own the domain, check! Email verification that am a valid sender, check! Budget check that I can afford 16 cents per 1000 messages, my accountant still has not get back to me on that.
It was all looking good but did not feel quite right. I was about to look through the SES terms of use when I did a search for SMTP relay services and to my surprise, there are plenty!
The discovery
As it turns out, the good vs evil spam battle has given rise to a whole industry of email marketing solutions. From embedded or even stand-alone subscription forms, to mailing list management, to SMTP relaying, these guys offer a full solution.
Like many other services these days, most of them use a pay as you go model, including a free tier. I thought this is too good to be true. I found a chatbot that turned out to be a human and explained my situation.
“I’ll be sending a dozen transactional emails a month, on a good month. I’m never really gonna exceed your free tier level. I don’t collect email subscriptions at all but if I did, I’d be using that fancy, double verified, triple confirmed with unsubscribe options and sugar on top, form you talk about.”
Yes, said Lucas, the human chatbot, just put all that in the “I won’t use your services to spam people, pinky promise” form. So I did… And it was all good… My account got approved and I was finally able to email myself again on my Yahoo account!
To be fair, Stalwart had done most of the heavy lifting with all the fancy DNS configuration. All I had to do, was add the relay’s domain to my SPF record, and their SMTP server as a relay route.
Preliminary Conclusions
This must be the best kept secret out there. A self-hosted mail server is turning into the holly grail of self-hosted because of all the hoops we have to jump through, to have our mail delivered.
Just like everything else self-hosted however, a thorough evaluation is needed before trusting it with anything important like life and limb, security, privacy, reputational risk, personal and professional, etc etc.
I can’t ignore the irony that the very services that were created to facilitate the sending of good spam, can be used to solve the problems that bad spam has created for everyone else. I suppose in a way the problem is exactly the same.
Evaluation
So why have I gotten this far without naming the chosen service? Because I’m not 100% happy with the one I chose, or should I say, signed up for. It was more of a rush to prove the concept rather than an in-depth analysis that triggered that choice. With the concept proven, now I can start looking closer.
Their primary focus is marketing emails so the most basic of all email features, delivery failure reports, is not supported. You can check all that online when “evaluating the results of your marketing campaign” but there is no angry sounding email from MAILER-DAEMON, telling you to check your spelling.
They add a “Sent with xyz.com” footer to the email like they are a giant blooming iPhone. They also add their special magic tracking sauce to see if and when the email is opened and of course all the EU-compliant unsubscribe headers that make Yahoo twitch and ready to pounce.
There are providers that focus on transactional emails who may be more suited to my needs, and now that the concept has been proven, it should just be a matter of just signing up for the right service.
What else?
A final word on SES. I have since found out that it’s a very basic version of all of these services. It does not offer any of the bells and whistles that seem to be ruining my current experience so I will definitely be having a second look.
Oh, and why do I keep going on about Yahoo? Because it’s my primary email account provider and the strictest of the two that I have an account with. If I can fix deliveries to Yahoo, I’d expect them to work with Gmail as well.
