I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver.
But ...
Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind.
I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile.
In no particular order, some of the things that I find overly complex and disturbing are:
- The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway.
- The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3].
- Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
So much complexity and fragility ...
[1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved.
[2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt.
[3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
I think of it as a process of punctuated equilibrium.
Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware.
Between these brief periods, everything is stable and solid. I don't have to think about it at all. In 25 years, I've rebuilt things 5 or 6 times. After the first few times, it has been more of a chore than an adventure, but it's still worth it to me, for now.
One development that might change my mind about that: Ubiquitous and non-vendor-dependent MFA. If email was not the security mechanism of last resort (or the skeleton key), then I might consider outsourcing it.
...
Re: Rspamd: It works well, but I did not enjoy the configuration effort. And yes, I never wanted Redis on my mailserver, but there it is -- along with an astonishing amount of RAM usage, which offends me in theory even if it has no practical impact. On the positive side, after the initial few hours of effort to get things set up, I have spent zero hours maintaining any of it, so I can't reasonably complain.
Re: letsencrypt: I use dehydrated for certificate renewal, and I run a temporary webserver as you describe. Not a full dedicated webserver package, just python (or ruby) which is already installed. It's launched from the same script that automates the license renewal. Not elegant, but adequate. You could probably get away with netcat.
Of course you can validate by DNS instead of HTTP. But that's not always convenient either.
"Every several years (or ten), it requires a flurry of activity to update to new preferred daemons, or new hardware."
I think you are correct and this is a decent description of how this has worked.
I don't begrudge the every 5-10 year burst of activity/implementation. What I begrudge is the increased complexity. We're both talking about Redis as a hard requirement for rspamd which is the typical recipe for implementing DKIM which is required for email. That's a lot of moving parts.
I've been running amavisd-new for ~10 years (it runs spamassassing and clamav) and have been DKIM signing with it the whole time. Never had the need to switch to rspamd. You can also sign emails with OpenDKIM.
I've also been self-hosting email for 25 years. I see these exact patterns you describe.
Currently I'm using postfix, dovecot, and rspamd, with several users using pine and mutt locally. Every time I do a major server upgrade, I re-evaluate the configuration.
My mail server also runs a web server for some static sites, so the letsencrypt part was simple.
..and even if you follow all the best practices you'll just randomly end up blacklisted for what I can only imagine is misguided users clicking on spam.
If that happens on the wrong email provider you're out of luck. One of the mail servers I have a hand in has been blocked by one of those difficult providers for the past 2 months and we sent them 4 mails/week before the block on average :)
> Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
I have DKIM set up just fine without running Redis or rspamd so this part is purely down to your choices.
> Speaking of letsencrypt
Let's Encrypt is a website/service. There are many clients and you can write your own. At some point Let's encrypt will need to verify that you control the domain so you will need to be able to either host something via HTTP or have sufficient dynamic control over the DNS server. Neither DNS nor HTTP needs to be hosted on the same host as your mail server though.
AFAIK dkimproxy is no longer maintained. That is the main reason I dropped my plan using dkimproxy with OpenSMTPD for a simple send-only mailsetup. The rspamd+redis setup felt way to overkill for a tiny vps.
Postfix+opendkim+opendmarc works fine and it is very simple to setup as well.
Can you elaborate on your problem with Yahoo? I ran into that a while back, where a yahoo email address couldn't send to me... strange since DKIM validates for the domain.
There are all valid points and I question my decision to run my mailserver, but in the spirit of the Internet I continue to do so.
> [3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
Let's Encrypt provides an ACME service. So, you will want to (run something that can) speak ACME, a somewhat RESTful HTTP API documented in RFC 8555 - https://tools.ietf.org/html/rfc8555. You can use a web service to do this part, but then you're relying on some random web service and you have to manually do a bunch of steps to renew at least every 90 days which I'm going to assume you agree sucks.
There are lots of people who have built such tools that can look after this for you automatically, perhaps you would like https://acme.sh/ which as its name might suggest is a shell script.
Let's Encrypt offers three of the (increasingly misnamed) Ten Blessed Methods, you can prove control over a DNS name by doing any of these three things:
1. Running an HTTP server on port 80 which can answer certain well-known requests in a specific way acknowledging your ACME keys. This server needn't have any privileges, acme.sh can help you spin one up that will basically say "Yes, rsync is allowed this certificate" for any request. Except instead of "rsync" it's a key only you have, so your server answering this way never helps any bad guys so long as you keep your ACME keys secure. [You may not be aware you have "ACME keys" the software you've used probably just minted some and squirreled them away in a dot directory]. With that server in place, your requests for a certificate for that name will always work, because the key matches, Let's Encrypt will call your server, make up a random nonce and ask if that's authorised and for who, your HTTP server will always answer "rsync is authorised" so it works, tidy.
2. ALPN on port 443. A TLS server answering port 443 can accept a special ACME-only ALPN protocol choice from the Let's Encrypt servers and use that to prove control. You almost certainly don't want this unless you implement web servers (which run on port 443 with TLS already)
3. DNS queries for a magic name that can't be a hostname. The queries ask for _acme-challenge which has an underscore in it. Underscores are allowed in DNS but can't be hostnames, which is fine because this isn't a hostname. Your ACME client will need to write to DNS, it can either have a way to write to your actual DNS, or you can use a CNAME to redirect the names you need to another DNS server and give the ACME client control over that. [Edited to fix DNS name mistake]
You can insulate the ACME client from the server completely with either the trick I described in (1) or using DNS like (3) and using a Certificate Signing Request, CSR, which is one of the many ASCII text blobs you can get from OpenSSL. If you're comfortable re-using a private key for longer periods than the 90-days assumed in Let's Encrypt (a 2048-bit RSA key seems safe to me for the usual lifetime of a mailserver) you can even re-use the same CSR again and again to request each new certificate without sending any data from the mail server to the system calling Let's Encrypt.
You will need to arrange to get each new certificate installed (the private keys don't change in this scenario). But certificates are public documents, if it came to it the mail server can (this is very silly but it would actually work) just wait until 24h after the certificate should have been issued and then go get it from https://crt.sh/ -- In reality you can probably find some sensible way to get this public information back to the mail server each time the certificate is renewed.
I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver.
But ...
Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind.
I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile.
In no particular order, some of the things that I find overly complex and disturbing are:
- The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway.
- The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3].
- Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story.
So much complexity and fragility ...
[1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved.
[2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt.
[3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?