How to setup your own mail server

July 27, 2021

Hi my lonely visitor. In this blog post I'll tell you about my experience with setting up my own mail server. You might ask why? How did you come to such an idea when there's a lot of all those free gmails around? Well, I'm not happy when these gmails lose my mail, I dislike the fact they sniff all my email, I hate when they start putting legitimate and important messages to the spam folder and sending me spam as legitimate emails. Their robots can easily ban you. And finally, I did this simply because I can.

Looking back at early 2000s, every system administrator was able to setup mail server. But all they get depraved soon after free email services for custom domains have emerged. Owning a mail server lost any sense for the most of those who was not concerned about their privacy. I was depraved for a long time too but recently I decided to fix this.

It was very interesting for me how things changed for all those years? Does it become harder to set that up or easier? After trying that out I can say things haven't changed. Setting up mail server is still quite difficult task (unless you do this every day). One things became easier, another became harder, but the overall complexity remains the same.

Here's a few useful links as of time of writing this:

Anyway,you should not follow any recipe blindly. Things may change over time as it happened with OpenSMTPD configuration and now many articles on the internet are outdated. And, even if it's sometimes too difficult for understanding (many projects still need a good technical writer), the official documentation for packages is always your best friend. For example,

Very useful info how to debug authentication to Dovecot: https://doc.dovecot.org/admin_manual/debugging/debugging_authentication/

This blog post is based on my notes I made during the installation process which was a series of trials and errors. I re-arranged those notes as if I had to repeat this process quickly.

Packages

Before you begin you need to install necessary packages. I'm using FreeBSD and the first thing was to turn ancient sendmail off which is still shipped with the system. I added the following lines to /etc/rc.conf:

sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

I don't remember now whether I rebooted or simply stopped sendmail service. Just do whatever you want and check sendmail is not running.

Then, install necessary packages:

pkg install opensmtpd rspamd opensmtpd-filter-rspamd dovecot

Rspamd

SMTP server depends on Rspamd so it's good to start from configuring Rspamd. Create configuration directory:

mkdir /usr/local/etc/rspamd/local.d

Rspamd provides DKIM signing, which is the necessity nowadays, so you don't need anything else, such as DKIM proxy. You need to generate keys for DKIM. First, create directory for them:

mkdir /usr/local/etc/mail/dkim

(note that /usr/local/etc is specific to FreeBSD, on Linux the configuration is not separated, everything is in /etc)

Generate key pairs for DKIM:

openssl genrsa -out /usr/local/etc/mail/dkim/declassed.art.key 2048
openssl rsa -in /usr/local/etc/mail/dkim/declassed.art.key -pubout -out /usr/local/etc/mail/dkim/declassed.art.key.pub

Create configuration file /usr/local/etc/rspamd/local.d/dkim_signing.conf:

allow_username_mismatch = true;

domain {
    declassed.art {
        path = "/usr/local/etc/mail/dkim/declassed.art.key";
        selector = "20210512";
    }
    # add more domains to this section if necessary
}

Set correct permissions:

chgrp -R rspamd /usr/local/etc/mail/dkim
chmod -R g+r /usr/local/etc/mail/dkim

SMTPD

So, after looking around I decided to use OpenSMTPD. Those guys made things really simple.

Here is the configuration file for OpenSMTPD, /usr/local/etc/mail/smtpd.conf:

local_if=lo0
ext_if=vtnet0

pki mail.declassed.art cert "/etc/mail/certificates/mail.declassed.art.crt"
pki mail.declassed.art key "/etc/mail/certificates/mail.declassed.art.key"

table aliases file:/etc/mail/aliases
table domains file:/etc/mail/domains
table passwd file:/etc/mail/passwd
table virtuals file:/etc/mail/virtuals

filter   "rdns" phase connect match   !rdns disconnect "550 DNS error"
filter "fcrdns" phase connect match !fcrdns disconnect "550 DNS error"
filter "rspamd" proc-exec "opensmtpd-filter-rspamd"

listen on $local_if
listen on $ext_if port 25 tls pki mail.declassed.art filter "rspamd"
listen on $ext_if mask-src port 587 tls-require pki mail.declassed.art auth &lt;passwd&gt; filter "rspamd"

srs key "hDaewwP4CZCQai7N5FLwzmUty+HYJEkgRxTVb7m4"
srs key backup "Q4U/gzSmRfWQPRdyO09JG0IE5RYNokWdtlxsZZIv"

action "RECV_LOCAL" mbox alias &lt;aliases&gt;
action "RECV" lmtp "/var/run/dovecot/lmtp" rcpt-to virtual &lt;virtuals&gt;
action "SEND" relay srs

match from any for domain &lt;domains&gt; action "RECV"
match from local for local action "RECV_LOCAL"
match from any auth for local action "RECV"
match from any auth for any action "SEND"

The first step is obtaining a certificate from Let's Encrypt for SMTP TLS. Of course you can order a certificate from any other certification authority, Let's Encrypt is simply free and using it is better than using self-signed certificates.

Although Let's Encrypt exists for years already, I never used it. Well, I used certbot once, a couple of years ago, but I have completely forgotten all the details since then. At work we used paid certificates and renewed them once a year manually.

The issuance of Let's Encrypt certificate worths a dedicated blog post. For now I'll only tell you that I stumbled on a simple question: do I need an email address or not to make Let's Encrypt account. For mail server it's a chicken and egg problem: if you're installing an email server for yourself, you have no email address yet.

All authors, from Certbot to acme.sh tell us "Hey, look how simple to use our thing! It's 3 minutes only!"

Well, in 3 minutes I was not able to find the definite answer. I dug into RFC 8555 and finally implemented my own ACME client Much later I have found an option in Certbot to omit email address.

Anyway, you should obtain a certificate with your favorite tool. These lines in /usr/local/etc/mail/smtpd.conf are responsible for TLS:

pki mail.declassed.art cert "/etc/mail/certificates/mail.declassed.art.crt"
pki mail.declassed.art key "/etc/mail/certificates/mail.declassed.art.key"

Don't forget about file permissions. Secret key should not be readable by everyone.

The next step is to create vmail account:

pw user add vmail -m -c "Virtual Mail" -d /var/vmail -s /sbin/nologin

For Linux it could be:

useradd -m -c "Virtual Mail" -d /var/vmail -s /sbin/nologin vmail

Create aliases, domains, virtuals, passwd, mentioned in the configuration:

table aliases file:/etc/mail/aliases table domains file:/etc/mail/domains table passwd file:/etc/mail/passwd table virtuals file:/etc/mail/virtuals

/etc/mail/aliases was already in the system, I just added to the beginning:

root:   axy@declassed.art

/etc/mail/domains contains a list of domains served by SMTP server. In its simple form may contain a single line:

declassed.art

/etc/mail/passwd contains password hashes for SMTP server. Here is a sample line:

axy@declassed.art $6$Bix2vmEWTs7OXCEQ$MoIypMnb5dVGNlD9Bogqf9hj1CGKf2x9deIKv3q.In7mI7sErPn/fZQU6n6/pImR9JXpQXDqP05S0m5eo11Ap/

Password hash can be generated with

smtpctl encrypt

and many passwords can be generated at once with such a command:

cat file-with-passwords-only | smtpctl encrypt

Although it's not convenient to manage many users this way. I don't know a tool which would make life easier, for me it's okay as is.

Another issue with passwords is that Dovecot can't use same passwords file. Previously it was possible, but now it wants such a format:

axy@declassed.art:$6$Bix2vmEWTs7OXCEQ$MoIypMnb5dVGNlD9Bogqf9hj1CGKf2x9deIKv3q.In7mI7sErPn/fZQU6n6/pImR9JXpQXDqP05S0m5eo11Ap/::::::

So you need some script to generate passwords for Dovecot if you don't want to manage it manually. Again, for me it's okay as is. I don't provide email for masses so far and I don't bother.

SRS keys are just random bytes in base64 encoding. They can be generated with the following command:

head -c 30 /dev/urandom | base64

It's recommended to re-generate SRS key once a year.

DNS records

To be able to receive mail, you need MX records. Here is an example:

declassed.art. IN MX 10 mail.declassed.art.

Of course, the name mail.declassed.art should be resolvable to a specific IP address, no matter how. After adding MX record you can start smtpd and try to receive an email. Of course this will result in error because Dovecot is not running yet and the email can't be delivered to a recipient, but the reception process should be successful, as you can see from logs.

Other DNS records allow sending email messages without treating them as spam by recipients.

Add SPF record:

declassed.art. IN TXT "v=spf1 mx -all"

Add DKIM record:

20210512._domainkey.declassed.art. IN TXT ("v=DKIM1; t=s; h=sha256; p=long long long public
                                        key from file /usr/local/etc/mail/dkim/declassed.art.key.pub;")

Note that TXT value is split in multiple lines because lines in BIND zone files should not be longer than 256 characters. 20210512 is the selector used in /usr/local/etc/rspamd/local.d/dkim_signing.conf

Add DMARC record:

_dmarc.declassed.art. IN TXT  "v=DMARC1;p=reject;aspf=s;adkim=s;pct=100;fo=1;ruf=mailto:axy@declassed.art;"

This is quite restrictive record to detect misconfiguration and misuse.

Dovecot

Copy sample configuration:

cp -a /usr/local/etc/dovecot/example-config/* /usr/local/etc/dovecot/

Generate Diffie-Hellman parameters:

openssl dhparam -out /usr/local/etc/dovecot/dh.pem 4096

Generate /etc/mail/passwd.dovecot, based on /etc/mail/passwd (see above).

Make changes to the configuration:

/usr/local/etc/dovecot/dovecot.conf
-----------------------------------

#protocols = imap pop3 lmtp submission
+protocols = imap lmtp


/usr/local/etc/dovecot/conf.d/10-auth.conf
------------------------------------------

-!include auth-system.conf.ext
+#!include auth-system.conf.ext

+
+passdb {
+  driver = passwd-file
+  args = scheme=SHA512-CRYPT username_format=%Lu /etc/mail/passwd.dovecot
+}
+
+userdb {
+  driver = passwd-file
+  args = username_format=%Lu /etc/mail/passwd.dovecot
+  override_fields = uid=vmail gid=vmail home=/var/vmail/%Lu
+}


/usr/local/etc/dovecot/conf.d/10-mail.conf
------------------------------------------

-#mail_location =
+mail_location = maildir:~/Maildir


/usr/local/etc/dovecot/conf.d/10-master.conf
--------------------------------------------

service lmtp {
unix_listener lmtp {
    #mode = 0666
+    user  = vmail
+    group = vmail
}


/usr/local/etc/dovecot/conf.d/10-ssl.conf
-----------------------------------------

#ssl = yes
+ssl = required

-ssl_cert = &lt;/etc/ssl/certs/dovecot.pem
-ssl_key = &lt;/etc/ssl/private/dovecot.pem
+ssl_cert = &lt;/etc/mail/certificates/mail.declassed.art.crt
+ssl_key = &lt;/etc/mail/certificates/mail.declassed.art.key

-#ssl_dh = &lt;/usr/local/etc/dovecot/dh.pem
+ssl_dh = &lt;/usr/local/etc/dovecot/dh.pem

-#ssl_min_protocol = TLSv1
+ssl_min_protocol = TLSv1.2

#ssl_prefer_server_ciphers = no
+ssl_prefer_server_ciphers = yes

Enable and start services

Enable services in /etc/rc.conf:

rspamd_enable="YES"
smtpd_enable="YES"
dovecot_enable="YES"

Start services:

service rspamd start
service dovecot start
service smtpd start

What about email client???

I'm using Thunderbird. It has a few bugs I encountered immediately, but it works for me, basically.

You can try out any SMTP/IMAP client. Connection settings for SMTP: port 587, STARTTLS. For IMAP the port is 993, SSL/TLS. Authentication method is "Normal password" for both.

I dislike all web-based clients. Some aren't bad but I want my own. I'm a programmer at last and I'm passionate about email as you could notice. But currently I have no time for this.

Miscellaneous notes

Don't forget to setup automatic renewal of TLS certificate.

Don't forget to renew DKIM keys, once a year.

Don't forget to renew SRS key, once a year.

Don't forget to setup a failover SMTP relay.

How to check smtpd configuration:

smtpd -n

How to test correctness of DNS records: send email to a gmail account. If it drops the message to spam, view message source. They display the status of DKIM check.