Self-Hosted Email: Running Mailcow and Mail-in-a-Box in Your Homelab

Self-hosting email is one of those homelab projects that sounds straightforward, teaches you an enormous amount about internet infrastructure, and then punches you in the face with deliverability problems. It's a rite of passage. You should do it at least once, even if you end up going back to a hosted provider.

Here's the honest truth upfront: running your own email server is totally doable for receiving mail, internal communication, and learning. It's significantly harder for reliably sending mail to Gmail, Outlook, and Yahoo accounts because those providers are increasingly hostile to small mail servers. We'll cover both the setup and the strategies for dealing with deliverability.

Let's get into it.

Why Self-Host Email (And Why Not)

Reasons to Self-Host

• Full control over your data — No provider scanning your email for ads
• Custom domains made easy — Unlimited addresses, catch-all, aliases
• Learning experience — You'll understand DNS, SMTP, TLS, and authentication deeply
• Internal homelab email — Server notifications, cron reports, monitoring alerts
• Privacy — Your email stays on your hardware

Reasons Not To

• Deliverability is a constant battle — IP reputation, blacklists, and big provider gatekeeping
• It's a 24/7 responsibility — Email has to work. If your server goes down at 2 AM, mail bounces
• Security is critical — An open relay or misconfigured server can get you blacklisted instantly
• Big providers are hostile — Gmail and Microsoft routinely reject mail from small servers even when everything is configured perfectly
• Spam management is ongoing work — You'll need to maintain spam filters and deal with false positives

My recommendation: run a self-hosted email server for homelab notifications, internal communication, and learning. For important personal or business email that absolutely must reach Gmail/Outlook, use a hosted provider or a smarthost relay (more on that later).

The Two Best Options: Mailcow vs. Mail-in-a-Box

Both are excellent. Mailcow gives you more control. Mail-in-a-Box gives you a working email server in 15 minutes.

Prerequisites

Before setting up either solution, you need:

1. A Static Public IP (or Dynamic DNS)

Your email server needs to be reachable from the internet. Most ISPs assign dynamic IPs, which is a problem because:

• You need a consistent PTR (reverse DNS) record
• IP reputation matters, and dynamic IPs are often pre-blacklisted
• DNS records need to point to a stable address

If your ISP offers a static IP, get it. If not, consider running your email server on a small VPS (a $5/month VPS works fine) and connecting it to your homelab via VPN.

2. Port 25 Unblocked

Many residential ISPs block outbound port 25 to prevent spam. Check:

# Test if port 25 is blocked
telnet smtp.gmail.com 25

# If you get "Connected to smtp.gmail.com", port 25 is open
# If it hangs or times out, your ISP is blocking it

If port 25 is blocked, you'll need either a VPS or a smarthost relay for outbound mail.

3. A Domain Name

You need a domain for your email. Something like example.com so your addresses are [email protected].

4. Reverse DNS (PTR Record)

Your IP address needs a PTR record that resolves to your mail server's hostname. This is set by your ISP or VPS provider, not in your domain's DNS.

# Check your current PTR record
dig -x YOUR_IP_ADDRESS +short

# It should return something like:
# mail.example.com.

Without a PTR record, most email providers will reject your mail immediately.

Deploying Mailcow with Docker

Mailcow is the more flexible option and my personal recommendation for homelabbers who want to learn the internals.

System Requirements

• 2+ CPU cores
• 4GB+ RAM (6GB+ recommended with ClamAV)
• 20GB+ disk space
• Docker and Docker Compose v2
• A domain name with DNS access

Step 1: Install Docker

# On Debian/Ubuntu
sudo apt update && sudo apt install -y curl git
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER

# Verify
docker --version
docker compose version

Step 2: Clone and Configure Mailcow

cd /opt
sudo git clone https://github.com/mailcow/mailcow-dockerized.git mailcow
cd /opt/mailcow

# Run the configuration script
sudo ./generate_config.sh

The script will ask for your mail server hostname. Use something like mail.example.com.

Now edit the generated configuration:

sudo nano mailcow.conf

Key settings to review:

# mailcow.conf — key settings

# Your mail server's FQDN
MAILCOW_HOSTNAME=mail.example.com

# HTTP and HTTPS ports (change if you have a reverse proxy)
HTTP_PORT=80
HTTPS_PORT=443
HTTP_BIND=0.0.0.0
HTTPS_BIND=0.0.0.0

# SMTP ports
SMTP_PORT=25
SMTPS_PORT=465
SUBMISSION_PORT=587

# IMAP ports
IMAP_PORT=143
IMAPS_PORT=993

# Timezone
TZ=America/New_York

# Skip ClamAV to save RAM (optional, saves ~1-2GB)
SKIP_CLAMD=y

# Skip Solr full-text search to save RAM (optional)
SKIP_SOLR=y

# API key (generate a strong one)
API_KEY=your-random-api-key-here
API_ALLOW_FROM=127.0.0.1,::1

Step 3: Start Mailcow

cd /opt/mailcow
sudo docker compose pull
sudo docker compose up -d

This will download and start about 15 containers. Give it a few minutes.

# Check that everything is running
sudo docker compose ps

# You should see containers like:
# mailcow-postfix-1      running
# mailcow-dovecot-1      running
# mailcow-rspamd-1       running
# mailcow-nginx-1        running
# mailcow-mysql-1        running
# mailcow-redis-1        running
# mailcow-sogo-1         running
# ... and more

Step 4: Access the Admin Panel

Open https://mail.example.com in your browser. The default admin credentials are:

• Username: admin
• Password: moohoo

Change these immediately.

From the admin panel, you can:

• Add domains
• Create mailboxes
• Configure aliases
• View mail queue
• Check rspamd statistics

Step 5: Add Your Domain and Mailbox

In the Mailcow admin panel:

1. Go to Configuration > Mail Setup > Domains
2. Add your domain (e.g., example.com)
3. Go to Mailboxes and create your first mailbox

Mailcow will show you the DNS records you need to create. More on that in the DNS section below.

Deploying Mail-in-a-Box

Mail-in-a-Box is the "it just works" option. It sets up everything on a single Ubuntu server with one command.

System Requirements

• A fresh Ubuntu 22.04 LTS server (dedicated — no other services)
• 1GB+ RAM (2GB recommended)
• 10GB+ disk space
• A public IP with a PTR record

Step 1: Set Up the Server

Start with a fresh Ubuntu 22.04 installation. Mail-in-a-Box expects to be the only thing running on the server.

# Set the hostname
sudo hostnamectl set-hostname box.example.com

# Update the system
sudo apt update && sudo apt upgrade -y

Step 2: Run the Installer

curl -s https://mailinabox.email/setup.sh | sudo bash

The installer will walk you through:

1. Your email address (the first admin account)
2. Your hostname (e.g., box.example.com)
3. Country code for Let's Encrypt

That's it. Seriously. The installer handles Postfix, Dovecot, Roundcube, spamassassin, ClamAV, Let's Encrypt, DNS, and more.

Step 3: Access the Admin Panel

After installation, open https://box.example.com/admin in your browser. Log in with the email and password you provided during setup.

The admin panel shows you:

• System Status — Whether everything is configured correctly
• DNS Records — Exactly what DNS records you need to set
• Mailboxes — Manage users and aliases
• TLS Certificates — Let's Encrypt status

The system status page is incredibly helpful. It checks everything and tells you exactly what's wrong and how to fix it.

DNS Records: The Critical Part

Email authentication lives in DNS. Get this wrong and your mail will be rejected or marked as spam by every major provider.

Required DNS Records

Here's every DNS record you need, using example.com as the domain and mail.example.com as the mail server:

; MX record — tells the world where to send mail for your domain
example.com.        IN  MX  10 mail.example.com.

; A record — points your mail server hostname to its IP
mail.example.com.   IN  A   203.0.113.50

; SPF record — declares which servers can send mail for your domain
example.com.        IN  TXT "v=spf1 mx a:mail.example.com -all"

; DKIM record — public key for email signing (generated by your mail server)
; The selector varies — Mailcow uses "dkim", Mail-in-a-Box generates its own
dkim._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhki..."

; DMARC record — policy for handling failed SPF/DKIM checks
_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]; pct=100"

; Autodiscover/Autoconfig — helps email clients find your server
_autodiscover._tcp.example.com. IN SRV 0 1 443 mail.example.com.
autoconfig.example.com.         IN CNAME mail.example.com.

; MTA-STS — enforce TLS for inbound mail (optional but recommended)
_mta-sts.example.com.  IN TXT "v=STSv1; id=20260209"
mta-sts.example.com.   IN CNAME mail.example.com.

; TLSRPT — TLS reporting (optional)
_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:[email protected]"

PTR Record (Reverse DNS)

This is set by your ISP or VPS provider, not in your domain's DNS panel:

# Your IP should resolve to your mail hostname
dig -x 203.0.113.50 +short
# Expected: mail.example.com.

Contact your ISP or VPS provider to set the PTR record.

Understanding SPF

SPF (Sender Policy Framework) tells receiving servers which IPs are allowed to send mail for your domain.

; Allow mail from your MX servers, the server at mail.example.com, and nothing else
example.com. IN TXT "v=spf1 mx a:mail.example.com -all"

; If you also use a smarthost relay:
example.com. IN TXT "v=spf1 mx a:mail.example.com include:relay.example.net -all"

The -all at the end means "reject everything not listed." Use ~all (soft fail) during testing.

Understanding DKIM

DKIM (DomainKeys Identified Mail) signs each outgoing email with a private key. The recipient verifies the signature using the public key in your DNS.

Mailcow generates DKIM keys automatically:

# In Mailcow, get your DKIM public key
cd /opt/mailcow
sudo docker compose exec rspamd-mailcow cat /var/lib/rspamd/dkim/example.com.dkim.pub

Mail-in-a-Box shows the DKIM record in the admin panel under DNS.

Understanding DMARC

DMARC (Domain-based Message Authentication, Reporting, and Conformance) tells receiving servers what to do when SPF and DKIM checks fail.

; Quarantine failed messages (send to spam folder)
_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]; pct=100"

; Reject failed messages outright (use after you're confident everything works)
_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=mailto:[email protected]; pct=100"

Start with p=quarantine and monitor the reports. Once you're confident your SPF and DKIM are working correctly, switch to p=reject.

Verifying Your DNS Setup

# Check MX records
dig MX example.com +short
# Expected: 10 mail.example.com.

# Check SPF
dig TXT example.com +short
# Should include your SPF record

# Check DKIM
dig TXT dkim._domainkey.example.com +short
# Should return your DKIM public key

# Check DMARC
dig TXT _dmarc.example.com +short
# Should return your DMARC policy

# Check PTR
dig -x 203.0.113.50 +short
# Expected: mail.example.com.

Use online tools for a more thorough check:

• MXToolbox — Comprehensive email health check
• mail-tester.com — Send a test email and get a score out of 10
• learndmarc.com — Visual DMARC analysis

TLS Certificates with Let's Encrypt

Both Mailcow and Mail-in-a-Box handle Let's Encrypt automatically, but here's what's happening under the hood.

Mailcow TLS Configuration

Mailcow uses acme.sh for certificate management. Certificates are obtained automatically when you start the stack, as long as:

1. Port 80 is reachable from the internet
2. Your MAILCOW_HOSTNAME DNS record points to the server

# Check certificate status
cd /opt/mailcow
sudo docker compose exec acme-mailcow openssl x509 \
  -in /var/lib/acme/cert.pem -text -noout | grep -A2 "Validity"

# Force certificate renewal
sudo docker compose exec acme-mailcow /usr/local/bin/acme.sh \
  --renew --domain mail.example.com --force

Manual TLS Configuration

If you're behind a reverse proxy or need custom certificate handling:

# In mailcow.conf, disable built-in ACME
SKIP_LETS_ENCRYPT=y

# Mount your own certificates
# Edit docker-compose.override.yml:

# docker-compose.override.yml
services:
  nginx-mailcow:
    volumes:
      - /etc/letsencrypt/live/mail.example.com/fullchain.pem:/etc/ssl/mail/cert.pem:ro
      - /etc/letsencrypt/live/mail.example.com/privkey.pem:/etc/ssl/mail/key.pem:ro
  postfix-mailcow:
    volumes:
      - /etc/letsencrypt/live/mail.example.com/fullchain.pem:/etc/ssl/mail/cert.pem:ro
      - /etc/letsencrypt/live/mail.example.com/privkey.pem:/etc/ssl/mail/key.pem:ro
  dovecot-mailcow:
    volumes:
      - /etc/letsencrypt/live/mail.example.com/fullchain.pem:/etc/ssl/mail/cert.pem:ro
      - /etc/letsencrypt/live/mail.example.com/privkey.pem:/etc/ssl/mail/key.pem:ro

Spam Filtering: rspamd

Mailcow uses rspamd for spam filtering, which is modern, fast, and highly configurable.

Accessing the rspamd Web Interface

Mailcow exposes rspamd's web UI at https://mail.example.com/rspamd. The password is set in your mailcow.conf.

Key rspamd Configuration

# rspamd local overrides (create these in the mailcow data directory)
# /opt/mailcow/data/conf/rspamd/local.d/

# Adjust spam thresholds
# local.d/actions.conf
reject = 15;          # Reject messages scoring above 15
add_header = 6;       # Add spam header above score 6
greylist = 4;         # Greylist above score 4

Training rspamd

rspamd learns from the messages you mark as spam or ham:

# Train a message as spam (from the command line)
sudo docker compose exec rspamd-mailcow rspamc learn_spam < /path/to/spam-message.eml

# Train a message as ham (not spam)
sudo docker compose exec rspamd-mailcow rspamc learn_ham < /path/to/ham-message.eml

In SOGo (Mailcow's webmail), moving a message to the Junk folder automatically trains it as spam. Moving it back trains it as ham.

Greylisting

Greylisting temporarily rejects the first delivery attempt from unknown senders. Legitimate mail servers retry; spambots usually don't.

# local.d/greylist.conf
# Adjust greylisting behavior
expire = 7d;          # Remember known senders for 7 days
timeout = 5min;       # Reject for 5 minutes before allowing retry
whitelisted_ip = [
  "127.0.0.0/8",
  "10.0.0.0/8",       # Your local network
];

The Deliverability Problem

Here's where self-hosted email gets painful. You can have perfect DNS records, perfect SPF/DKIM/DMARC, a clean IP, and still get rejected by Gmail or Microsoft.

Why Big Providers Reject Small Servers

1. IP reputation — Your IP has no sending history, so it's untrusted by default
2. IP range reputation — Your ISP's IP range might be flagged because other users sent spam
3. Volume — You send 10 emails a day. Gmail expects legitimate mail servers to send thousands
4. Residential IP blocks — Many providers auto-reject residential IP ranges

Checking Your IP Reputation

# Check if your IP is on any blacklists
# Use MXToolbox: https://mxtoolbox.com/blacklists.aspx

# Or from the command line, check common blacklists:
for bl in zen.spamhaus.org bl.spamcop.net b.barracudacentral.org; do
  result=$(dig +short $(echo YOUR_IP | awk -F. '{print $4"."$3"."$2"."$1}').$bl)
  if [ -n "$result" ]; then
    echo "LISTED on $bl: $result"
  else
    echo "Clean on $bl"
  fi
done

Warming Up Your IP

If you're on a VPS with a clean IP, you need to "warm up" your sending reputation:

1. Start by sending a few emails per day to different providers
2. Ask friends with Gmail/Outlook to mark your email as "not spam" if it lands there
3. Set up proper authentication (SPF, DKIM, DMARC) from day one
4. Don't send bulk mail from a new IP
5. Wait 2-4 weeks before expecting reliable delivery to major providers

The Smarthost Relay Solution

The most practical approach for reliable outbound delivery is to use a smarthost — a third-party SMTP relay service that has established IP reputation.

Popular smarthost options:

Configuring Mailcow to Use a Smarthost

In the Mailcow admin panel:

1. Go to Configuration > Mail Setup > Routing
2. Add a sender-dependent transport
3. Set the smarthost details

Or configure it directly in Postfix:

# /opt/mailcow/data/conf/postfix/extra.cf
# Add smarthost relay configuration

relayhost = [smtp.mailgun.org]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/opt/mailcow/data/conf/postfix/smarthost_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt

# /opt/mailcow/data/conf/postfix/smarthost_passwd
[smtp.mailgun.org]:587 [email protected]:your-mailgun-smtp-password

# Generate the password database
cd /opt/mailcow
sudo docker compose exec postfix-mailcow postmap /opt/postfix/conf/smarthost_passwd
sudo docker compose restart postfix-mailcow

This way, you receive mail directly on your server (keeping control) but send through a relay with good reputation (ensuring deliverability). It's the best of both worlds.

Monitoring and Maintenance

A mail server needs regular attention. Here's what to monitor and how.

Log Monitoring

# Mailcow — view mail logs
cd /opt/mailcow
sudo docker compose logs -f postfix-mailcow
sudo docker compose logs -f dovecot-mailcow

# Check for delivery failures
sudo docker compose logs postfix-mailcow | grep "status=bounced"

# Check for authentication failures (potential brute force)
sudo docker compose logs dovecot-mailcow | grep "auth failed"

# Mail-in-a-Box — mail logs
sudo tail -f /var/log/mail.log

Mail Queue Monitoring

# Mailcow — check the mail queue
sudo docker compose exec postfix-mailcow mailq

# Count queued messages
sudo docker compose exec postfix-mailcow mailq | tail -1

# Flush the queue (retry all queued messages)
sudo docker compose exec postfix-mailcow postfix flush

# Delete all queued messages (nuclear option)
sudo docker compose exec postfix-mailcow postsuper -d ALL

Automated Health Checks

Create a simple monitoring script:

#!/bin/bash
# mail-health-check.sh — Check email server health
set -euo pipefail

MAIL_HOST="mail.example.com"
ALERT_EMAIL="[email protected]"
NTFY_TOPIC="homelab-alerts"

check_failed=0

# Check SMTP
if ! timeout 10 bash -c "echo QUIT | openssl s_client -connect ${MAIL_HOST}:465 -quiet 2>/dev/null | head -1 | grep -q '220'"; then
  echo "SMTP (465) check failed"
  check_failed=1
fi

# Check IMAP
if ! timeout 10 bash -c "echo QUIT | openssl s_client -connect ${MAIL_HOST}:993 -quiet 2>/dev/null | head -1 | grep -q 'OK'"; then
  echo "IMAP (993) check failed"
  check_failed=1
fi

# Check submission port
if ! timeout 10 bash -c "echo QUIT | openssl s_client -connect ${MAIL_HOST}:587 -starttls smtp -quiet 2>/dev/null | head -1 | grep -q '220'"; then
  echo "Submission (587) check failed"
  check_failed=1
fi

# Check web interface
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${MAIL_HOST}")
if [ "$HTTP_CODE" != "200" ] && [ "$HTTP_CODE" != "302" ]; then
  echo "Web interface returned HTTP $HTTP_CODE"
  check_failed=1
fi

# Check mail queue size
if command -v docker &>/dev/null; then
  QUEUE_SIZE=$(docker compose -f /opt/mailcow/docker-compose.yml exec -T postfix-mailcow mailq 2>/dev/null | tail -1 | grep -oP '\d+ Request' | grep -oP '\d+' || echo "0")
  if [ "${QUEUE_SIZE:-0}" -gt 50 ]; then
    echo "Mail queue has $QUEUE_SIZE messages (threshold: 50)"
    check_failed=1
  fi
fi

if [ $check_failed -eq 1 ]; then
  curl -s -o /dev/null "https://ntfy.sh/${NTFY_TOPIC}" \
    -d "Email server health check failed on ${MAIL_HOST}"
else
  echo "All checks passed"
fi

Regular Maintenance Tasks

Backing Up Your Mail Server

# Mailcow backup script (built-in)
cd /opt/mailcow
sudo ./helper-scripts/backup_and_restore.sh backup all

# Manual backup of key directories
sudo tar czf /backup/mailcow-data-$(date +%Y%m%d).tar.gz /opt/mailcow/data/

# Backup individual mailboxes (via IMAP)
# Install imapsync
sudo apt install imapsync

# Sync a mailbox to another IMAP server
imapsync \
  --host1 mail.example.com --user1 [email protected] --password1 'password' \
  --host2 backup-imap.example.com --user2 [email protected] --password2 'password' \
  --ssl1 --ssl2

Hardening Your Mail Server

An email server exposed to the internet is a high-value target. Lock it down.

Fail2ban for Brute Force Protection

Mailcow includes netfilter (fail2ban equivalent) by default. Check its configuration:

# /opt/mailcow/data/conf/rspamd/local.d/ — rspamd handles rate limiting
# Mailcow's built-in netfilter handles IP banning

# Check banned IPs
cd /opt/mailcow
sudo docker compose exec netfilter-mailcow cat /tmp/fail2ban_blockip

# Manually unban an IP
sudo docker compose exec netfilter-mailcow /usr/local/bin/unban.sh 192.168.1.100

For Mail-in-a-Box, fail2ban is configured automatically.

Firewall Rules

Only open the ports you need:

# Required ports for email
sudo ufw allow 25/tcp    # SMTP (receiving mail)
sudo ufw allow 465/tcp   # SMTPS (sending mail, implicit TLS)
sudo ufw allow 587/tcp   # Submission (sending mail, STARTTLS)
sudo ufw allow 143/tcp   # IMAP (optional, prefer IMAPS)
sudo ufw allow 993/tcp   # IMAPS (reading mail)
sudo ufw allow 80/tcp    # HTTP (Let's Encrypt + webmail redirect)
sudo ufw allow 443/tcp   # HTTPS (webmail + admin)
sudo ufw enable

Rate Limiting

# Postfix rate limiting (add to /opt/mailcow/data/conf/postfix/extra.cf)
smtpd_client_connection_rate_limit = 30
smtpd_client_message_rate_limit = 100
smtpd_client_recipient_rate_limit = 200
anvil_rate_time_unit = 60s

Testing Your Setup

After everything is configured, run through this checklist:

# 1. Send a test email to mail-tester.com
#    Go to https://www.mail-tester.com, get a test address, send an email to it
#    Aim for a score of 9/10 or higher

# 2. Check all authentication records
dig MX example.com +short
dig TXT example.com +short        # SPF
dig TXT _dmarc.example.com +short # DMARC
dig -x YOUR_IP +short             # PTR

# 3. Test SMTP connectivity
openssl s_client -connect mail.example.com:465
openssl s_client -connect mail.example.com:587 -starttls smtp

# 4. Test IMAP connectivity
openssl s_client -connect mail.example.com:993

# 5. Send test emails to major providers
# Send to Gmail, Outlook, Yahoo, and check:
# - Did it arrive in inbox (not spam)?
# - Are SPF, DKIM, DMARC all passing? (Check email headers)

# 6. Check email headers for authentication results
# In Gmail, click "Show original" on a received message
# Look for:
#   Authentication-Results:
#     spf=pass
#     dkim=pass
#     dmarc=pass

The Realistic Setup

After going through all of this, here's what I actually recommend for most homelabbers:

1. Run Mailcow on a $5 VPS with a clean IP and proper PTR record
2. Connect it to your homelab via WireGuard so it feels local
3. Use a smarthost relay for outbound mail to big providers
4. Receive mail directly — inbound deliverability isn't a problem
5. Use it for homelab notifications — monitoring alerts, cron reports, service notifications
6. Keep a hosted provider for important personal/business email until your reputation is established

Self-hosted email is a fantastic learning project and genuinely useful for homelab infrastructure. Just go in with realistic expectations about deliverability, and have a plan for when Gmail decides your perfectly configured server isn't good enough.

Your mail, your rules. Just maybe relay it through someone Gmail trusts.