Is Gmail killing independent email?
People report that self-hosted emails always end up in Gmail spam. Is there anything Google can do about it?
Email works with everyone
Email is a great protocol as it basically works with everyone. Since the early days of the internet, people are able to send and receive messages via email to and from any email server. Later on email services like Hotmail - now Outlook, Yahoo and Gmail joined the party so that people and businesses could just create an email address with these services - instead of hosting their own email server. This made email the most widely used communication tool worldwide - and it still is
The use of email services made things incredibly convenient because the services took care of setting up the servers, including setting up SPF, DKIM and DMARC records to make sure the email is being received by other email providers.
Regardless, to this day it is possible to host your own mail server, and lots of companies do so. However, it seems that even if you get SPF, DKIM etc. right, Gmail does not like it when you send emails from your own mail servers.
Recently a company from new Zealand, School interviews, complained that their emails to parents that expect booking confirmations from this site do not receive those due to rate limiting in Gmail.
Gmail is rate-limiting emails
"Those blue dots show over 3,500 Gmail customers having the booking confirmation email they asked for delayed by up to 12 hours. Our support people get several calls a day asking about missing confirmation emails, and they wearily explain that Gmail is delaying delivery for no good reason."
The actual error from Google according to School interviews is:
Our system has detected an unusual rate of 421-4.7.28 unsolicited mail originating from your IP address. To protect our 421-4.7.28 users from spam, mail sent from your IP address has been temporarily 421-4.7.28 rate limited.
Given how many people on Reddit complain that their emails from self-hosted servers do not arrive in the Gmail inbox at all, but in spam, School interviews is still in a rather good position. But to do business, timely reception of these emails is paramount.
Can Gmail do better?
On Hacker News people argue that Google has no other choice because setting up your mail servers correctly does not prove that you are not a spammer - today even spammers are able to set up their mail server correctly. Thus, newly set up mail servers are regarded as spammers by huge email providers in general:
They say "Both email servers have PTR records set up, and SPF [...] DKIM, and DMARC records[...]". Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer.
Or:
I understand this guy's frustration but this: "And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem." is just wrong. Tons of spam comes from servers with SPF, DKIM and DMARC now. It stopped being a trustworthy signal of not-spam many years ago.
However, the main question remains: Is it okay that Gmail has the power to decide whether a business is sending spam or not?
At the very least, Gmail support team should have listened to the company and looked into the issue to fix it.
If Google is not willing to do this, it is just another sign of how Google can abuse their market power and hinder smaller services or - in this case - self-hosting emails, limiting the options people and businesses have when they want that their emails are reliably received by Gmail.
The question that remains is this:
Should I host my one mail server?
This has been explained quite nicely on Reddit:
How can you tell if you should host your own mail server? Ask these questions:
Am I wanting to send and receive email with other domains? If not, some of these questions won't apply so keep that in mind.
Am I willing to do some work to make this work? If not, stop.
Is this my first venture into Linux or Docker or self hosting or any kind of new technology? If it is, stop. Host something else first, get your feet under you.
Will my ISP allow incoming unfiltered access to ports 25, 143, 465, 587, 993 to my IP? If not, stop.
Do I have a static IP which will also let me edit my rDNS entry? If not, Stop. Your ability to prove you aren't spam relies a lot on rDNS. Check my domain mx records (which is my username) to see what I mean. I have a static IP with Verizon FiOS and it is business class. Anything less, and I can't touch that entry. I also pay more for internet because of this
Will I do the due diligence of receiving and looking at the SPF and DMARC reports you can get about your email? If not, stop. These are daily (or weekly) emails from other domains about any issues they saw. You need to pay attention to these and if you don't, you do so at your own peril.
Lastly, am I willing to bear the burden of being the coolest of all of my friends since I host my own email? If not, that's fine you won't be the coolest, but I have to say I love hosting my own email, I would never go back to anything else.
Still looking to self-host?
If the list above still makes you feel comfortable of hosting your own emails, also read this amusing explanation on "Why you really DON'T want to self-host your own e-mail server":
So, you are thinking about hosting your very own e-mail server, and that can't be too hard, right? Ah, gather round as we go through the options and understand why the answer to almost every IT Question is "it depends"! :) At a basic level, e-mail is a simple protocol, especially on the Internet. If you are talking about a LOCAL ONLY server, which is just sending e-mail within a single "site" -- that is fairly easy. What gets more complicated is when you want to talk to others on the Internet, and you want them to be able to talk to you.
The main problem, frankly, is SPAM. Unsolicited bulk e-mail, ads, scams, junk mail -- we'll put them under the generic heading of "spam". This is in contrast to "ham", or the e-mails you want, from people you know and care about. Now, I hear some of you in the back -- saying "who cares", because really, I can sort the wheat from the chaff, so no biggie. But this spam concern isn't just for e-mail flowing TO you, it's also a major concern for e-mail flowing OUT (or FROM you) -- how do providers out there know you aren't just another J. random spammer?
History time: In the beginning, the Internet was small and people knew and trusted each other. Out of this trust was born the protocol that forms the foundation of our modern email, "Simple Mail Transfer Protocol", or SMTP. It was written with quick transport in mind, and didn't incorporate any security or validation to speak of -- meaning it was easy to pretend to be someone else, or "spoof" addresses. Also, early on, machines were anemic in processing (by todays standards) so no one used encryption, meaning all e-mail was sent in the clear. Now, you should know it was hard to get on the Internet early, it was fairly pricey so that limited who had (direct) access. But as the Internet expanded, no one thought too much about problems that might show up from the open/trusting nature of protocols, SMTP included.
Still here? Okay, let's talk about how to tame some of this spam problem. For incoming mail, there are a number of things you can do. If you run a smaller volume site (like at home), you might be able to use a client-side software which has something called a Bayesian filter -- you train the filter by marking e-mails, and then the filter "scores" the emails. Very effective, but lots of end-user effort required. While not strictly spam, unwanted malicious e-mails (virus, trojan, etc) might be something you want to scan for. You might want one to scan attachments automatically, to try and prevent you and your users from getting nasties. NOTE: Such tools are not 100% effective, so you still have to have an awareness. Going back to our core spam problem, one thing we can do to help this out is to make sure we aren't contributing to the problem, and even looking at tools that restrict known spammers. For the first part, we want to make sure we "relay", or accept and then forward only e-mail we really mean to. We do this by restricting which machines can send, or even forcing them to authenticate (provide user/pass) before accepting e-mail from them. For the second part -- known spammers, it would be using so called lookup lists (sometimes called blacklists or blackholes) which dynamically track machines spamming. Sometimes this requires a subscription. For outgoing mail, the problem becomes a bit more challenging. Now instead of controlling the mail, you are at the mercy of remote admins who know that MOST new e-mail servers are spam, and why should they think YOUR new server isn't? Some of this is patience -- if you run your server well and DON'T spam, your reputation will improve with time. But a lot of this is a labyrinth of layers set up over the years to help figure out if you are really a spammer or not. With names like SPF and DKIM, they seem weird and hard to figure out, but it's a matter of setting them up right. The usually have a bit of magic needed in your DNS (Domain Name System) records, and they can have a bit of software you run, which "signs" e-mails. Now none of this prevents you from spamming, but spammers have to send LOTS of spam to be effective, and providers only have to now let them send lots (lots here are millions of messages). But the signed messages and DNS help to track, and block somewhat quickly -- or at least if a sender isn't themselves a spammer -- have a communications channel to alert a good e-mail manager that someone is abusing their system.
Wait, are you STILL here? :) If after all that, you still want to run your own e-mail -- you still have to worry about site to site encryption, how do your users read their e-mail (webmail or clients), things like storage issues (how big can e-mail boxes get?) and debugging sending problems (eg your significant other is mad their e-mail isn't getting to their family member inside a 5 minute window). Not to mention things like having your e-mail show up in recipients spam and junk, going through a bunch of hoops, and it still lands in junk.
Okay, if you are still here: Congrats, you now know it won't be easy. As a long veteran of running company e-mail servers, there are lots of things you have to worry about. TIPS: Make some things easy on yourself, though. First, when getting an IP from a provider, do some checking to see if it's CURRENTLY or was RECENTLY on a blackhole list. (This ONLY applies to e-mail servers. If you are running a web server that never e-mails anybody -- who cares if it was on a list). Choose a mainstream TLD (top level domain) for your first/early e-mail domain, something ending like .COM/.NET/.ORG or an established country one, like .us/.ca/.eu etc. Try to stay away from esoteric domains or "newer" TLDs for e-mail -- some badly coded programs will choke on newer domains, and some big providers seem to frown on newer TLDs. CAVEAT: Any NEW domain, no matter WHERE it is registered, will take a bit of time to "prove itself" is non-spammy. So, if you want instant deliver-ability, use an already (non-spammy) domain name that currently exists. Pick an EASY to SPELL and SAY domain name for your e-mail. That cutesy domain which swaps "i" for "y" and is 30 letters ... good luck explaining all of that on the phone. Ideally you want a domain you can just say and people know it, like "hello world dot com" (already registered, not by me, just an example)
