The email landscape is substantially more complex now than it was three decades ago. The poster seems to be frustrated that he is mostly getting technical explanations to his rather emotionally fueled question. He is literally phishing for an argument on the morals of a big company deciding he is a potential spammer without absolute proof. He made it clear that his opinion is that there should be a system in place at Google that he can petition for his reputation, and that emails that look like spam should say 'potentially spam' rather than send it to a spam folder. All this while sending emails from an origin that is very well known for spammers and whose actions depict a typical bot account warming up. (Poster claims to on occasion initiate personal conversations with people who have their emails publicly listed, such as an author from a news site he might disagree with). There is just no reliable system today that allows for someone to host their email servers in a spam haven, not use relays, act like a spammer, and expect not to get your emails sent to spam folders.
It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
Rarely have I seen such concise and respectful way of describing the typical rants I see on HN way to often, bravo! I can only wish I had your writing skills.
> It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
Yes, I have been bitten by this many times. Every time I see a topic like this one I try to help that person by offering technical advice, basically doing my consulting job for free for everyone to read. And many times the post gets downvoted till its dead.
Slowly I learned that people don't want a solution to their problem, they just want to vent their (understandable) frustration about $BIGCORP. But it still hurts to get downvoted when you just try to help.
This is essentially HN with every single Google related post nowadays. Good or bad, people hate Google, they hate it the way they used to hate Microsoft, they see everything it does as being nefarious.
People ignore the email spammers, SEO spammers, automated password crackers and blame google for the solutions they put in place to combat those things.
It's impossible to reason with them, honestly I think it's just best to ignore posts like this.
Have you considered using a full TLD and hosting and/or relaying through a reputable service. I've used mailgun and sendgrid both for moderate accounts for outbound email and didn't have any issues. There's also hosting options for mail ranging from fastmail, to google, MS and others.
In the end, TFA is from a subdomain that seems to have a lot of spam, and to avoid it may mean using A commercial option, if not a specific commercial option. It sucks.. but the mail relays I mention above are relatively cheap and easy to properly setup with limited issues on the other side.
If you were doing everything right it would work, unless you sent email from an IP known to send spam in the past. That doesn't necessarily mean you did, but it does mean your IP did at some point. Given the saturation of the ipv4 space this isn't exceptionally unlikely. In any case, your post is contrarian and doesn't offer any substance aside from the fact that you are angry. That's kind of saddening because that was already pointed out as exactly the problem.
Believe it or not, its possible to carefully look into things and check them, but I understand its much easier to be smug and assume that its my fault.
Judging by your username I don't expect you to share the domain name here, but if you need a second opinion, send me a PM and I'll analyse the domain for you.
There is usually some misconfiguration somewhere that goes overlooked. Happens to the best of us :-)
I've thought about setting up a mail server on a DigitalOcean or Vultr VPS without any experience with mail servers (for person use). Whenever I read opinions about operating a private mail server, there's usually a few people that express great satisfaction in doing so, and say that it's not too difficult. However, there are also many more people that say it's not worth the effort since there's maintenance to consider, complex setup, and the cost of email providers is usually the same or less than running your own server.
What is your opinion (or others' if they'd like to chime in)? If you enjoy running your own, are there any guides in particular that you'd recommend? Does the future of email look even more prohibitively complex for self-hosting?
Self hosting has become harder due to email abuse (spam, phishing, impersonation, etc) and the measures against this that have been introduced.
Following a guide to setup a basic email service is quite easy to do, but after a while the lack of proper implementation and maintenance of the standards will start to cause deliverability issues. Those are frustrating and most end up writing a long rant on why Google is evil because their email ended up in spam. When in fact, it usually is a misconfiguration, but those can be really hard to detect.
IMO there is still great satisfaction in running any internet service yourself. But, if you rely on this service to work, it just no longer makes financial sense to do so.
It takes resources to keep any internet service running correctly. Even the simple stuff can quickly get overlooked, I have seen so many self-hosted email servers that have expired certificates (like with the domain of the OP) or lack support for any of the modern security extensions such as DMARC or MTA-STS.
For most businesses my recommendation is not to run it yourself. Like you wouldn't run your own DNS service anymore. Of course it can be done, and in some cases it makes sense to run stuff yourself (usually when operating at a very large scale). But it almost always makes more financial sense to use a hosted email solution.
We haven't written a guide (yet) on setting up an email server with all the bells and whistles. But if you want to read more on all the email security standards, have a look here: https://www.mailhardener.com/kb/
Thank you so much for the detailed reply. I'm fairly new to programming, development, Linux, the command line, etc etc, and so am always on the lookout to learn something new. My day job is web development, but I'm looking for something that I really want to dive deep into. Email seems to be a whole universe of its own! From what you've said, maybe I'll try it once to learn a bit more someday, but for now I'll let sleeping dogs lie and keep G Suite as my custom domain email host. Thanks again!
Honestly, running an E-Mail is so tedious that I just allow Migadu to manage it for me. You have to deal with configuration (as the original poster said) and run into issues with systems just blocking you outright (especially Outlook). There are some utilities or Docker images that make it easier, but in the end its not worth the hassle. I still think domain-based email is way better than allowing G-Mail or Outlook to host your email though.
When you say domain-based, you mean using a email-hosting provider with a custom domain, is that right? I currently have my custom domain set up with G Suite for email and am thinking that's probably good enough.
I ran my own for a number of years... it was a pain in the ass and took too much time for what it is/was worth. I'm a frugal man, but it's just not worth it to me. I now have several relays pointing to my gmail and/or a custom outlook.com domain (grandfathered free account). I've thought about putting something up, but it's a headache and a half.
For delivery, if you are really just using it for personal use a mailgun or sendgrid account can allow you to handle most delivery issues, while still handling inbound.
Which still leaves the other side, actually dealing with inbound mail issues which is just painful with either too much or too little filtering. The likes of google and MS have teams of brilliant people to handle dealing with spam, and despite some little guys caught in the middle, is still far better than I could ever do on my own.
Thanks for your take on it. I'm fairly new to the world of programming, development, Linux, etc., so I'm always looking to learn things and make something of my own. A mail server sounds very neat in that regard, like building your first computer, but it sounds like in reality it's best left to the experts. I currently have a G Suite account for my custom domain, so that's probably good enough.
I don't mean to discourage anyone who wants to experiment... It's actually really interesting with a lot of options in terms of running your own. It helps to learn things from the security aspects, to the use of tools for spam, greylisting, dns lists etc. On the flip side, I wouldn't do it for my primary email again.
In the end, if you want a career in IT, or find it interesting, I'd say go for it. I've often thought about building my own end to end open-source mail service aimed at ease of administration. If I was ever rich enough to not have to work, that's one of the things I'd probably do after a few months off.
I'd suggest starting with a secondary domain and using Mail-In-A-Box on a VPS or smaller Cloud host. It's a decent starting point, and there are many other options. One of my favorites is SmarterMail (commercial, windows only though). I also ran a BBS for a number of years using Synchronet, which does email/pop/smtop and even nntp for group messages. I did have it configured with SendGrid at the time for outbound for a while, which handled delivery issues for me.
You will have headaches if you take it seriously and/or use it as a primary service without using a delivery service like MailGun or SendGrid, and even then you probably will have other headaches. My point isn't really do discourage so much as let people have a more realistic understanding. The issues the OP has are real. However, there is so much junk from bad actors that the well is poison.
If this is the kind of technical advice you provide, which merely assumes that Google can do no wrong, and the user on the other end is incapable of pressing a button on a yet another website that checks email configuration, merely to confirm that the configuration of their mail server is not in any way incorrect, then it's not a surprise that you get downvoted into oblivion, according to your own statement earlier in this thread.
I can't provide any technical advice unless the op shows their config. I am downvoted because this community is already biased thinking Google must be at fault and the configuration can't be wrong at all ?.
I would say the community is biased the other way around, where everyone always thinks that it's the little guy that has invalid configuration, instead of Google being at fault.
What sort of config do you require from the OP? Their domain name is public and uncensored right in the top of the post. Their configuration has been cross-checked by many people so far. A lot of folks claimed that it has issues (like a `?all` in SPF, or `p=none` in DMARC, or some other nonsense that any spammer would easily get right), but those issues have been rebuked as not being significant to the issue at stake, plus, there's absolutely no confirmation that fixing these "issues" would resolve the problem, either.
Filtering SPAM mail with an acceptable false positive and false negative rate is part of the service.
Unfortunately for you, the e-mail providers' customers (the users receiving the mail) get to decide what is an unacceptably high false positive/negative rate --- and not you. If e-mail recipients get too much SPAM or if they get mail that they want landing in their Spam folder too often, then they will switch. This provides the economic incentive for mail providers to Get Things Right.
Unfortunately, there is so much spam out there that if you are too relaxed with letting marginal e-mails (or e-mails from marginal network neighborhoods through) the false negative rate could result in so much spam in users' inbox that they will get mad, and switch.
However, it can happen that the email provider can be too sensitive, and there can be too high of a false positive rate. For example MIT recently tried relying Microsoft's spam filtering system. Way too much stuff landed in the spam filter, and while you could go to the spam filter and manually mark e-mails as "not spam" and eventually Microsoft's ML algorithms would hopefully figure it out, the false positive rate was too high and the MIT community (especially the ones with the real power, e.g., the professors) rose up with an outcry, and MIT abandoned using Microsoft's Spam filtering service.
My experience is that the email provider's customers have very little say in the matter. I deal with complaints from Customers who use Office 365 and Gmail about false positives regularly. They aren't MIT-profile Customers, so their complaints are ignored. The complaint usually turns into "Why can't this sender just use Google or Microsoft for their email like 'everybody' else?"
It depends on the customer. If you're an Office 365 customer who is a major enterprise with huge numbers of paid users, and your CIO goes golfing with Microsoft board members, you have a huge amount of power. Similarly, if you are an MIT Professor you have vast amounts of power.
They can be pile driving right next to the machine room, and causing disk drive errors, and Physical Plant won't care if the I/T folks complain. But if some Sloan School professors complained that it was disturbing their cogitations, it immediately stopped and Phys. Plant called a "stakeholder meeting" and it was really obvious who had the power.
All customers are equal, but some customers are more equal than others.
"People irrationally hate X" may be perfectly true, but is not a reason why any particular, or most, or the most significant, criticisms of X are wrong or should be ignored.
Even the particular people who irrationally hate something are obviously likely to also run across the good reasons for hating it.
It's not reasonable to say "criticism X doesn't count anymore because we've heard it too many times".
Even technical people are people. This tip can help defuse the frustration, making the recipient more willing to listen to the technical solution.
People forget sometimes that the person on the other end of the screen isn’t a computer that is completely unmoved by silly things like emotion and frustration.
I don't think OP stop replying, he seems pretty active on another thread of the same mailing list concerning another big player (MSFT) filtering out some messages for some reasons.
https://www.mail-archive.com/mailop@mailop.org/msg09066.html
As a hobbit* , I'm actually pretty happy this whole conversation is starting to gain more traction.
My domain is always green on every test I could run (mailtester.com & all) yet some of my mails go to the junk without any warning. At least warn postmaster when you decide to block a domain because of bad neighbour/hosting!
> I'm actually pretty happy this whole conversation is starting to gain more traction.
This exactly. I've been a Gmail user since the day when they've required an invitation in order to signup, and I'm not quite a Google hater, either, but it's extremely disappointing to see the number of people who simply defend Google against the wrongdoing irrespective of whether or not the complainant has a valid complaint, like in this very case.
What happened to the whole notion of believing the victims? Why no matter how hard you try to explain your setup, and the things you've tried, and that all the automated tests pass, that you still get blamed to be the culprit for the issues you experience when dealing with someone who has a monopoly on the service at stake, instead of considering the possibility that maybe it's Google Mail having the monopoly on email that's an issue here.
Oh, I do think that Google is absolutely more aggressive than they probably should be on filtering... On the flip side, there's very little other choice. The VAST majority of email comes from a handful of hosts/providers and everything else has a much higher noise to signal ratio and gets much more scrutinized period.
Also, if you aren't sending a lot of mail, consider relaying outbound through mailgun or a similar service (I used to use sendgrid myself). Nearly zero issues when doing this on the lowest level paid account. That said, inbound spam was so painful to deal with I just gave it all up.
Even after reading that, I'm not sure what an email hobbit is. Someone who just wants to live in a cozy hole in the ground with a few nice email amenities?
It was an extended metaphor categorizing an ISP's (or a cloud provider) customers into multiple buckets: Kobolds, Lizardmen, Hobbits, and Princesses.
Princeses were the customres which sometimes required a lot of customer support, but paid really well because they purchased a lot of services/VM's. Hobbits were customers who basically minded their own business, and didn't guy a lot of stuff, but didn't cause any problems.
Lizardmen were customers which had possibly clueless marketing departments that would occasionally send e-mail marketing campaigns which crossed the line, but which also sent plenty of legitimate e-mails, and which were mostly trainable after the ISP/cloud provider smacked them on the nose with a rolled-up newspaper --- and Kobolds were spamvertising companies that did nothing but send spam, and were constantly switching ip addresses / VM's, and knew d*mn well that they were doing something evil, and were not trainable.
The problem is that if the ISP / cloud provider has the vast majority of their customers being Kobolds and Lizardmen, it becomes economically hard to just cut off the bad actors, because the vast majority of their customer base are bad actors. If the provider originally had a good mix of all of these customer mixes, but wasn't proactively cutting off the Kobolds and educating the Lizardmen, then their network block would start getting a bad reputation, and all of the major e-mail providers (not just GMail) would start blacklisting the netblock, or at the very least, treating any e-mail sent from that network block as a strong SPAM signal, and this would cause the other customer types to find other providers.
So if you happen to be a customer of such a network/cloud provider, you could try complaining to the provider, but they are really in between a rock and a hard place, because if you're the 1 good hobbit and all of the otheer customers for that provider are Kobolds and Lizardmen, what is the provider supposed to do? You're probably better off switching to a better provider. Unfortunately, the better provider may also cost more, but you get what you pay for.
Basically. His fundamental issue is that Internet services that lack an authority framework organically bred "ghetto" infrastructures, where things work worse because a person lacks access to "the right kind of people" in a system that's supposed to be democratized. He's not wrong, but it's not like someone sat down and decreed some email sources should be untrusted for fun; the system started out very implicitly trusting, we added a pile of humans to the system, and oops! They acted like humans, took advantage of other people, implicit trust broke down, and now there are "good" and "bad" online neighborhoods.
I'm curious how you define "democracy". Email delivery is very democratic, with each person having the power to choose with whom they associate and who they choose to delegate decisions to. Anyone who doesn't like Google's spam blocker can advertise an alternate address on a mail server that welcomes all submissions (or host a blog, or join a freely open IRC channel,...)
It doesn't really matter if he or the gmail user initiates the conversation, in my experience. Probably the only reliable way to communicate with gmail users is to send them 5xx response right away when they first contact you, and inform them there about the situation, and that they should check spam and why, tell them to resend, and let the other messages from the same gmail address through (maybe repeat this from time to time so they will not forget to use their e-mail properly and check spam folder). Let them have some of the fucking time waste their provider causes others.
Because it's really infuriating if someone writets to you, you put time to help them with something, and their demented (did I say it's infuriating? :D - also what kind of company can't detect something is a response, if you even quote the original email, it's ridiculous) e-mail provider trashes your response.
This is actually a great idea. We should implement this two-step approach for anyone trying to email us from Gmail.
Sadly, I'd imagine some users don't really have a choice, e.g., an employee whose company is using Gmail from G Suite for its email, so, it'd be an uphill battle for sure.
Maybe, one other thing where this would have to have a shortcircuited is when I initiate the contact, so that people responding would not get hit by this, even if their reply is the first e-mail from them sent to my server. So it should be fairly smart.
I can't really care less about those people. Most people I'm talking about are people who're contacting me because I already provided them quite a bit of free service via my free software.
So if someone can't be bothered to check so that I'm not wasting time responding to their trash folder, that's a good filter in the first place.
Anyway, better thing might be responding by adding a (F)AQ item publicly, and just sending a link.
> He made it clear that his opinion is that there should be a system in place at Google that he can petition for his reputation
The common thread among many of the bad-Google stories I've seen in the last 10 years is that Google has made a deliberate decision to not give two shits if they accidentally hurt a small minority of their users. People get messages out of the blue that essentially say "We've just wiped out your email account, we're not telling you why and there's no one you can appeal to." I don't know if this particular guy is on the level or not, but it's not credible that all of these people are secretly spammers and scammers. Occasionally you can get things fixed if you know someone at Google or your complaint goes viral, but that's your only option.
I'm sure someone at Google did some math and concluded that the ill will of a few customers costs less than maintaining a support department. That's not an excuse. I don't think we should expect Google to never make mistakes, but we should absolutely expect some avenue to correct those mistakes.
No idea how well it is or isn't correlated, but a note on twitter tagging the appropriate dept at google usually sees the issue fixed within two days, though no actual feedback and likely coincidence.
I would argue in this narrow case - I would not point to nefarious undercurrents.
Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.
If you can get into people's email inbox, there's still quite high conversion rate - better than clicks/videos on a pure percentage basis. Even the author wants to reach out to people in a cold fashion.
As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.
As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on.
I can't have it both ways. I would not be on Gmail if it was like yahoo. My goodness... my parent's inbox, on yahoo, makes me anxious.
> Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.
Google's spam models freaking suck donkey dick. I use Fastmail but my university uses Google for email. A service we use (Piazza, class Q&A) sends notifications via email. Over the years I have gotten probably about 1k email notifications from them, and replied to a few of them too. Google has recently started marking about 5% of their notifications as spam. 95% of the notifications come through; 5% get rejected. There's no discernible logic to this.
There are organizations and people within the university which have me on their mailing lists; with them too 90% of the emails get through but 5% get randomly marked as spam. And they're perfectly normal emails that Google just decides to move to spam. Even with people that I have had email conversations with, it just spams random stuff.
Every couple of months I check my Fastmail spam folder just for the lolz of seeing what's in there. With Google I have to check spam every day or I'll miss important stuff.
There are tons of people who have posted very similar experiences on HN. On this thread I just saw another person comment that Google's own emails go to spam - I've observed this too on my personal Gmail account which I occasionally check.
The spam filters have a tough job, but other companies are succeeding at keeping all the spam out without a gigantic false positive rate. Google's filter is hilariously bad.
I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.
I've seen this with my SaaS. We ONLY send transactional emails like purchase receipts and login links (using postmark). We're very intentional about not sending anything remotely resembling marketing or spam, and sending as little as possible. Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.
I've noticed that in the past year or so gmail has made hitting the spam button more explicit and now have a confirmation message. Obviously they did this because some people were simply mashing that spam button instead of the delete or archive button.
I don't even blame the users. Most don't know the impact it has. But bad actors sending tons of spam have trained them to be cynical of any generic form email.
> I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.
Google does put some senders often into spam for me, but not always. I'm absolutely sure I've never reported them as spam, on the contrary: I've marked their emails as "Not Spam" when I caught them in the spam folder.
That had, so far, no influence on the filter. Those aren't "weird" emails either, they are in my language and individual communication, not mass mails.
>Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it.
Maybe gmail shouldn't have the spam button right next to delete :)
At least that's better than web Outlook's invisible, until you click on the sender, microscopic disposition buttons. I can't count the number of times I've deleted an email instead of opening it.
It might be because the service sends both spam emails and legitimate emails. Many (maybe even most) sites do this. If this person is getting a thousand emails over the course of a few years, how on Earth are all of those legitimate, especially when they've only replied to a handful of them, implying the majority of emails can oe safely ignored.
Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.
I have a relatively high profile email address (first initial and last name) and I almost daily get at least one form of "receipt" from a transaction someone made where they entered my email by mistake.
Sometimes I'm in a bad mood and I'll mark anything I see like that as spam partly because its frustrating and partly because I suspect a good number of these emails actually are spam masquerading as receipt/login/etc emails.
And I just don't have the energy to figure out which are legit or not.
I wonder how much bad spam marking happens maliciously. Sign up lots of accounts to your competitor's mailing lists, mark everything they send as spam so that hopefully their email will not make it to their customers giving you more of a chance.
> I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.
And yet other companies with spam reporting features are able to maintain a lower false positive rate with their spam filters.
I'm not sure that other companies near the scale of Gmail have a lower false positive rate. Sure something niche like fastmail which is going to have, on average, much more tech-savy users, will have more accurate user-reported spam filters. Yahoo / Microsoft don't seem to do any better than Google here though, in my experience they're both actually worse.
I don't think it's always the user. The gmail and outlook UI makes report as spam more prominent then delete. In addition it's easy for accidental inclusion in a users batch spam report because of poor multi select UX.
What would be nice is a useful mail ML filter which would scan mail flagged spam to trigger a "are you sure this is spam, do you want to delete instead" question.
May be you should not send purchase receipt by default? I don't know why everyone does that as I don't care about it at all. If it's not required by law, add button to your UI which will mail receipt (or better just let user download it) and that's about it. While I'm not marking those receipts as spam, I'm sure that some users would do that as it's unwanted mail for them.
We actually have users contact our help desk asking for the receipt not realizing it's already in their inbox or spam folder.
I've considered adding an opt-out checkbox during checkout but I honestly doubt it would ever be unchecked and if it was opt-in we'd have users contacting support asking for it because they didn't realize they had to check the box. Either way it's one more toggle on the UI which I'm not a fan of unless absolutely necessary.
I'm curious if you've run a business and experimented with not sending receipts, and if so how it played out?
Yeah, gmail is really not great. I host my own email but use gmail for work. I hate everything in gmail, from the UI to the lack of really simple way to filter thing (like sieve) and to the totally random behaviour like you described.
In the end, I don't really read emails on gmail anymore because it's just a huge pain.
My own server has a good reputation, but it meant setting up SPF, DKIM, DMARC etc.... Where it was the most painful was with microsoft email (I don't know the name of their service now, still live maybe). By default mails goes to spam, but really quickly they goes in limbo if people don't respond (on the smtp side the email is accepted but the recipient never see it, not even in spam). On new microsoft accounts I was always able to send a few emails in spam before it went into limbo.
I ended up creating ~20 different accounts on their system (it was before they enforced giving a phone number) and had random discussions with myself for a while which was enough to build a reputation and not being put in spam anymore.
I don't know if this behaviour was intended or not, I try to get in touch with someone from microsoft but couldn't find a way. Since then I do strongly advise not to use their system because that makes it really unreliable.
I have exactly the same problem on my personal email server, ms silently drops my email! Their server replies with OK, but the email is nowhere to be found in the recipient’s inbox. Even if the conversation was initiated by the microsoft account and I merely reply to it, the email still disappears.
I’ve got everything setup properly, all test websites come up green, and google receives my email just fine. SPF, DMARK, DKIM, PTR, HELO are all setup correctly. This isn’t an isolated problem either, the ms forums are filled with people reporting similar problems.
One thing I've noticed too, make sure that your time is synchronized with an NTP server.
Even with a good reputation , MS drops emails with no error if you're a bit off.
"I have exactly the same problem on my personal email server, ms silently drops my email! Their server replies with OK, but the email is nowhere to be found in the recipient’s inbox."
Slightly tangential ... I heard recently that "proper" in-line responses (ie, not top-posting) is somehow penalized by Microsoft (or Exchange online or Outlook online or ... ?) and was advised to stop my very proper 1990s style in-line email quoting/responding and just top-post to avoid either non-delivery or something like the body of the email not being shown ...
Or something ?
Does anyone know what I am referring to and can shed some light here ?
Yeah, I only bottom-reply when I know (or can reasonably expect, e.g. on a mailing list) that the recipient is using a mail client that's not entirely brain damaged. Otherwise, I'll either top-reply or (if I really do want to respond to things in-line) actually say "See below for responses in-line" to clue in the recipient that yes, there's actually content in this email.
But yeah, that's usually more an issue with the mail client (or perhaps the mail server if it's being exceptionally stupid and modifying emails before final delivery). Top-replies became the norm for some god-forsaken reason, and now clients treat the quote and everything below it as entirely optional to display.
Same expirience here, but I must say it is becoming better since I have reverse resolve. Previously every mail went into spam. Regardless of SPF and DKIM.
I've run my own personal email for around 20 years at this point using the same IP address, and have set up all the bells and whistles (SPF, DKIM) over the years as they became in vogue. To my knowledge, I haven't had any issues with getting caught up in Google's or MS's spam filters, but I have to wonder if that's simply because I don't send a lot of messages and for the most part reply to emails rather than being the initiator of a conversation.
Absolutely agree with you. Many people on HN suffers from Stockholm syndrome when it comes to Google. If Google wanted to be a customer focused company instead if being an ad revenue focused one they could expose the knobs to you to fine tune the spam filter. They do not care about users, Gmail does not even have a customer support line that you can call when you ended up in the wrong side of their automation code. I am in the middle of moving out every email address from Google asap. I just realized how disgusting their way of operating is and I do not want to continue contribute to this problem by using their service.
It's curious that people have such diverging experiences with the spam filter. I think the claim that Google's filter is objectively worse should at least be supported by some evidence if you're going to make a bold statement like that.
As another data point, for me and friends I it beats competitors by a large margin.
As a normal gmail user you won't notice this, all you see is that you get mail from your friends that you always mailed with. If you register on popular sites, their mail will also work. If however you are sitting on a new domain, sending mail to an old friend on gmail, it will very likely end up in the spam folder unless you are using a big provider that has reputation with google already.
I guess it's another case of survival bias. If it works you won't notice that it does not work in all cases.
Even a lot of “popular” site stuff ends up in spam. I basically have to regularly check my spam box, which previously I would look at a few times a year when a password reset or something didn’t show up.
Anti-trust regulators should specifically look at situations where Google began directly competing with other companies and those other companies gmail messages are being put in spam while Google’s messages are not. Whether or not it is intentional, it is a problem if Google is whitelisting their own stuff but not competitors.
Ultimately what is going on here is Google is using more deep learning for spam detection. Your friend’s email ending up in spam may be a false positive. It may also be a test, to see if you notice it.
So I own a very short, easy to type domain, and maintain a wildcard. By volume, I probably get 60% spam to 40% content. I apologize to all the hard working vendors being fed into my spam list, but there's a secret: double opt in.
As a bonus, if you require opt in, I won't be able to cancel your car reservations with budget ROBERT.
Your comment prompted me to go through my GMail spam folder. I noticed 4 emails (out of 100) that weren't spam, including one from Blue Origin (!), 2 from a mailing list that's usually delivered to my Inbox, and 1 from an online shop (confirming my purchase).
Did the same thing on work email and found only messages from gitlab. But only 10% of those messages were marked as spam the others were in inbox so I did not notice. This convinced me not to trust gmail antispam.
I've noticed this too. I've stopped telling people that their emails are getting spammed because I've determined it's not their fault, that google just randomly spams emails coming from lists. It's the main reason I'm working on moving away from google back to my own mail server.
"As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on."
We[1] run our own mail and DNS servers and, as you might imagine, have never sent a single HTML email or any email with any kind of "element" in it, etc.
Just plain, ASCII text.
We've got a 15 year clean IP, perfect score on the "mail config checkers" and almost every single email we send is in response to an email from that sender.
And yet ... often enough to make me anxious, we get marked as spam.
We have the same issue. Online surveys being sent as invites to 100s of 1000s of people per day and although we are good at spotting SPAMMERs setting up free accounts, obviously people will always receive an invite they might consider SPAM and we get derated. Persnally, we find Yahoo is the worst for blocking.
It would be good to have a central registry of mail forwarders so that SPAM is not detected against registered IPs and then issues can be followed up directly with anyone who abuses the system. Do you really want to greylist Amazon SES because one of its users sends out spam?
Sending text-only is probably a weak spam signal as far as Gmail is concerned; for the majority of mail to the majority of recipients (though yours may be exceptions), HTML email - even if very basic - offers better layout/design options and better readability.
On top of that, of very old-school users on Mutt or Pine who prefer text over HTML, I'd think a significant portion of them don't have their mail pass through Gmail's filters.
But it's not about how good the Google's ML models are, you're missing the point! It's about Google being the huge many-headed monster with tentacles.
You can't use part of Google service without being entangled in their new protocols, the way they code and build things, run operating systems or servers, browse the web, and so on. You have to run your business the Google way now. The most scary part of this - the monster is completely deaf. Cry as much as you like, it's not going to listen. Even if you bring thousands of gifts to it every day, you can be erased from the web in an instant, and it won't even notice.
It's not too evil (yet!) but it has so much power it's scary.
Not so much "deaf" as "huge." They're supporting the needs of a billion users with sub-50,000 engineers. Anything that effects only 1 user can't pull priority from the needs of the many. Anything that effects only 10, 100, 1000 users, etc.
> As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.
Interesting how different things are for different users. Gmails spam protection isn't working at all for me. Confirmation emails from reputable sources get into spam, while really obvious spam messages get into my inbox. I mean.. really, really obvious messages. With "dfdf" a thousand times at the end just to make the message longer. Or these porn-bitcoin-scam messages.
I spoke to somebody the other day about highly targeted ads that come from snooping into too much private information. The lady was happy about it because it helps her find things that she likes faster.
It's just interesting how perspectives differ on this type of thing.
There is no way that Google would suddenly start to mark as spam continuations of threads that were perfectly fine between senders that have been in contact for years unless their spam filters were either seriously broken or they tweak them to cause more people to switch to gmail.
When you refer to specific threads going back and forth this may be. But here's my counter example.
I sent emails to my girlfriend's gmail address from my own domain hosted on a mail server run by a reputable company for years with no problem, whatsoever.
Then, on vacation in Thailand, my rather elaborate trip reports for which there's really no plausible explanation that they could be spam, suddenly bounced.
Sent from the same domain from which I sent mail for years and Google suddenly branded me a spammer and refused to deliver those mails even to her spam folder.
The auto-reply from Google is about as useless as you would expect. There are a few links, which are neither informative nor do they help to resolve the issue.
I worked around the problem by sending mails from my own Gmail account, but it left a really rotten feeling around the lines:
NICE COMMUNICATION YOU HAVE GOING THERE, WOULD BE A SHAME IF SOMETHING HAPPENS TO IT!
So I guess your assertion is wrong. Our exchange went on for a minimum of four years. Always from the same account and suddenly I was deemed an evil spammer.
It really sucks.
or they tweak them to cause more people to switch to gmail.
That, actually, is my suspicion. It's easy to brand that as "protecting their user base", but in reality make it so cumbersome that you're not able to send mail to a significant percentage of users over which Google controls their email unless you switch to Google for email.
Exactly. Look at it this way: an account with good reputation for years, suddenly showing up from Thailand instead, is exactly what a spammer hijacking the account would look like.
Regarding a specific thread going back and forth (span of a month or so, not years):
Google marked as spam my 7th or 8th reply in a chain of emails with a prospective tenant (presumably) because my email contained instructions for sending me money. I assumed that they weren't interested anymore and had ghosted, but luckily we crossed paths at a grocery store or some such.
My emails continued to be marked as spam after they added me as a contact, individually fished each one from their spam folder, and continued to write replies. Weeks later (with no apparent cause) I started arriving in their inbox again.
> As a gmail user, I 100% appreciate keeping my inbox extremely clean.
That's a silly way to spin this. They can keep your inbox super clean by just not delivering any mail at all.
The way GMail is going, you'll soon only be able to get Google approved corporate spam using their email service. It took them longer, but they're pulling the same shit they did with XMPP. Open and standards compliant to begin with, and slowly converted to a bastardized Google only walled garden.
Maybe now's the time to fix it. If people don't get an email, suggest they use a better service.
> As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.
Not the case for me. I'm okay with some spam. It's not hard for me to delete few mails every day. But I'm extremely not okay with legitimate mail marked as spam. And gmail keep doing that. And that means that I must check spam daily. So basically gmail spam filtering does not exist for me, because I still must read at least all subjects. And that's dozens of mails every day.
This is exactly the viewpoint I have as well. I have missed some very important emails before, such as instructions for an interview, acceptance emails, etc... They don't even show up in Gmail's search system, because it doesn't search through spam. This has forced me to look through all of the spam I get daily, examining each one to make sure I'm not missing something important. Gmail may as well not filter spam at this point.
> I can't have it both ways. I would not be on Gmail if it was like yahoo.
I still use Yahoo e-mail for chit-chat (I doubt that it's less secure). Recently it started spam-boxing e-mails from a person I've corresponded with for years (without trouble). Since I have them forwarded to T-Bird, it does mean I have to visit Yahoo more often.
I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days.
The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.
I did a quick scan of rafa.eu.org and found the following:
- SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.
- A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.
- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).
- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this.
So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.
No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail.
And if you read the rest of that mailop thread, you'll see that the broader IP reputation of the /24 where your IP is located is a key part of whether or not your email gets delivered.
I would venture to guess that a substantial portion of people reporting issues delivering at small scale are sending from a low-reputation netblock. (Assuming auth is all set up optimally.) So it's not that it's not possible to send at low volume, it's just that most people don't know to look at the netblock, it's hard to evaluate netblock reputation, and it's not always possible to select a preferred netblock within a given provider (especially VPS providers with low-cost options).
The bottom line is: it's hard to run your own mail server, and much of the reason for the success of services like Sendgrid is a) it's easier to set up auth properly with most of them, and b) they actively manage the email reputation of their netblocks (though some do this better than others).
If you're approaching things from a practical perspective instead of making this your hill to die on, you just use an email service provider, and for the vast majority of senders, life is a little bit easier. There's plenty of ESPs to pick from, and there's a very good chance you can find a free one that delivers better than an average VPS or IP of a residential ISP.
My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.
Doing all of that from a popular low-cost hosting provider like Hetzner is still insufficient to be able to send emails to new contacts. You still end up in spam.
Same here. I also installed https://mailinabox.email on a server and can send email to gmail recipients without trouble. I had some trouble with one of my domains a couple of years ago but it was only related to misconfiguration on my side.
How are you verifying this? I encourage you to try sending to a gmail address you have never sent to before, I virtually guarantee you are going to end up spamboxed.
I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time.
Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.
I have verified it by helping people sign up for the small service I run and seeing the e-mails pop in their primary inbox. I guess I might be lucky then shrug.
If you would have actually read the link in the post you're replying to, you would know that Gmail is not accepting mail from me despite that I've spent a huge amount of time in verifying that everything is configured correctly.
Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.
There's plenty of low effort ways you could provide proof while maintaining relative anonymity (if you actually wanted to support your claims with evidence). For example, you can run a GlockApps test (5 minutes, free) and then take a screenshot of the results with identifying information blanked out.
Because OP talked about Google not allowing small players to send e-mail here's my server's deliverability results to Google: https://i.imgur.com/sceZogI.png
If you're wondering about the blacklist number, that's an error - the source site claims it knows nothing.
> Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.
I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim -- I can understand that they often block legitimate mail like in your case, but that doesn't mean that everyone has the same issue. I have occasional issues sending to Gmail but most of the time they accept it and don't mark it spam.
>I'm sure most people who run low-volume mail servers here have no problems sending to Gmail.
Based on direct experience and >testing< (I have talked to about 20 people with personal mail servers about this, and got them to attempt to send me email, on an account that had never interacted with their domain before), 100% of their emails were spam boxed.
Lots of people test gmail with their personal accounts, and think that email to gmail in general is delivered.
In my experience, it basically never is if you are a small sender.
> I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim
Where are all these magical people with these magical abilities? How curious that not a single one of them is able to post any kind of evidence to support these incredulous claims. And funny how my comments on HN are getting downvoted and hidden when I ask for proof.
Sounds like there's a niche business and/or OSS project there. Create gmail accounts. Install email send bot on email server. Send messages to Gmail accounts. Figure out how to avoid spam.
Its inaccurate to say that SPF, DMARC and DKIM setup this way is a "misconfiguration", It still allows anybody you are mailing to verify that email is from your mail server, and your domain, which is good enough for anti-spam purposes.
Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.
In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam.
> Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.
Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.
An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc.
> The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough.
A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does.
The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here.
> I did a quick scan of rafa.eu.org and found the following:
All the issues you list are non-issues, and merely the perpetuation of the security theatre.
SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either.
Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected.
Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.
1. Our ML models see that your emails spam? You're spam.
2. You're using Tor or a VPN? Here's 10 reCaptcha's.
3. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.
4. Youtube video includes education hacking tutorials? Demonetized.
Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
>2. You're using Tor or a VPN? Here's 10 reCaptcha's.
I get recapthca ALL the time for just not being logged on and deleting all filthy cookies on exit; and not "honoring" 3rd party cookies at all... and I don't care. Best part is that if you live in a country you don't speak the language you get google services in one of the country official languages -- so
what you should do and mark is often unclear. The next best part is a road trip through Europe (edit adding "?hl=en" to the url tends to fix the issue in most of google services)
>4. Youtube video includes education hacking tutorials? Demonetized.
History is a far greater offender.
>1. Our ML models see that your emails spam? You're spam.
Years ago, I recall my company hosted gmail started detecting CEO mails as spam... and he was pissed for not getting responses.
>Your browser sends language preferences, doesn't it?
Since http1.0[0]... I have wanted to add another header -> "RespectAlHeaders:All-Custom-CrRafted", so the wimpy implementation of ip->country->language is duly removed.
I was getting really long and annoying recaptcha all the time, presumably because I use Safari which has all that anti-tracking stuff. But lately (past 2-3 months?) I only get a single challenge, I wonder if they've found a way around the protection or if they've just changed the model
OMG, me too, and all I did was to tighten up cookies (disable 3rd party trackers, erase all cookies on close, etc).
Now I've got a PhD on solving "traffic light" and "crosswalk" Captchas as a result.
I started to resent Google for this, except now I think twice before going to a web site that will give me these hurdles, so it ended up helping me to focus on my work. Now I appreciate the extra work.
"Moving to Fastmail" is such a popular alternative that if they ever get to any kind of reasonable size we'll be in the same position all over again.
That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.
I have mostly moved to self hosting email. I still have a lot of legacy registrations with various companies that go to gmail, otherwise everything else is self host on a hetzner vps (2.50 eur per month) which doubles as a wireguard vpn.
First time setting up an email server was quite a chore as I have no background in email, getting all of the moving parts to play nice required some patience.
My second email server used Mail-in-a-box scripts which I highly recommend. Zero to functioning email server in a few minutes. The slowest part is DNS propagation. Anyone comfortable setting up a domain name and the basic DNS settings that are invloved will have no problem self hosting email. The setup includes web-based client. I use k-9 mail client on my phone.
I find the catch-all email feature very helpful in creating ad-hoc addresses for spammy companies (eg car insurance). I can sign-up with spammy.companyname@mydomain.com and it will direct to spammy@mydomain.com.
I'm not completely de-googled, but I am consciously and gradually reducing my exposure to google, and other large networks.
I've been using Fastmail for about 5 or 6 years now. My only problem is the 1GB mailbox size limit I have. Maybe I should upgrade. Otherwise I love it. Their webmail client is phenomenal and works with mobile and desktop. I don't like their Android app but I haven't tried it in a year. Their web client is so good I don't need an app for them.
I do see mail going to spam though. It's never bugged me that much.
You can also just use the native imap client provided by your operating system. The beauty of using a provider and service that adheres to open email standards.
We're still a fair way away from that! We're growing steadily, but not at the same rate that "give it away free and work out how to monetize it later" services do. Worth it for having a sound business model and no venture capital breathing down our necks to turn evil though.
Thanks! I was looking for an export button for them earlier today so when I saw you comment I couldn't help myself.
A zip file with all notes in it would be fantastic. I just want to feel like my data is safe from Apple having a bug that wipes it. Read a lot of Catalina horror stories.
IMAP looks promising and might be all I need. Neither Spark or Apple Mail seem to let me export mail as txt but presumably there's a mail client out there that does in which case problem solved. Thank you for point that feature out, should throw it up on:
It's not the ideal solution and certainly not as easy as clicking a button but, for a quick 'n dirty solution, you might look into using "offlineimap" and configuring it to sync only your notes folder instead of all of them.
I use Fastmail scared that my emails will go to spam. But ideally I want to live in a world where my emails hosted on my email server will send like they did 10-20 years ago.
I want to use Fastmail, but their minimum price for using your own domain is $50/year. At that price I can get Exchange Online with double the storage. And given that they have an office in the US, I would expect them to follow the same laws that Microsoft and Google have to regarding requests from government agencies.
This looks to be a good alternative except it does not have HIPAA support.
Even though I have shut down my practice for the near future, I will still need to have information protected for several years to the Health Department's requirements for psychological records.
Email providers who lack this support will have difficulty gaining traction in the health world.
Could you sue Google for slander in a case like this? They are explicitly labeling your messages as being manipulative or dishonest without any just cause. Why does it matter it’s an algorithm instead of a careless person in power?
I have all my mailboxes with gandi.net. Each domain you buy from them comes with 2 fully managed emails, and I think 3GB storage each, for just over $1/month. I basically never get spam, nor blocked emails. They have 2 webmail interfaces too, but I don't use them. As a bonus, gandi financially supports a ton of open source projects, which never hurts.
I use https://mailbox.org/en/, it's from a small German company and I can use my own domain with the smallest plan for just 1 € per month. Their spam filter also seems pretty good.
> Few bucks a month add up and can be considered expensive by many.
I pay around 3$/month for Fastmail, I don't think it's expensive, an email address is the primary communication method everyone uses nowadays and it's useful every day.
> but we need Google to stop this "my way or you don't exist" attitude
I think that's exactly opposite. We need more competition, not forcing a private company to behave in some way that "we" (and who is "we"?) find acceptable.
The problem is that Google is not just Google the search engine, or Google the email provider, or Google the video service, or Google the website analytics engine, or Google the online office suite, or Google the ads syndicate.
Because Google is all of those things, it has accumulated too much market power, to the extent that they are market distorting. Bing and DDG (which is not an independent engine, but collates results) are irrelevant to a company's Internet presence. There is no such thing as SEO for Bing. Whereas Google can make-or-break you.
When Google the ad syndicate and Google the search engine collude (as it were) to place your competitor's ads ahead of your organic search result, a top result even, and to distinguish such ad results as ads only technically, by the narrowest of definitions, this compels you to advertise with Google the ad syndicate. Just as an example.
This behavior is harmful to the market (not to mention to users, but that's a different subject) and it is a given that Google the monopoly^Wsingle entity will continue to make such moves, whack-a-mole style so it's futile to squash any individual behavior even if you could. Ergo the reasonable solution is to break them up.
Have you actually had a lot of problems sending e-mail to Gmail users from your mail server? For me Gmail has not been great about accepting e-mail and not marking it spam (occasionally have issues), but others like icloud.com have been worse.
>Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
I mean, I get captcha's all the time while working from Google's network. You're exhausting from a shared quota. That's life on the modern internet.
Bitch about Google all you want, it's not like they decided this is how the internet should work. Prior to Google, my company integrated with reCaptcha because no other choice existed to reduce automated abuse.
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things.
It seems you're not getting the point. The problem is not for users of those "awesome" Google services the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account.
Google tags those as spam and refuses to deliver them, while always shifting the goal posts and providing sub par support (an number of essentially useless links) to the sender of the email to a Google account.
This happens, btw, also when the sender account never spammed and sent legitimate mails to the Gmail user for years.
So your snarky comment is completely out of line here.
It's not about people mooching free "awesome" Google services for free it's that Google is a rotten net citizen and is behaving like shit to the entire world.
> the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account
It's funny to me that people in this thread are complaining about Google being large enough to serve as a de-facto authority, when the only way one could solve a "legitimate emails are being blocked" problem is to have a definition of 'legitimate' that would come from... an authority.
Google is dropping the ball but I don't know what the solution would look like that isn't just "Someone else be Google now." ;)
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things. It’s so demoralizing. Get off your high horse.
I don't use gmail these days (except as a honeypot out of laziness), but I get plenty of email from google hosted domains. The issue isn't (just) that people want to use these services and want Google to have more reasonable policies, there is also the issue of having to tolerate Google's policies because other people use their services.
Emails from Google servers are notable to me because there is no way to report spam (and the signal to noise ratio from Google is pretty low for my case). Google's opaque approach to email means that deliverability issues become even more tedious to deal with than they should be.
It’s been a while since google made anything awesome hasn’t it? Maps is still great, but things like AMPs are so terrible that they were a bigger part in driving me to DDG than privacy.
Maybe I’m weird, but I’ve never been in an AMP where I wouldn’t rather go to the actual side and use reader mode.
I'm actually having a similar problem recently, although my messages are declined at the SMTP stage, "due to the very low reputation of the sending domain", which I attribute to the output of certain cron jobs being sent towards my own Gmail account — it seems like my whole primary domain name has subsequently been blacklisted a few weeks ago:
It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.
Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.
I have a hotmail throwaway account (it has since been migrated to outlook with the microsoft buyout) and its spam filter is completely backward. There is currently 373 e-mails in my inbox (100% spam, expected since it is a throwaway...) and 4 e-mails in my spam folder--100% of them the account registration/confirmation e-mails that I was wanted.
I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.
In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...
Honestly not sure, but certainly sometime in the 20th century. It was my main e-mail back then, I migrated to Yahoo! shortly after the Microsoft buyout and kept the HoTMaiL as a throwaway. When Google announced Gmail and the 1GB (with projection to infinite) storage I switched to them as soon as I got an invite, but when they changed their slogan from "Don't be evil" to "Do the right thing" with the Alphabet restructure I ditched it and moved to my own domain with Fastmail. If Fastmail starts to show signs of evil it will be as easy as updating my domain records to point toward OurMailIsActuallyGoodIPromiseThisTime.com and click through the certificate warnings on my mail clients.
That Hotmail branding survived long after the purchase; in the last few years it's the same interface as their 365 business offering now, and under outlook.com branding.
When I last had to wrangle with MS (c.5 years ago, who were dropping all our email _replies_ to customers, they had a third party that you could pay to ensure your emails got through ... apparently the problem was our domain was on an IP hosted by a company who'd previously hosted (but didn't now) a company on a different IP and that other company has sent some spam.
I sometimes wonder if certain companies can get white listed.
There's a particular perfume company that sends spam to my gmail account every month. I've been marking it spam for at least five years, but somehow it always gets through.
Remarkably, Fastmail has gotten to the point where its spam filter works better than Google's for the kind of messages I get.
The interesting thing about whitelisting is that you have to be sending a lot of emails so the company determine if you are sending spam or not. So if you have a private domain, your own mail server, and are sending only a small number of emails then your not going to get whitelisted. Sometimes you can get on a feedback loop but all that will tell you is if someone clicked on "this is spam".
In testing different mail servers I find that Gmail will treat them differently based upon there hello line. I can't remember which one was accepted most but there is a difference. Also you need to look into if others on your netblock are sending spam. If you are on the same netblock as a spammer then gmail will treat you as if you are the spammer.
The biggest problem to me with Gmails spam filter is that it is so totally hit and miss, and there seems to be no reliable way of ensuring that messages stop going to spam, even from other Gmail accounts, from people on my contact list, that I've persistently marked not spam.
At this point the only reason I'm still using Gmail is sheer inertia, and it's a question of when I get time to start untangling myself from it.
Yes. I've also used Google Apps and had it file e-mails from our own users as spam despite settings that's supposed to bypass it.
Meanwhile some Gmail addresses I have rarely have this problem, so it's not even consistent across accounts or domains.
The most annoying bit (and probably the first untangling from Gmail I'll be doing) is that I have my own domain hosted on Google Apps, but since my main Gmail mailbox is a different one, it's set to forward everything. It's meant to bypass all filtering and just forward. Or so the settings and support claims. It doesn't. I have to regularly log in for the sole purpose of marking things not spam.
Not sure about OP but to me rules in GMail don't work.
Gmail filters don't seem to work anymore. I had filters and labels and it was great but I noticed they seems to be failing.
If you go down the rabbit hole of Google Gmail support you'll find filters don't apply to email ("conversations") already in your inbox. But they also don't seem to apply to new email.
Asking on various groups people seem to be giving instructions for options in GMail that no longer exist.
I've tried that without success. Delete all rules/filters, labels and re-create it all again. Nothing will work on old or new messages. And "apply to zero conversations" is the only thing I see that seems like it would be it but the zero part means no chance of working.
Gmail even replied to me on Twitter saying the filters no longer work.
It doesn't seem to work. In fact, IIRC, they simply add a huge button dialogue to any such emails that pass due to a filter to mark it as spam, without any buttons to dismiss their improper suggestion — e.g., you can't even confirm that the mail isn't spam anymore if you use such filters.
If the mailing list is using a bulk email service, then they should use DomainKeys Identified Mail (DKIM) and Sender Policy Framework (SPF) for identifying ownership & not getting marked as spam i.e. If they aren't really a spam.
Sorry, but your advice have been detached from reality for a long time.
SPF is not a "should" anymore, it is a must. Many major e-mail providers will outright reject your e-mail at SMTP level, unless you have correct SPF record. Rest will move SPF-less e-mail to spam on automatic basis. In this sense SPF does help you to get delivered, and avoid getting into spam bin, but your chances can only go from "100% spam" to "90% likely spam", not "0% likehood of being spam".
DKIM is not a major consideration for Gmail (or anyone, as far as I know) and have never been one. You will not magically avoid spam bin by using DKIM. Instead DKIM provides a great opportunity to botch your reputation with mail servers: as soon as you forward a e-mail (between providers or within the same provider) it will lose it's DKIM validity, and thus may get classified as spam (!!) on the basis of invalid DKIM signature. Boom! you are a spammer now.
Valid SSL cert and valid SPF are the only things, that have noticeable impact on Gmail's treatment of your email. DKIM and DMARK and largely useless and can only get you in extra trouble.
As for bulk email, it is going to be classified as spam until your domain and IP gain sufficient reputation. The only way to gain that reputation is to send slowly increasing amount of email for long time without being marked as spam. Being a rich business, Google's client, hanging on Google's forums, using Gmail as your mail server and having at least dozen contacts among Google's employees also helps.
This same thing started a few weeks ago for me as well. I have a cron job that parses my log file and sends me usage statistics every morning. I had been receiving this email for 100+ days when suddenly it stopped. I investigated thinking it was a problem in my script only to find that all worked well and all emails were in my spam.
I use SES from AWS and now my new user sign ups emails for email verification also end up in spam. It’s very frustrating because they changed some rule and now my low volume email sending is considered spam
I thought this had been getting better. At least more e-mail from my personal server has been getting through to its destinations. I've written about this too a few years back:
..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).
I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.
My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).
Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.
> but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
This is why I like that my email provider (https://mailbox.org/en/) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.
so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.
depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)
if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.
the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.
I have no idea if it actually is, but it might be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.
Yes, but that has it's own challenges with spam protection, privacy laws, users not spelling their email correctly etc, and you still need an email address to receive those messages (or check for new form entries regularly), so an email address is often used as a shortcut.
That said, if you read far down enough in the thread, it's pretty clear that the OP has very, very unrealistic expectations about others' obligations to him, and very incorrect notions about how statistics work. The argument eventually turns to "well google has a lot of processing power at its disposal so I don't see why it can't use this in order to turn incomplete data into more complete data" and that's when I checked out :\
Edit: also this Brandon Long has the patience of a saint.
The attitude of the replies he gets lower down in the thread is shocking. I hope whatever mailing list that is doesn't have any kind of good reputation, because it seems like the consensus opinion there is shockingly anti-cooperative, anti-internet, "me before the network", even going as far to say "if Google thinks you're a spammer then you probably are".
It’s posted elsewhere in this comment thread but that’s after the point where it becomes clear that he wrapped a moral argument in a technical envelope and people spent a lot of time trying to help him out with the technical argument.
Put another way, play stupid games, win stupid prizes.
That's a meeting of technical and moral view. I'd love for corps to have no impact on people, for distributed knowledge and resources, for everyone being in control of their data, and general socialist direction.
But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.
> Posting about how unfair that is is not going to change anything.
It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:
"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - and that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."
Might be a good time to mention a new development with Yahoo Mail. They now appear to use a form of shadow ban.
They will accept an email, with all outward symptoms being normal, but it will not appear anywhere in the user's mailbox. Not even in the Spam folder.
This started few months ago when we were sending out our (rather infrequent) newsletter. 5th or so message to ...@yahoo.com address failed with
421 4.7.0 [TSS04] Messages from xx.xx.xx.xx temporarily
deferred due to user complaints - 4.16.55.1; see
https://help.yahoo.com/kb/postmaster/SLN3434.html
(in reply to MAIL FROM command)
That's with just a couple of seconds in. They kept replying with 421 for a while, but later switched to just dropping connections. Then, in the course of the next few days, they would accept 1-2 emails and then go back to denying them again.
As I mentioned above, the "best" part is that none of these emails would actually make it into recipient's mailboxes. They disappear.
Better yet, all subsequent (non-bulk) emails have the same fate as do _replies_ to emails from Ymail users. This last bit is what makes this look like a shadow ban.
All attempts to contact Yahoo went unanswered, including those made through Gmail. Ditto for whitelisting requests through their web form.
If anyone has any ideas what the hell's going on, I'd be curious to hear them.
"Many providers" - any concrete examples? For an SPF/DKIM'd server that's been at the same IP for nearly a decade and had not a single spamming incident.
PS. The list is immaculately maintained, the server is configured as properly as it gets and very conservatively rate-throttled. There's nothing obvious.
This is the consequence of the spam filters doing a good job. I too remember when consumers would clamour for better spam detections and different companies would hawk products that purported to do so.
But now that spam detection mostly works well for most people, it’s a forgotten issue. Now we have people on this thread questioning Google’s authority to filter spam because from their PoV, Google isn’t adding much value to begin with.
The problem is that Google errs too much on the it-is-spam side. Having to delete a few spam mails isn't a big issue if the alternative is losing legitimate mails because of overzealous filters.
Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.
Sandstorm should have really caught on. We need to make self hosting drop dead simple so that even non-technical people can do it. The power of these mega-tech companies can only be whittled down if we take collective, well thought out action. And it is not just about tackling power imbalances. Owning your own data is a fundamental aspect of our digital lives which ought to have become simple and natural, given Moore's Law and the phenomenal device capabilities in recent years, yet somehow we are going the other way. Even as our devices become ever more capable (hardware wise), the software is moving to ever more to the Cloud and we're being locked out of the full capacities of our devices except in the narrow realm of media consumption.
I actually check Spam daily now, after finding support emails whose logs got snagged in the dragnet & personal stuff — even email from my Dad got shunted to Spam.
I still tip the hat, though, to Google, whose algorithms are wise enough to assiduously stow both the “jump on a call” and “donate to my campaign” emails to spam.
The vast majority of messages in my spam folder are not spam on Gmail at the moment. Most of them are from providers I've semi-regularly tried to get it to recognize as not spam for months or years. Of course Gmail also outright rejects a lot of stuff at the SMTP stage, so their filters are better than they look, but the amount of misclassification is extremely frustrating.
I have this issue with OSSEC messages (automated messages from an Open Source IDS). Same thing, too, have been marking them as “not spam” for years. I think it’s because I don’t always review them...:/ But just because you don’t always open a message class doesn’t mean it’s spam, Google!
At one place we even had them start bulk-rejecting messages for our Google Apps domain from a whitelisted source.
The entire point of the whitelisting was to allow our internal mail server to forward transactional messages to addresses on our own domain, so it was not even that it was ever reported as spam by anyone, or ever hitting external addresses where I'd understand if Google was more trigger happy.
It was ridiculous realising that they basically had no way for even paying customers to tell them that "that e-mail server we told you is ours and we always want to receive all e-mail from no matter how spammy you think it is, using the mechanism you say it there to do exactly that, well, we really mean it". Instead it'd randomly just stop working. [this is a few years ago; who knows, maybe they've fixed it]
If you're really curious setup DMARC as well and check out the reports. I used to work for a company that did DMARC analytics, and after setting it up for my personal domain it's kind of interesting to see who's trying to impersonate you (or who's getting spam purportedly from you).
None of the big providers send DMARC messages. Google doesn't. Microsoft doesn't. The only DMARC messages I've ever gotten were from Yahoo and mail.ru.
Google/Microsoft don't send DMARC reports because it would make it easier for spammers to know they had gotten past those filters. It's easier to just drop everything they don't like silently. It's why e-mail is terribly unreliable today (much more reliable if you DO NOT use Microsoft or Google).
I regularily receive dmarc reports from Google (every day) and yahoo for instance. Not sure I've ever received any dmarc report from hotmail though.
And email is pretty reliable with Google. It is a disaster with hotmail: you have to send a big number of emails to have a chance that hotmail treats your email as non spam. They put your self hosted emails in the spam folder of your regular contacts, and the fact that they respond to your emails and systematically mark them as non spam is an irrelevant detail for them.
Google/Microsoft don't send DMARC reports because it would make it easier for spammers to know they had gotten past those filters.
Is that a recent thing? I was getting DMARC reports from both Google and Yahoo as late as April this year. Around then I pared down my Sonic account and they ate a bunch of stuff that I was too lazy to setup again including DMARC. While Google and Yahoo make up most of the reports I got this year, I also see some from Microsoft and AOL from 2017. 163 (China) used to send out reports too.
Aggregate or forensic? I’m not sure I’ve ever seen a forensic report.
Both. In looking at my mailbox I see some forensic reports from 163 and aggregate reports from everyone I mentioned plus Comcast, Emailsrvr, XS4ALL, and QQ.
I don't remember if I eventually turned off forensic reporting, but I do remember that LinkedIn and Facebook were particularly strong proponents of DMARC (and both were really good about accidentally load testing our infrastructure). Oh and one very large provider that actually gave us unredacted messages.
Email is not the same it has been for years/decades. These days you need to use all the security tools available, most notably TLS and DKIM, set a strict DMARC policy, and don't do anything unexpected such as sending lots of emails out of the blue.
Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
From the comments here I get the impression that this means, “don't do anything Google's machine learning environment cannot interpret”, which has the added drawback of not ever knowing if you are doing something it considers unexpected.
> Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
Is it though? That may well be true for small individuals, but the Spam I get these days comes via the large mail providers, Amazon SES, sendgrid etc, and nobody in their right mind is going to block them. "Too big to block" is a thing now, "too big to have to care about abuse reports" is too.
Yes, the big mail providers are an issue. But it's also an issue the other way around. We use Postmark at my place of work and unfortunately, that doesn't prevent all of our mail from being marked as spam. Some mail gets blocked because it comes from the same IP address as some spam mail. I know that Postmark puts effort in preventing spammers to use their service but Amazon probably doesn't care a lot about people misusing SES.
Wouldn't that imply that they have a lot of spammers on their network? As far as I understand, the value proposition is both "you don't need to worry about email" and "we have so many customers (sending legitimate emails that are very similar in content) and so little spam, we have great reputation and you're inheriting that reputation".
If that doesn't work out, is their reputation not good enough? Is the receiving server too strict? Are those blocks occurring on major providers or for individual servers where an annoyed person might just blacklist an IP if they get spam from them more than once?
Note that "a lot of spammers" isn't really a lot in laymen terms. It varies by provider, but last I worked on this if anywhere from 2-5% of all emails you send are clicked as spam the provider will start blocking you.
We got blocked once because we got a single spam click: the ISP said they have a zero tolerance policy. We had years of sending hundreds of email per week without a spam click.
> Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
In my experience, the email blacklists and filters are somewhat forgiving. I accidentally configured my email server as an open relay (didn't know what that was at the time), and for a brief period was sending a not insignificant percentage of spam worldwide.
Once I discovered my mistake and corrected the issue, I was able to submit domain removal requests to the various blacklists, or just wait for my domain to fall off the radar. I expected it to be worse, and was pleasantly surprised to find that this kind of "forgiveness" is really an industry standard.
But then, that was just one bad move. I'm not so sure about two bad moves, and I'd rather not find out.
> A quick look at Talos Intelligence reports a 1600% increase in email from your domain over the last day; email reputation os 'neutral' at this time. Not good, not bad, but neutral.
> Also you're on OVH, about which a quick look through the list's archives will
possibly prove instructive. It's reasonably likely (as likely as not) that
you're running on an IP in a neighbourhood with some poor neighbours.
> As Mathieu pointed out in his second email, building a good reputation takes time. Losing a good reputation is a matter of rather less time and can be influenced by factors outside of your control.
The rest of that reply chain is the usual debate:
OP. My domain's emails are marked as spam by Gmail
No, it's not their network. It's our common network. If anybody imposes own rules on their part of the network, we are losing interoperability. Internet loses any sense.
He apparently thinks that he (or maybe more accurately "we") should dictate how private companies treat traffic. Despite it's huge footprint, Google does not have a monopoly on email, if they are treating users unfairly, they can switch -- but I use Gmail because their spam filtering is so good. And I'm ok with dropping a few legit emails if it means not seeing spam in my mailbox.
From my old law courses (IANAL but someone decided programmers should have a basic knowledge of law):
* An illegal monopoly should not be absolute. Having >20% of the market has been enough te be considered a monopoliy for antitrust. The main question is if your actions dominate the market.
* But having a monopoly is not enough. You can have 100% of the market and be legal. Abusing the monopoly is the problem. Abuse is typically crushing smaller competitors, or using your monopoly in 1 market to push yourself to another market.
* Is google abusing its potential email monopoly? My non-lawyer guess is no. Being non-responsive and arrogant is not illegal.
* But the google browser and search engine might very well be other stories. Here google seems dominant enough to influence web standards and change other corporations behaviour to raise in their search index.
* I don't think US or EU governement are willing to step in, though.
This finally get me to the point of understanding the "bad neighbor" signal that probably get my personal email (and some of my jobs) rejected from Gmail.
Reading the thread was very instructive, but the most irritating point is that this rejection is mostly silent even if you send only 20 mail a week from your email address and all DKIM & all are in place. Lots of pedantic comment in this thread however, it seems that nasty provider are well known by experts, but for the love of internet, please spread your mighty feedback to small-folk so that we can act accordingly.
Gonna open a ticket at my provider OVH (now I know).
Ok, so the way OVH can solve the problem is the same way e.g. Amazon SES solves the problem.
1. They block all outgoing mail (outgoing 25, 465 and 587) and only open it up once you have submitted a ticket where you decribe for what purpose and how often you want to send mail (if you start off wanting to send a million mails per week they won't open outgoing mail at all). Manually review of those tickets is required so that bad actors can't get through. Difficulty of getting through the review needs to be adjusted upwards till it is too much work for bad actors (and they switch to another provider).
2. Monitor bounce-backs and have a spam reporting form for OVH IP ranges. Start blocking outgoing mail pretty soon if a small number of those arrive. Of course this is a cut-throat business ("email marketing"/spam), so competitors may submit false spam reports...
3. Therefore, for (2) you'll have to keep some records of the mail sent, i.e. you'll require mail to be send via relay... The review and the mail relay costs something, so you make it a monthly/per mail cost. Voilà, you have Amazon SES.
Currently they probably block individual instances if they get a lot of complaints about one sending a lot of spam. They can't distinguish legitimate mail from spam because it is all SSL encrypted.
Until now I trusted them so I use their own email servers. They are perfectly aware of my usage since while sending they even implement a trick so that one can't send bulk email from their SMPTP.
And that's why I now fear they are utterly incompetent...
I feel like there is no way to do this. I was once very happy when I finally got my own basement email server to work (I tried it off and on for years before often giving up), running on Arch. For some time I was the happiest person in the world, I could send emails of >100 mb to my wife, have unlimited storage, unlimited aliases (which can even go to multiple people). I could email my server and make it do things. I was using TLS (whereas me.com and hotmail.com at the time were not!), I was using the internet the way it was intended!
But then Outlook.com stopped accepting my emails, they were not even in any spam folders. Several emails later I got a response from someone at MS: We don't know how the filters work, they are black boxes to us. Sorry, try to write emails that don't look like spam...
Soon after I gave up on my server.
Maybe you can use a third party (paid) smtp forwarding service?
"It is always better to let the possible spam go through than to filter out something that is not spam."
Spoken like an email sender, not an email receiver.
I'm deeply sympathetic with the sender, and not inclined to defend Google here, but I'm also aware that Google is a very large target with a lot of bad actors specifically targeting them, and I just checked my spam folder in gmail: wow!
The problem here is spammers, and a communication system that assumes good faith on the part of senders. Gmail is choosing one way of trying to counteract that problem which is causing problems for people in certain IP blocks, and that sucks. But the problem is still spammers, not Google.
This guy has an odd (seemingly nonsensical) quote in his email signature.
To a machine learning algorithm, it might look kind of like the random nonsense some spammers put in spam to corrupt Bayesian filters. He might want to try temporarily taking out that signature if he’s using that as part of his emails.
good catch. In my limited experience, complicated signatures (e.g. hotlinked images), or emails with few words and several links are most likely by gmail to be filtered as spam.
I get like 50 emails a day entirely made up of hotlinked images from companies trying to sell me shit but god forbid someone have one in their signature. Thanks google.
I also have my own mail server, and I have the same problem.
I'm guessing this is machine learning algorithms confusing correlation with causation once again. Being a small private mail server is strongly correlated with being a spammer. So even if you are not relying any spam, you certainly look like someone that would. And even if you have not done it yet, you will probably in the near future, so they might as well treat you as a spammer right away to avoid potential problems. (analogy with racism intended).
That's most likely the reason: OVH is well known for their lack of attention on spammers using their VPSs, so most operators will just filter their IP blocks.
I run a low volume email server and 100% of the connections from OVH blocks are spam/phishing/vuln-scanning/etc. It's true that Gmail could do more to help low volume operators understand deliverability problems, but in my opinion OVH should be doing something about their spam issue.
My advice when looking for a VPS provider for running an email server is to read their AUP and check their blocks on Talos.
Subbing to this, I’m having this exact issue. I’ve got dkim, dmarc, spf, the TLS flay and rdns. No other tested mail server blocks me, just gmail.
I’ve googled and cannot find a legitimate way to determine why I’m marked as spam or how to unmark it. I’m coming from a fresh domain and have no intention of ever spamming.
Googles support, or lack thereof, will hopefully be their downfall.
Because of this post I checked my gmail spam folder for the first time in months. Of the 429 messages, 8 were not spam, which honestly is kind of a high false positive rate.
Incidentally, two messages from Google themselves were correctly marked as spam, in that they were advertising for services that I didn't care about.
In my experience struggling to deliver email from my self-hosted server, Gmail's by far the worst. Not that I haven't had the occasional issue with Yahoo, Outlook and AOL. But here's the key difference: Yahoo, Outlook and AOL actually offer tools to resolve the issue! For small senders like me who fall below Gmail's postmaster tools limits, we have no recourse other than trying to find some sympathetic Google employee and DMing them. The experience really blows.
I desperately wish Gmail would offer some sort of registration program that would let me attest to my identity and thereby give my server the benefit of the doubt. I'd happily cough up my private medical records if that meant my family's emails reached Gmail's inboxes most of the time.
Thanks for giving it a look! Enforcing DMARC and strict SPF tend to screw up mailing lists and forwarding. I don't see how I can enable them without causing a lot of other problems for me and my users.
No arguments about my provider. Any tips on how to select a better one? I'm thinking someone who adds some friction to unblocking outbound port 25, e.g. Vultr, might be better.
DMARC shouldn't be an issue for mailing lists, as long as one of DKIM/SPF passes. Are you seeing reports of both DKIM and SPF failing for mailing lists you use?
Properly configured mailing lists shouldn't modify contents of emails causing DKIM to fail.
I personally use Linode and would recommend them. But basically any tier 1/2 provider, i.e. too big/popular to block, will do. However, I would avoid DigitalOcean per my experience due to them completely blocking SMTP on IPv6 in addition to badly implemented IPv6 support and seemingly subpar support.
The problem is that Gmail and G Suite essentially have monopoly on email nowadays. Even some pretty big companies are amongst the users. All startups are using G Suite. As are many billion-dollar companies like Twitter, Arista Networks, Netflix, WeWork, a16z, YCombinator. What do you do if you can't email any of the people at all these corps?
On fastmail I haven't seen a single legit e-mail marked as spam, whereas it happened nearly weekly under gmail. In the last year I have not seen a single (non-mailing list) spam e-mail show up in my inbox, and I have received ~10k spam e-mail this year.
Wild guess but I'm wondering if the problem isn't domain name. *.eu.org looks like a perfect domain to send phishing campaigns from
(i.e. perhaps some other subdomains of it were caught sending spam and his domain got a penalty as a side-effect?)
It's a bit surprising that he got anything through without SPF and an MX record. That seems to be the minimum to get mail into the inbox for a new domain at least, so his domain probably had some positive reputation at one point.
A while back I had someone setup a new domain with office 365 email and a completely broken spf record that only allowed his web server. Gmail accepted everything without a problem.
I agree an SPF record is an expected minimum but "use a big provider" is more relevant ime.
we can argue all day long about why this happens. more interesting for me would be, what can i do about it.
i have noticed the same problem. since a few weeks, i can no longer reach gmail from my mail-server.
my mail volume is to small that marking mails non-spam would help. besides i nowadays mainly use email for situations where i don't have an alternative way to communicate, so when i email someone, and it gets rejected, there is no backchannel to let the recipient know about that.
reading through that thread, one solution suggests itself: don't host your email server with a provider that tolerates spammers.
does anyone have recommendations?
where can i host my own mailserver, and not suffer from being stuck in an IP block that is on any blacklists?
are there other solutions?
can i use a reputable mailserver as a proxy? so that mail gets forwarded to my own server, and mail i send out gets sent through them, but i still do all the filtering, routing, management of email addresses, storage, etc, of emails on my own server?
i know this is technically possible but which servers out there support that?
Although the post is about self-hosted mail, I wouldn't be surprised to learn something fairly major changed in Google's spam filter a couple of weeks ago.
We use Google Apps (i.e. Gmail) and two weeks ago one of my co-workers noticed that another co-worker's emails were going to spam. Even though we're sending/receiving within the one GApps domain.
Upon investigation I discovered that one co-workers emails were indeed going to spam for everyone, myself included unless he sent the email from his Windows machine. In which case it passed Google's spam filter no problem. His mail signature is more complicated on Apple Mail, 2 URL links (to our own domain) and an embedded image (NOT a URL).
He's been using that same signature structure for yonks; yet just in the last two weeks it's a no go.
It was two weeks ago when I started suddenly having delivery problems (ie consistently delivered to the spam folder) for emails sent from SES from AWS.
It’s very obvious to me now when reading these replies that Google changed their spam filtering and it’s affecting many more than just me
The issue affects only outbound mail, not the ability to operate your mail domain as such.
The fix/workaround is to find an SMTP relay which has reputation, and send your SMTP through that. At least temporarily, until the domain you want to be gets better reputation.
This is what people do who operate little domains that are physically hooked up via residential subscriber IP addresses. You will never be able to send SMTP anywhere from a subscriber IP. Rather, you typically use your ISP's mail forwarding hosts (usually via an authenticated connection: your SMTP server logs in to theirs to do its business). (Otherwise they would be operating an "open relay".)
The legal naivete and indifference of otherwise technological savvy people is mind boggling.
It is not the task of mail providers to block or hide email considered to be spam. It is the job of the end users, ie the email providers cusstomers and their software.
Consider physical junk mail, ie the type that comes through the letter box.
The post office does not decide what is junk mail or not, their job is to deliver it and it is the customer's job to discard it.
The proper thing is for the mail provider to tag the emails their systems consider to be junk and deliver it the customers inbox and let the customers decide.
Google's customers are the problem, not Google itself.
> Google's customers are the problem, not Google itself.
Kind-of?
Ultimately, Google's customers are relying on Google's own filtering methodology. That's normally reasonable -- for the vast majority of people the spam filter works (from the recipient's perspective) so often that the spam folder is rarely even checked for false positives.
However, Google has a scaling problem as noted by Brad Long later in the linked thread: linear scaling of human effort is too costly for them. That ultimately means that Google's services are offered on a like-it-or-leave-it basis, but leaving is made harder by the integration across the whole "Google ecosystem."
I remember when Gmail was new. You got an e-mail that said "hey, we think you are sending spam, please do this". I replied and said that I always send from this domain, so if they think I'm spamming, just block the domain, then they white-listed it instead.
I always see DMARC as a solution in these kind of threads, but the problem is that it's trivial for spammers to also setup DMARC. In fact almost 100% of the spam I receive follow the e-mail guidelines better then most legitimate e-mails.
We need a new protocol! Where e-mails are opt-in, like with messaging apps, you first need to be introduced.
No, not at all. There are reports that even if you use G Suite, and whitelist your own domain to send yourself logs through crontab, your may still have problems with the email deliverability.
My own domain was blocked for nothing more than sending myself a list of a few dozen domain names through cron, which Google deems as containing malicious links, even though there are no links present.
For some reason, Google also marks "Google Alerts" notifications as spam. They contain spammy search results, but the email comes from Google.
At least they are consequent in not white-listing their own services :)
Gmail also marks messages from gmail or other google services as spam. Gmail also marks all legitimate mails from Paypal as spam (phishing). I use several email addresses and google accounts, but in the end I forward most mails into the same gmail box. Their spam filters cannot handle the concept of forwarded mail correctly.
I'm 100% okay with this aspect of Google controlling mails, etc.
The level of spam I have received has dropped precipitously because of it. For that I'm grateful. Whatever they have done has worked, and people don't seem to remember how bad it was 10 years ago. Now my email is relatively clear of spam and that's a great thing for safety and security as well.
OT: Outreach via email and I am not talking about doing some crappy newsletters has the highest sophistication level in our company. And I am not talking about setting up some lame DMARC/SPF, we (must) do 10x more. DB adminstration is easy compared to getting emails through on a large scale.
Just to add another experience, our company also started getting a lot of legitimate email marked as spam as of about two weeks ago. This included emails coming from the same domain (with all the proper SPF and DKIM setup in place).
I see most people saying Google errs too much on the side of its spam," but it strikes me as better for everyone that spam never get through so as to incentivize people not to send spam, thereby destroying most spam before it even makes it to your spam box. You can always check your spam box, so its just a mild inconvenience mbst of the time. If you're expecting an email, you can whitelist the sender. If you're sending email to someone who isn't suspecting it, then maybe you need to take a moment out to think about whether you're spamming people's inboxes with emails they don't want.
I run into a similar situation, and I have done every thing you are supposed to. DMARC, SPF, MTA-STS, DKIM, all of it. I have only one IP address that ever sends legitimate email from my domain (I proxy all my personal messages through that IP), on AWS, and it's been that way for years.
Despite that, the reputation systems give my system a neutral reputation. Sure, I know that there are lots of people who spam through AWS, but AWS IPs are not allocated in bunches, and anyone can get one. There should be some way to get my reputation up to good, but I haven't yet figured it out.
Neutral, in your case, probably means that there is not enough volume coming from your server, and as far as I know the reputation of your /32 does not depend on your block (of course, providers may still decide to filter your block). Did you set up the PTR record and SMTP banner?
I had similar problems with Gmail rejecting mail from my perfectly configured domain. I eventually gave up after finding, sometimes months afterwards, that an email was not delivered (and when found within 30 days, it was always in the recipient’s spam folder).
I eventually switched to fastmail (from a reputable, but too tiny for google to care about provider), and all the problems were gone.
But ... I am almost sure the issue was that
I had a catchall address cc:ing a Gmail address; not sure why that would trigger spaminess detection, but that’s the best I could reason with my data.
I think the major problem here is IP reputation. Especially if you are hosting on public VPS or worse your home network. The root of the problem is spam abuse. Spammers have no problem to spin up random VPS to send spam. They can set rDNS, SPF, DMARC too. There is enormous amount of spam. Of course it should be handled on message-by-message basis, using proper antispam and ML. But it looks like some providers simply block various IP prefixes. :( It makes their work easier and it is marketing to use their services "which work".
The domain didn't even have an MX record defined until recently, let alone DKIM and SPIF. I'm surprised Gmail ever delivered emails from that domain to anywhere other than spam.
I have the problem of spammers sending emails faking they are from my domain but through hacked computers. So I get thousands of rejects. DMARC shows you where it is happening, spf can cause some of it to get blocked, but you are stuck with anyone can make up email addresses and send them out through other mail servers and in-turn your domain is certainly going to start going to spam due to this.
A few weeks ago, we organized a competition and we asked the top 3 participants to send us their solution to our gmail. Two were using their own domains (.io and .su) and one was using Gmail. The two with their own domains just got in the spam folder, and without surprise that Gmail one survived.
"If you want to set up your own email domain rather than using Gmail, good luck."
I sent a text format mail containing a couple of innocent words and a youtube link of a completely innocent clip to my wife. Gmail in their infinite wisdom classified it as spam. Obviously users need to be protected from watching youtube clips.
I used the smtp server of my ISP. I'm know that it is tightly monitored and not the source of any significant amount of real spam.
This is truly a severe problem, and I've given up trying to fix it. I don't run my own mail servers anymore. I have an Outlook business account. Microsoft is big enough that google doesn't mess with their email.
I sort of understand why Google does this--probably 95% of email from random mail servers are spam! But that doesn't make it right.
As far as I can tell, they're based in Switzerland. The political pressure if either of them EU cough a bit will probably cause Swiss authorities to fall over immediately. I wouldn't consider myself safe in your position.
My email is not valuable enough for the US or the EU to put pressure on them. If you are so worried you have to encrypt your email anyway. A Russian hoster likely would not bend. Be advised that a Russian IP address does not help you to pass Spam filters...
This was the single reason I couldn't operate my own mail server. My mails were just always marked as spam by Google, no matter how much I made sure the whole TLS/DKIM/DMARC/DNSSEC/etc setup was correct. We tend to think about E-mail as something that is still distributed, but Google really killed that.
I ran into this issue many years ago, and I eventually gave up trying to increase the reputation of my mail server. For all my projects that require mailing, I've now succumbed to spoofing headers using Gmail's SMTP, which surprisingly doesn't go to spam, or using a third party mailing service like AWS.
I've had multiple legitimate emails from Google employees and their sales teams all end up in my Gmail spam folder, even from senior members of various divisions.
I still get spam in my inbox.
They definitely know this is a problem and have people working on it - but spam filtering at that scale is much harder than it may appear at first.
If you have an email provider that classifies ham as spam, complain.
If complaining doesn't help, consider switching away.
Remember at Google you're not the customer. The reason Google want to keep you using their product is so that they can learn more about you and then ask their customers more money to show ads to you.
Gmail always marks the emails from my veterinarian, who has {business_name}@gmail.com, as spam. It's slightly irritating. However, seeing the mails that come in to our sales endpoints, gmail addresses are responsible for a huge percentage of the obvious spam.
It's not just gmail. When my university went to office365 mail for faculty, pretty much all of my email from the upper administration (e.g. the president of the university) was marked as junk. I complained, but there were no solutions.
Plain text email with no attachments is almost never marked as spam from what I have seen (unless there are several spammy keywords). I wonder why people don't use plain text email to ensure more reliable delivery of important messages?
Unfortunately, OVH doesn't care about spam from their networks, so people really need to move. Too bad we don't have other ways to properly punish network administrators who ignore complaints.
Google will of course argue it's for it's users sake. What are the odds that if legislation passes, Google simply lets unnecessary spam through and pretends they can't do better?
I would hazard a guess that this mostly happens when many people mark your messages as spam from your domain. Or if you send enough email to gmail users with certain keywords/phrases.
This is why I disabled Google spam capture entirely on my account. They were capturing things that I had marked as non-spam many times over, so to me, the feature became worthless.
Trust me it’s not, I’ve read theM countless times, google just decided I’m spam, and don’t tell me why. I run a small business and have a business name registered in the same name which cost $500 AUD, I don’t appear to have an option apart from paying AWS or Microsoft to host me.
One could just use a Gmail account for contacting people with Gmail addresses. If there's extended conversation, then migrate to ones preferred email account.
The do that to me all the time and also tell me there is suspicious activity on my account when I login and there is no way to turn off these "features"
it sucks that even though i haven't actively used gmail in at least 5 years, google is still able to hoover up my personal, and what should be private, info. and this kind of anti-competitiveness, which seems to come to light occasionally and only dimly at that, stinks just as bad.
i can't wait until everyone else realizes how insidious a company google is. it's starting to happen. i'm hearing the low rumbles among (non-technical) folks i know. thank goodness!
google is trying to become a large monopoly for email services tbh. Not only that but more and more businesses start using gsuite to avoid getting marked as spam for most clients and customers
Yep the person in charge of this at Google should be fired.
And I say this as someone who has no reliance on or use of outbound email marketing whatsoever.
In other words, I’m not an embittered emailer who thinks their email is being unfairly treated as spam. I’m just a user who sometimes looks in the spam folder and is aghast at some of the false positives Google has. And yes some of the false negatives in the inbox as well.
I think it should suffice to know that after one repeatedly trains the google spam filter with "report not spam" and "mark as spam", it should learn that well enough, no?
As an example: Some dumb ass Verizon customer gave them my email address as their own. I got a ton of communications from Verizon that I could not turn off (bill past due, thank you for you promise to pay, and all sorts of really annoying messages). I called Verizon, I was told that neither them nor I could do anything about it because I did not own the Verizon account, and I did not have the account PIN. Their official solution for this problem: Mark all the Verizon emails as spam and don't call them again (I kid you not).
The result: I marked about 100 Verizon emails as spam and Gmail learned that really well; now I do not see Verizon emails in my inbox. Why TF can't gmail do this with all the other emails I mark as spam and not spam?
I just got in the habit of firing up the hideous gmail app on my phone to go through my spam folder every day and vet out the non-spam out of there. I usually go though about 250 emails per day. I end up looking at all the spam, but it's far faster to fish out the good messages from spam than if I turned spam detection off altogether.
No matter how complex the problem to solve is, Google is failing at a level where even I could program it better, which is not a high bar at all.
> Google is failing at a level where even I could program it better
You should do so. You can set up filters to hook into keywords, domain names, etc., and route to spam. If the stuff you're getting is that obvious in structure, you should be able to kill it with a handful of such filters.
Do you know by chance know if user-0defined filters take precedence over Google's spam detection? If so, I might go for it.
I say "might" because no matter what I program into it, I'll never be sure there's not an important false positive that got trapped, and so I'll forever be checking my spam anyway. It might be all for naught.
Why don't you build your solution and show it to HN and Google or Microsoft?
I'm sure someone at a big company would convince an exec to pay you at least $10M for your improved spam filter.
Not at Google. The Google one is so bad there’s pretty much no explanation I can think of for it other than nepotism. Which means it isn’t getting displaced any time soon.
Scale is Google’s go-to excuse for some things but in this case it doesn’t hold up to scrutiny.
Having in my sent mail and inbox a traceable record of me, authenticated with a Google login, subscribing to a mailing list and then getting a subscription request acknowledgment back and then sending a further confirmation email, should be enough to tell the spam system that yes dammit I am interested in mail from this list and it is not to be considered spam, assuming it is on topic (a determination that is like ML 101 first quarter stuff and has been for some time, even across languages, like going back to, say, Damashek). That heuristic is so simply basic that I would be embarrassed for Google if I had to explain it to them. And yet there it is. Just one of many possible examples of how lame their spam filtering system is.
Solving it perfectly is very hard, you are right on that.
But solving it better than Google has? Not hard at all. A high school programming class could do it better.
Unlike other parts of Google, which are doing amazing things like beating human masters at Go, something is seriously fishy in the spam filter tool department.
What a facile response. First, I said it would be hard, not easy. More importantly it would be pointless since Google controls which system they choose to use and their choice is clearly not driven by quality, but by something else.
Did you try changing your name to Netflix, Etrade or American Express? I swear, no matter how many times you mark their emails as spam, they eventually wind up back in your inbox ( admittedly in the Promotions section rather than the Primary inbox ).
Prior to GMail, we used to check our "Junk" folder occasionally in whatever email service we used.
When GMail arrived with its nice spam classification, it was great. Worked well, better than any competitors out there. This gradually conditioned us to ignore the "Spam" directory completely. Now things have moved to the opposite extreme. It is quite difficult to host independent email servers and send emails from it without GMail classifying it as spam. The high false-positive rate has become the norm. Instead of GMail trying to reduce the false-positive rate, it has now become the responsibility of those hosting email servers to jump through the hoops to ensure that their emails are not classified as spam.
Makes one think: If we knew what we know today about spam and abuse of Internet services, would SMTP have been designed the way it is designed today? What fundamental design change could have made it immune to spam? Perhaps a pull-based design instead of push-based? Perhaps something even cleverer?
My conspiracy theory is that google intentionally has degraded the ability of spam to be detected.
Why? Google Inbox did it absolutely perfectly. They wrapped up every email, from every major source, into nearly perfect bundles.
You could receive hundreds of unsolicited ("sOlIcItEd") marketing emails a.k.a. spam and Inbox would perfectly roll them into one bundle with a 1 swipe gesture to archive.
Then they killed Inbox and Gmail has functionally no ability to collate spam into a bundle and hide it.
That's intentional. Email marketing is huge, and google has a massive vested interest in ensuring that gmail users can see advertisements.
Thanks, that means you support these practices and you like that only way to prevent your mail to go to the spam folder is to actually pay for their services and send email only through them. Soon we will have to confirm our identity and pay google in order to use the internet as a whole, otherwise all recaptchas will fail. It will soon become survailance like in China just controlled by monopolies (tight to the government anyway). :)
It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.