This update comes from Wayne Graham, Technical Director at CLIR + DLF. You can follow him on Twitter @wayne_graham and Github @waynegraham.
Late last week we started getting notices that folks on our various email lists were getting unsubscribed.
@wayne_graham a bunch of folks with gmail on the code4lib list are reporting receiving “You have been automatically removed” email. Ideas?
— chad nelson (@bibliotechy) September 14, 2016
That’s not good :/
After a bit of digging around in the log files on the server, I discovered this issue was affecting all email going through Google’s servers (Gmail and any of their domains for work).
That’s really not good.
Digging even deeper (and with some helpful folks in the code4lib Slack), we discovered that Google recently updated how it handles mass emails. Since then, I’ve been working to 1) stop this unintended unsubscribe from happening and 2) make sure this doesn’t happen again.
We currently have a temporary fix in place and, from what I can tell, these emails are now reaching their destination. The longer-term fix involves migrating domain registrars, and it should be completed by September 26th.
In the meantime, if you run into any problems, feel free to reach out to us on Twitter or find me in the code4lib Slack.
The Technical Bits
As I started looking at this issue, I noticed a consistent entry in the log files for addresses routing through Google’s servers.
https://gist.github.com/waynegraham/ed30eba7f4db6aac6e31932939abf3d9#file-mail-log
Ugh, I know what this is… So a little background (and a warning that this is going to get pretty technical).
For a few years now, Google has had an email policy that places IPv6 requirements for servers sending bulk email (well, all email basically), requiring a PTR record for reverse DNS. Google also suggests having an SPF and/or DKIM check in your DNS. Not too difficult, so I added an SPF record for the IP (v4) for the server. However, eth0 has both IPv4 and IPv6 addresses, and Google’s servers, understandably, try to do everything in IPv6.
Easy, right? Just head over to the domain registrar and add a AAAA record. Much to much shock, the only way to do this is to send them an email to request someone to register it for you. Boo. Time to start planning a domain migration.
But the more immediate problem still exists on how to handle delivering emails through Google’s servers. Google’s guidelines say that IPv4 should work, so I updated the SPF record to use the IP for the server. However, Google still thought email was coming from the IPv6 interface. After some research, I found a lot of people complaining about Google’s IPv6 policies as well as a workaround that would take the permanent rejection from Google and turn it into a temporary one for Postfix to retry the delivery via IPv4.
In my main.cf file, I added this:
https://gist.github.com/waynegraham/ed30eba7f4db6aac6e31932939abf3d9#file-main-cf
And in /etc/postfix/smtp_reply_filter had this:
https://gist.github.com/waynegraham/ed30eba7f4db6aac6e31932939abf3d9#file-original_smtp_reply_filter-sh
This served as a stopgap while I planned to migrate domains to a service we could manage AAAA records better. Everything was working pretty well until early September (the best I can tell), when Google updated the error message sent after unsuccessful IPv6 reverse DNS lookups. Thanks to @jeff in code4lib slack, we discovered that the string information was removed from the header, causing the regex to fail.
With this knowledge, I updated /etc/postfix/smtp_reply_filter with a regex to read:
https://gist.github.com/waynegraham/ed30eba7f4db6aac6e31932939abf3d9#file-smtp_reply_filter-sh
This appears to stem the email messages from being blocked, and in watching the mail.log file, the responses from gsmtp servers are like this now:
https://gist.github.com/waynegraham/ed30eba7f4db6aac6e31932939abf3d9#file-mail2-log
At this point, I’m continuing to monitor the email server logs for issues (if you’re having one, please let me know). I’m also working with our registrar to add the IPv6 address as I start the domain transfer (I couldn’t even update the name servers while I waited), but conducting domain registrations via email is a frustratingly slow process.
