from the mail-only-accepted-from-ourselves dept.
On his blog, Peter N. M. Hansteen sometimes writes about the problems with getting certain mail service providers to up their game. This time his post provides the details on how a particularly large service not only fails at SMTP sender verification but also at many other tasks necessary for professional mail hosting.
Whenever I encounter incredibly stupid and functionally destructive configuration errors like this I tend to believe they're down to simple incompetence and not malice.
But this one has me wondering. If you essentially require incoming mail to include the contents of spf.outlook.com (currently no less than 81 subnets) as valid senders for the domain, you are essentially saying that only outlook.com customers are allowed to communicate.
If that restriction is a result of a deliberate choice rather than a simple configuration error, the problem moves out of the technical sphere and could conceivably become a legal matter, depending on what outlook.com have specified in their contracts that they are selling to their customers.
One takeaway is that spam-fighting decisions from decades past have left us with technologies that have led to the centralization of mail on fewer and fewer providers. As such it is increasingly difficult for even skilled professionals to operate their own mail hosting smoothly.
Source : A Life Lesson in Mishandling SMTP Sender Verification
(Score: 1, Informative) by Anonymous Coward on Monday February 19 2018, @06:19AM (25 children)
Is that it was mostly vulnerable windoze boxes that lead to the whole spam issue and the mostly retarded methods to fight it. If you were into conspiracy theories, you could even think M$ was playing the long game here. And this current behavior is no less than one would expect from a convicted monopolist.
(Score: 4, Informative) by frojack on Monday February 19 2018, @06:43AM (24 children)
While windows played a part, the real problem with email was that it's usefulness and popularity exploded in the world before the system was even half baked. The design was incompetent, the protocol pathetic, and the security model non existent.
None of that was Microsoft's fault. The idea that you could send mail to any address in the world with absolutely no way for the recipient to know for sure who sent it, or from wence it came is absurd.
The fact that world plus dog accepted this situation shows how desperately such a digital mail system was needed.
Expecting Microsoft to fix this mess, which they didn't design, and were late to embrace, is silly.
(Perhaps you were too young to remember that Microsoft was caught flat footed by this whole internet thingie).
No, you are mistaken. I've always had this sig.
(Score: 3, Informative) by Anonymous Coward on Monday February 19 2018, @06:47AM (15 children)
> The idea that you could send mail to any address in the world with absolutely no way for the recipient to know for sure who sent it, or from wence it came is absurd.
I wouldn't call it "absurd" -- AFAIK, you can do the exact same thing with physical mail.
(Score: 2) by Apparition on Monday February 19 2018, @06:57AM (13 children)
Yes, but it's far easier and much less costly to send out an e-mail. Thus there's more of it, and no financial discouragement to prevent abuse.
(Score: 4, Informative) by c0lo on Monday February 19 2018, @07:05AM (9 children)
FTFY.
At the time SMTP was specified [ietf.org] (1982), sending an email was way more expensive than sending snail-mails.
(just from curiosity: where you born at that time?)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by c0lo on Monday February 19 2018, @07:19AM (8 children)
(sorry for the typo. Q: were you born at that time?)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by Apparition on Monday February 19 2018, @07:50AM (7 children)
I was born in the late 1970s, so yes, although I was a wee lad at the time. Yes, I am aware that at the time SMTP was designed through the early 1990s, sending e-mails was expensive, but the ubiquity of the Internet in the '00s and '10s has broken SMTP completely. It needs to be replaced.
(Score: 2, Touché) by Anonymous Coward on Monday February 19 2018, @08:21AM
So far, the only alternative the "e-mail must be replaced" club has come up with is Facebook.
Not exactly an improvement.
(Score: 2, Touché) by Anonymous Coward on Monday February 19 2018, @09:57AM
If you propose a replacement that includes a blockchain, you might even get money for that.
(Score: 2) by sjames on Monday February 19 2018, @02:37PM (4 children)
So what's your proposal? Who will validate that you are who you say you are and how many hoops will you have to jump through to get them to do it? How much will they charge you? Who will keep them honest?
Now, why will that very special stamp of approval from whoever require a replacement to smtp rather than just another header?
(Score: 3, Informative) by c0lo on Monday February 19 2018, @02:50PM (3 children)
GPG with a public key I handed to you personally in a key signing party [archive.org]. Trusting anything else is delusion.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 1, Interesting) by Anonymous Coward on Monday February 19 2018, @03:17PM
I've always felt that blockchain would work fairly well to validate public keys in a distributed way.
Sign up for service, generate keys/username, post username and keys, validators incorporate those into the blockchain.
You send a message to a new person, query the chain, save the public key. Periodically compare the chain and personal key lists. Publicly post about discrepancies (could be automated even).
(Score: 2) by sjames on Monday February 19 2018, @03:19PM
That's a great way to make sure emails from my friends are really from my friends, but what about the zillion other people that might (or might not) have a legitimate reason to email me?
And, of course, that works just fine over SMTP.
But note, it's 20 years old and freely available but it hasn't solved the problem yet.
(Score: 0) by Anonymous Coward on Monday February 19 2018, @08:14PM
There. FTFY.
(Score: 0) by Anonymous Coward on Monday February 19 2018, @08:28AM (2 children)
Greylisting poses a burden on the spam sending bot in term of resources. It can choose to send less mail (and fight the sending by retrying to send the spam that was greylisted, but it needs to keep track of resending)... or just ignore it, send more, but the greylist-using servers effectively rejected the spam.
(Score: 0) by Anonymous Coward on Monday February 19 2018, @03:10PM (1 child)
Greylisting worked 10 years ago. Now the armies of windoze boxes send mail via their gmail or outlook servers, who will make repeat attempts.
(Score: 2) by frojack on Monday February 19 2018, @07:48PM
Hundreds of attempts? So what?
Gray listing means none of those attempts get through.
Spam has to work (even at a tiny fraction of attempts) or there is simply no point.
No, you are mistaken. I've always had this sig.
(Score: 0) by Anonymous Coward on Tuesday February 20 2018, @03:13AM
If you needed authentication, then you really were expected to include a signature/return contact address, or later on, use cryptographic signatures that would verify your identity.
Email, Instant Messaging, etc are all compromised due to their server based nature, so the only way for the client endpoints to know they are really interacting with the authentic party is by verifying them with secrets only each endpoint knows. Not by relying on the server as the authenticating medium (where corporate players, government agents, or hackers could compromise it in order to spoof the identity of either the send or recieving email account in a manner that appears legitimate.)
The problem is really the post-Eternal September plebs cluttering up the internet with idiocy and the sort of absentminded centralization that is making the world into an even bigger shithole than it already was.
(Score: 2) by c0lo on Monday February 19 2018, @07:01AM (5 children)
Really, frojack... slow down, mate! At this rate, you'll make the prices of straw go.... mmm... haywire!
Just exactly who asked Microsoft to clean the SMTP design?
Or did the common-sense of "do the best you can to play nice for your users" reached the level of heresy today and it can no longer be considered an idea any sane person can have? (today = a day post "customer era" and deep into the "consumer" territory).
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by frojack on Monday February 19 2018, @07:48AM (2 children)
The first post by the AC implicitly laid blame at Microsoft's door for spam and the absurd tools used to fight it.
Perhaps you missed that by never viewing AC posts. A wise choice.
No, you are mistaken. I've always had this sig.
(Score: 1, Insightful) by Anonymous Coward on Monday February 19 2018, @08:16AM
Yes, and the reason for the blame was "vulnerable windows boxes".
Going from there to expecting Microsoft to fix SMTP design is a huge leap, which can only be based on the idea that it's the rest of the worlds responsibility to be compatible with Microsoft bugs, not Microsofts responsibility to fix those bugs.
(Score: 4, Informative) by c0lo on Monday February 19 2018, @10:40AM
Have you RTFA? It's not the "absurd tools" it is the "absurd configuration of the tools" the story is about. TFS quote:
The abusrd configuration:
- allows spam be send outside outlook.com
- does not allow abuses to be reported if using an email address outside outlook.com
The result of that absurd configuration?
1. outlook.com starts to be intensively used as a source for spam...
2. ... all the while, I assume, Microsoft does the needed to keep the outlook.com mailboxes free of spam.
If that's not incompetence, the only interpretation is "Microsoft plays the long extortion game of letting spam go outside and protecting their consumers inside".
Which, I suppose is a possible interpretation of:
Yes, I admit, the AC may be right for the wrong reason; I do find the "vulnerable windoze boxes that lead to the whole spam issue and the mostly retarded methods to fight it." a bit of a... (mmm, to use some pretentiously exaggerated terminology...) poetic hyperbole.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by TheRaven on Monday February 19 2018, @09:38AM (1 child)
Just in case that was a serious question, Microsoft was one of the contributors to DMARC and has implemented support for it in their products [wikipedia.org].
sudo mod me up
(Score: 2) by c0lo on Monday February 19 2018, @10:44AM
That wasn't the point.
The point was "what are you debating, frojack? I haven't seen anyone asking that MSFT should fix SMTP".
And frojack clarified what he though can be interpreted as such.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Informative) by TheRaven on Monday February 19 2018, @09:41AM
SMTP was created in 1982 (a decade before commercial entities were allowed on the Internet). The security model was fine for the imagined deployment model: you had a few dozen, maybe a few hundred, (large, multi-user) computers on a network. Users could log into one and send mail from them. If a computer was sending email without correctly authenticating its users, or claiming to send email from someone else then you'd have a chat with their administrator and if they didn't fix it then you'd just reject email coming from their computer. The problem was trying to use SMTP on a large Internet where it wasn't possible to maintain a list of known-good email servers (or a list of known-bad ones).
sudo mod me up
(Score: 2) by sjames on Monday February 19 2018, @02:55PM
Agreed MS can't fix this. What's your proposal? I think you'll find that the more you think about the problem, the more you realize that an answer isn't really forthcoming. For the partial answers you might come up with, ask why SMTP wouldn't be the right transport protocol.
As for MS, they didn't cause the spam problem. They did, however make email and document viruses an actual thing. Before MS came along, the email virus was a recurring joke. The noobs feared the "Good Times" virus. Everyone else laughed because the idea of getting a virus from an email was absurd. Then Microsoft, in spite of many warnings from people they should have listened to, made the email virus a reality.
(Score: 0) by Anonymous Coward on Monday February 19 2018, @08:23AM
Last month I learned about DNS amplification attacks, which are often the results of poorly configured public DNS servers. These servers seem to be the cause of often used DDOS methods (which was the reason I started to read up about them).
(Score: 5, Informative) by NotSanguine on Monday February 19 2018, @08:32AM (1 child)
And read TFA.
The author received spam from an outlook.com user. He attempted to report the user to abuse@outlook.com. A normal, if usually not very effective, step in attempting to address spam.
The problem experienced by the author was that emails to "abuse@outlook.com" are auto-forwarded to "staff@hotmail.com"
Emails to the outlook.com address are accepted, but the SPF [wikipedia.org] record for the host(s) servicing the forwarding address will only accept emails which originate from outlook.com email addresses.
TFA states that it is unknown whether this is a misconfiguration or if its' outlook.com's policy to only acceppt abuse emails from outlook.com users/customers.
Either way, this isn't a story about how SMTP is broken (it's not). It's a story about how MS is either incompetent or consciously decided that those who aren't customers aren't entitled to file abuse reports.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 4, Funny) by chromas on Monday February 19 2018, @06:43PM
Traitor!
Maybe that's to cut down spam :)
(Score: 4, Interesting) by ledow on Monday February 19 2018, @08:43AM (6 children)
SMTP is fundamentally flawed.
Rather than keep patching it up and have to apply manual "approval" by various means (greylisting, spam filtering, DKIM, SPF, etc.), we need to replace it.
Where is SMTP 2? With certificate-based ownership of an email address, signed by the domain certificate, with specified origin servers which means you can encrypt all email, verify the origin of any piece of mail by whether or not the sender is able to digitally sign using the published certificate or not, reject anything that fails the signature test, and then use the normal certificate processes to verify / blacklist people (literally "do you want to accept mail from?" with sensible defaults, and the ability to look up the certificate reputation - only they could have sent those emails, so it's on their heads if valid-signed email is received - and revoke certs just the way we do now for CA's that abuse their SSL certs).
Because a vast majority of spam and malware dies when you have to set up expensive SSL sessions using private keys to send email, when you can't fake the return path (which means you can have a proper "Hey, you sent me spam, don't do it again" response unlike the unauthenticated bouncebanks which are STUPIDLY required by the current protocols), when it's a pain to revoke and change your certificates because they were marked as spamming, when it doesn't matter where email comes from (if it's not authenticated it won't arrive), and where every stage of the process DOES NOT expose plain-text messages to untrusted intermediate mail servers. Hell, you can even forward mail properly without rewriting envelopes, etc. Just have the forwarder re-sign with the proper certificate for the final destination and resend and on it goes successfully.
Rather than try and fix what we have, we need to just replace SMTP with a whole new incompatible protocol. Much like HTTP vs HTTPS, it's time "Secure Email" became a thing, completely separate from email. In fact, it's 20 years overdue.
(Score: 0) by Anonymous Coward on Monday February 19 2018, @08:51AM
Right, because "SSL" is the magic sauce that's going to fix everything. SSL (or better TLS) hasn't got its own set of problems.
(Score: 5, Informative) by TheRaven on Monday February 19 2018, @09:46AM (3 children)
You've been able to do that with S/MIME for ages. Here's the problem: Who signs the domain certificate? If you use the current list of a few hundred CAs, then it's wide open to abuse for a number of reasons. First, various CAs are not nearly as trustworthy as they are trusted. Second, now the domain cert that they sign for you is a signing cert and so you need to handle it very carefully - if that's compromised then anyone can sign email for any address in that domain. For large companies, insider threat is a big issue (think how valuable a big company's email cert signing cert would be to scammers if people actually checked and trusted S/MIME certs). If you don't use the existing CAs, now you need some centralised authority that will issue the signing certs, and that gives a single point of failure. Once DNSSEC is widely deployed (any decade now), it may be possible to use that to distribute the public key of the signing cert for a domain, but that doesn't really help with the requirement to maintain the security of the cert.
sudo mod me up
(Score: 2) by ledow on Monday February 19 2018, @02:20PM (2 children)
All your issues have been pretty well handled for everything from DKIM to code-signing itself.
Are you suggesting that they are even on the scale that I can send a million messages claiming to be from just about any domain I like and some of them will get through because of the myriad technical hurdles necessary to stop that happening at the moment.
The code-certs stored in, say, Exchange are pretty well locked down to the average user. You aren't going to be able to do that without having control of the Exchange server anyway. And to sign into the Exchange server you need valid AD credentials which limit you to what messages you can send anyway.
Are you suggesting that Exchange servers, and their ilk, are corporate liabilities from "insiders" too?
It's the mail server that needs the key. Security of an authentication to the mail server is another matter entirely. But at the moment, mailservers can claim to be "from" anyone, which they wouldn't be able to do if they don't have, say, "microsoft.com''s signing key.
S/MIME just piggybacks on existing SMTP... which is already broken in ways more than S/MIME can fix by layering over the top.
(Score: 2) by sjames on Monday February 19 2018, @03:04PM
So what's your proposal that doesn't just move the problem around without solving it or make the problem worse? Think carefully before replying, but please do reply.
(Score: 2) by TheRaven on Monday February 19 2018, @05:14PM
No they haven't. DKIM is purely server side. The server signs a key and publishes it in DNS so that no one will trust a key that doesn't come from that server. Assuming that you're using DNSSEC, this is pretty robust (if you aren't, then you have no chain of trust for the public key). The key is relatively easy to keep secure, because you need it on precisely one machine: the mail server (DKIM lets you have multiple keys, so you can have a per-server key if you use multiple MXs). These problems become a lot more complex if you need to use that key to sign a client cert for everyone in your organisation, because now you need everyone to submit a (potentially malicious file containing a) CSR, which you then sign.
Code signing is also different for two reasons. First, there is a single source of authority. If you sign a Windows device driver, Microsoft issues your signing cert. If you sign an iOS app, Apple issues the signing cert. There is not equivalent source of authority for email. Second, unless the org is run by idiots, the signing cert is not installed on dev machines and is kept on a separate machine that is solely used for distribution. Even then, some orgs have had their signing certs compromised.
The rest of your post is too incoherent for me to understand what you're proposing.
sudo mod me up
(Score: 2) by anotherblackhat on Monday February 19 2018, @03:25PM
Lots of people have made lots of suggestions for replacing email, but so far none have been successful.
The biggest problem for the anti-spam effort is that any method of eliminating spam must prevent current emails from going through.
I.e. to stop spam, you have to stop "normal" email from working.
The great thing about email is anyone can talk to anyone.
The horrible thing about email is anyone can talk to anyone.
So far, no one has come up with a way to fix the second part without damaging the first part.
(Score: 3, Insightful) by daver!west!fmc on Monday February 19 2018, @09:37AM (4 children)
As I read the source article, I got the idea that outlook.com's mail service is autoforwarding its abuse@ mail off to something in hotmail.com which is perhaps read by the actual operators of the outlook.com service. And then hotmail.com's mail service is doing SPF-type validation of the SMTP MAIL FROM address, which is of course still the original sender of the e-mail. And of course the IP addresses of the outlook.com mail service aren't in the author address domain's SPF records, so hotmail.com's validation fails.
Running an e-mail service is hard, and MICROS~1 fail at it, and in a way that means they don't see much mail from non-customers to abuse@outlook.com, but doesn't much get in the way of mail to and from its customers. Imagine that. I do, and I don't see malice, I see a simple forgetting to understand what was done and how it really works and thus a failure to test that abuse reports are possible from outside MICROS~1's networks.
(Score: 2) by nobu_the_bard on Monday February 19 2018, @01:24PM (3 children)
That is probably correct.
The solution is a SRS (Sender Rewriting Scheme) which replaces the FROM field with an address that your SPF matches.
(Score: 2) by Hyperturtle on Monday February 19 2018, @04:58PM (2 children)
Well I sure hope not. You are asking to punish people like me, so that the incompetent can go on being unpunished and reward those choosing low priced options that outsource to the lowest bidder.
In order for me to do appropriate business-to-business billing (or corp-to-corp) with several companies, they ***REQUIRE*** that I send them an invoice from the domain of the customer.
That isn't possible. I don't have addresses with the customer. They are not my email provider. It would cost them money to add an email account, to manage it, etc. Instead, that customer has required me to submit invoices to their billing company, typically something outsourced, and more and more frequently, overseas to a country with cheap IT labor. They didn't give me an email account to do this -- the norms in the US is that a business doesn't use another businesses resource to actually bill them with. Some people may do it, but I try not to. Something about credibility and boundaries...
That cheap accounting labor has a problem with me using my own business email, because they only whitelist their customers for invoice processing. They will not add me to the white list -- doesn't matter, no arguing has helped, and the people I spoke to were not even able to have full discussions because I don't speak their native language even poorly. No amount of protests from the mutual customer has helped get my domain added to a whitelist. I suspect there isn't even whitelist.
So what I have to do is exactly what you said, Nobu_the-Bard -- I have to type in the customer domain in the FROM field with a fake email address that is like "Hyperturtle@thisistotallythecustomersdomain.com" and then CC me on it for "FYI" purposes, then send it to "timesheets@wescrewitallupforeelcheap.com" and hope for the best. I have to fake where the address is coming from in order to get paid. And it works, because the billing companies overseas in the IT centers of the world are incapable of properly configuring their email systems or train their users on how to use email. (I suspect they had a hand in configuring Outlook.com..)
It's not malice on their part; it is incompetence. Why they can't accept an email from my domain is beyond me, especially since they can reply and say sorry we need it from the other domain because we can't read this.
This can't even be blamed on dysfunctional bureaucracy, yet it feels a lot like it. Instead, it is just purely stupid, instead of power being wielded stupidly.
I can't argue with the costs of unskilled labor like that and decided I can't reason with it either, so I faked it. I used the from field in email to list the customer domains when I send my bill, and hope I get paid. When I asked if the email came through OK, and they confirmed they did and replied to me that "were glad u solved ur pc issue". Yeah, I wish they'd solve theirs because now I am the one that looks stupid.
(Score: 2) by nobu_the_bard on Monday February 19 2018, @06:22PM (1 child)
I'm not sure what you're talking about. You're taking advantage of the weak enforcement of SPF. That's... fine. It's normally not strongly enforced. It is not strongly enforced because things like this happen.
I complain about it, because it'd be a better tool if it was strongly enforced, but it isn't, and there's real reasons it isn't strongly enforced, and that's the reality of it.
However if @outlook.com is going to forward to @hotmail.com and they're going to enforce SPF properly (because they probably are crazy), then they should have SRS in place for @outlook.com. That's all I meant. I just said it so people would know it was possible to get working.
(Score: 2) by Hyperturtle on Tuesday February 20 2018, @03:16PM
I didn't want to come out and disagree completely without some detail explaining why. I'd much prefer it all worked as intended, if perhaps not as originally designed. Unfortunately, it is too much to expect people to even follow standards sometimes.
(Score: 3, Interesting) by nobu_the_bard on Monday February 19 2018, @01:33PM (1 child)
Most of the big mail providers won't block mails that fail only SPF. I suspect they just score on it.
There's probably two reasons for it:
1. Many, many people have incorrectly written SPF records.
2. Many of the largest mail providers can get by on nearly just content scanning; providers like Gmail have MASSIVE amounts of information about spam from having enough users that actually report feedback due to their scale, making content scanning a lot easier than for smaller providers.
(Score: 0) by Anonymous Coward on Tuesday February 20 2018, @01:18AM
1. You got that right. I cannot tell you how many domains I have looked at after onboarding a customer only to find wrong SPF settings and it is disturbingly common to find multiple SPF records. One customer even had 5!
2. I support tons of Office 365 and most customers just have the basic SPF record set even though MS did most of the heavy lifting to set up DKIM with a DMARC policy. I had one customer with no SPF record, no DKIM, no DMARC, nothing. I was astounded that people received anything they sent. There are so many hokey mail setups it is no wonder spam is such a problem.
I think a big part of the problem is that, when an email gets blocked, the users blame their email admin. They do not know or care that the sender may have completely wrong SPF or DMARC or nothing at all; all they know is that somebody tried to email them and "you" are blocking them from getting the email. So, naturally, email admins have to do things like not block on SPF failure, not block on DKIM failure, accept email from domains with no records, and all sorts of other terrible things. So, we end up content filters that are basically magic boxes. We end up with the lowest common denominator.