Subject: Re: [EAI] EAI review topic: VRFY/EXPN

From: Shawn Steele <[email-masked]>
Date: Tue, 28 Dec 2010 04:54:09 +0000

I'm a bit confused about VRFY/EXPN.  They are both potential vectors for brute force determination that accounts exist.  I'm not sure what the value is for adding that spammer support to EAI?  Since we're discussing it anyway, I'd be happy just dropping those.

-Shawn

 
http://blogs.msdn.com/shawnste

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 28 Dec 2010 20:35:12 +0000

Shawn Steele wrote:

>I'm a bit confused about VRFY/EXPN.  They are both potential vectors for brute force determination that accounts exist.
>
Some implementations require use of SMTP AUTH before these commands can 
be used. Our MTA (Isode M-Switch) has a configuration option for that.

>  I'm not sure what the value is for adding that spammer support to EAI?  Since we're discussing it anyway, I'd be happy just dropping those.
>  
>

---

From: Shawn Steele <[email-masked]>
Date: Tue, 28 Dec 2010 23:04:49 +0000

Forgive my ignorance, but us this  useful?  Like if you trust me, aren't there better was for me to get addresses?  Like from a directory or something?

- Shawn

Sent from my Windows Phone 7

---

From: [email-masked]
Date: Tue, 28 Dec 2010 15:35:25 -0800 (PST)

> Forgive my ignorance, but us this  useful?

It's useful for infrastructure probes and checks, to the point where I've seen
several sites impose a requirement that all MTAs have VRFY/EXPN enabled.

> Like if you trust me, aren't there better was for me to get addresses?  Like
> from a directory or something?

What matters is how things are actually handled by the software routing
messages around. Peering directly into the underlying data sources
necessarily ignores the multitude of semantics implemented and applied by the
MTAs themselves.

If you really need to know how something is going to be routed, you're much
better off asking the router what it would do than trying to second guess it by
looking at a few of the data sources it uses. In our MTA, for example, the
functional specification of just the semantics that are applied to LDAP entries
during alias expansion is around 50 pages long. Do you seriously think you're
better off going at LDAP directly and having to implement all of that yourself
when you can just send a command and get the information back?

Oh, and your point about how VRFY can be used to guess addresses is mostly
specious. Given the need to reject messages as soon as possible so as to avoid
expensive and unnecessary antispam processing, address guessing can be done
using RCPT TO. I've never been entirely clear on how VRFY earned such a bad
rep, but it's mostly undeserved.

				Ned

P.S. We also support requiring authentication prior to being able to use these
commands, but I'm not sure I've ever seen anyone actually use such a setup. The
closest I've seen was access control based on the IP of the client, which we
also support.

---

From: Shawn Steele <[email-masked]>
Date: Wed, 29 Dec 2010 00:26:14 +0000

Good enough for me, thanks :)

-----Original Message-----
From: Ned Freed [mailto:[email-masked]] 
Sent: Pōʻ, Kekemapa 28, 2010 3:35 PM
To: Shawn Steele
Cc: Alexey Melnikov; [email-masked]
Subject: Re: [EAI] EAI review topic: VRFY/EXPN

> Forgive my ignorance, but us this  useful?

It's useful for infrastructure probes and checks, to the point where I've seen several sites impose a requirement that all MTAs have VRFY/EXPN enabled.

> Like if you trust me, aren't there better was for me to get addresses?  
> Like from a directory or something?

What matters is how things are actually handled by the software routing messages around. Peering directly into the underlying data sources necessarily ignores the multitude of semantics implemented and applied by the MTAs themselves.

If you really need to know how something is going to be routed, you're much better off asking the router what it would do than trying to second guess it by looking at a few of the data sources it uses. In our MTA, for example, the functional specification of just the semantics that are applied to LDAP entries during alias expansion is around 50 pages long. Do you seriously think you're better off going at LDAP directly and having to implement all of that yourself when you can just send a command and get the information back?

Oh, and your point about how VRFY can be used to guess addresses is mostly specious. Given the need to reject messages as soon as possible so as to avoid expensive and unnecessary antispam processing, address guessing can be done using RCPT TO. I've never been entirely clear on how VRFY earned such a bad rep, but it's mostly undeserved.

				Ned

P.S. We also support requiring authentication prior to being able to use these commands, but I'm not sure I've ever seen anyone actually use such a setup. The closest I've seen was access control based on the IP of the client, which we also support.

---

From: John Levine <[email-masked]>
Date: 29 Dec 2010 15:59:23 -0000

Just as a reality check, does anyone have any idea how many mail
servers still support EXPN or VRFY on the public Internet?  I tried a
bunch of large familiar mail systems, and got mostly 502 responses,
with a smattering of 500 and 252.  It is my impression that online
directory lookups happen with LDAP if they happen at all, and this
tempest is in a very small teapot.

If we do think that it's worth EAI-zing them, I reiterate my
suggestion of a STARTEAI command, rather than adding syntactically
tricky parameters to MAIL, VRFY, and/or EXPN.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: [email-masked]
Date: Wed, 29 Dec 2010 09:32:14 -0800 (PST)

> servers still support EXPN or VRFY on the public Internet?

Which, given how VRFY/EXPN are actually used in practice, is a completely
meaningless number and a check of absolutely nothing close to reality. Just
because an option isn't offered to random schmoes like you who happen by
doesn't mean it's of no use. You might as well argue that because you tested a
bunch of MTAs and found that they weren't willing to relay mail for you, we
shouldn't concern ourselves with relay operations in our standards.

> I tried a

Well, all I can say is that you are quite simply wrong. And I have already
explained why in previous notes.

> tricky parameters to MAIL, VRFY, and/or EXPN.

First of all, there's nothing problematic about parameters to MAIL - we have
already defined a bunch of them, and aside from a couple of glitches having to
do with encoding disagreements - which had nothing to do with the overall
parsing of the line - I know of no interoperability issues they've caused.

Second, the notion of using a general announcer is a truly terrible idea. The
reason it's a terrible idea is that only some subset of the messages a given
MTA handles will be EAI messages. So when you process an EAI message followed
by a non-EAI message, you most certainly do not want the non-EAI message to be
flagged as EAI. So you need a way to turn that indicator off. There are two
ways to do that: Disconnect/reconnect or provide an off-switch. But now you
have what amounts to a global state bit that you have to keep track of. And
this is a sure-fire recipe for implementation issues and interoperability
problems.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 29 Dec 2010 13:13:34 -0500

--On Wednesday, December 29, 2010 9:32 AM -0800
[email-masked] wrote:

>...
> Second, the notion of using a general announcer is a truly
> terrible idea. The reason it's a terrible idea is that only
> some subset of the messages a given MTA handles will be EAI
> messages. So when you process an EAI message followed by a
> non-EAI message, you most certainly do not want the non-EAI
> message to be flagged as EAI. So you need a way to turn that
> indicator off. There are two ways to do that:
> Disconnect/reconnect or provide an off-switch. But now you
> have what amounts to a global state bit that you have to keep
> track of. And this is a sure-fire recipe for implementation
> issues and interoperability problems.

Well, one could also organize things so that RSET would do
provide the "off" function, rather that requiring (bleech)
STOPEAI or "STARTEAI STOP".  But that is just a way to
manipulate the global state and does not make it a less terrible
idea.

     john

---

From: "John R Levine" <[email-masked]>
Date: 29 Dec 2010 13:27:26 -0500

>> servers still support EXPN or VRFY on the public Internet? ...

> explained why in previous notes.

I saw that you said that some places use them internally, perhaps after 
AUTH.  That's fine, but that seems like private network stuff, which 
hasn't generally been a priority to standardize.

> already defined a bunch of them, ...

> Second, the notion of using a general announcer is a truly terrible 
> idea. ...

If you have a flag on MAIL, the MAIL command turns on a global state bit, 
the flag has to be remembered through the session through RCPT and DATA. 
If there are other random commands in between such as NOOP or EXPN or 
VRFY, it has to be remembered past them, too.  The bit gets turned off 
sometime between the dot at end of data and the next MAIL command.

If you have a flag command, STARTEAI turns the bit on and RSET or EHLO 
turns it off.  I'm having trouble seeing why the first bit is good and the 
second bit is bad.

There's nothing wrong with a MAIL flag, but flagging EXPN and VRFY seems 
to be a problem due to the loose syntax of those commands that, as far as 
I can tell, would have to retroactively make whatever the flag is into a 
reserved word in those commands.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: [email-masked]
Date: Wed, 29 Dec 2010 20:39:12 -0800 (PST)

> hasn't generally been a priority to standardize.

First of all, nowhere did I say that this is "private network stuff", and in
fact it isn't. It is fairly common to outsource various sorts of network
auditing and management operations, including these sorts of checks, with all
that such outsourcing implies.

But let's suppose for a moment that this really is just "private network
stuff". So what? The IETF routinely works on protocols that are relevant on the
LAN but have essentially no presence on the wider Internet like DHCP or
mDNS/Bonjour. Even LDAP, while it was intended for Whites Pages services, is in
practice "private network stuff".

And all this is exactly as it should be: While it is vitally important that the
protocols actually operating across the Internet be properly specified and
standardized, it is equally important that the operations and management
aspects of the network also be standardized so that products from different
vendors can interoperate, even if said protocols operate entirely within the
confines of private networks.

And insofar as the email community is concerned the "private network stuff
isn't the IETF's business" has a somewhat unpleasant history: This is the main
argument that was used to block standardization work on POP and IMAP for a
considerable period. In hindsight, this argument should have been rejected as
bogus a lot sooner, and had that been done a number of issues might have been
avoided.

> If you have a flag on MAIL, the MAIL command turns on a global state bit,

Nope. It turns on a transaction-specific information bit, a bit the MTA has to
maintain as part of the message metadata order to fulfill it's obligations
should the message require further relaying. There is no state bit involved and
no need for one.

> sometime between the dot at end of data and the next MAIL command.

Nope. But even even if this were the case, you're focusing on the server and
ignoring the client. It's the client, not the server, where the complexity of
using a global state bit really manifests - in order to implement your approach
the client has to keep track of the state it has put the server in. This is
mostly because servers can be, and often are, written as bunch of state
variables and a simple command loop. Clients, OTOH, tend to manifest as a bunch
of branching code paths that work very differently depending on what extensions
are present and needed. And sure enough, history has shown that client state
maintenance is one of the most common, if not the most common, places where
implementors make mistakes - they miss some client code path and things
desynchonize.

> turns it off.  I'm having trouble seeing why the first bit is good and the

Wow. I didn't think there was an even worse idea than a STARTEAI/STOPEAI
command set, but you managed to come with it. Your proposal now is to have
commands that can be - and often are - issued at various times during a session
result in a change of state that's totally unrelated to the purpose the command
is typically being issued for. In our client code, for example, there are at
least four places that issue EHLO and well over a dozen that send out a RSET.
All of those would have to be modified to track the EAI state changes.

> reserved word in those commands.

Then you need to reread my proposed ABNF for adding a parameter. You're
imagining issues that don't exist.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 30 Dec 2010 09:00:06 -0500

> All of those would have to be modified to track the EAI state changes.

OK, you win.  I hadn't realized your code was so bad.

Regards,

---

From: [email-masked]
Date: Thu, 30 Dec 2010 10:43:18 -0800 (PST)

> manipulate the global state and does not make it a less terrible

The nice thing about the way RSET works presently is unless your client needs
some fairly wierd command ordering that gets into the odd historical coupling
between RSET and HELO, a RSET maps directly and exclusively to a reset of the
transaction state. So when an implementation wants to reset the transaction
state, it just needs to call its "send command, check status" routine with
"RSET" as the parameter, done - no need for additional state checks or an
additional layer of processing to track any other state variables.

But when you add additional glboal state, you either need to add code to track
that everywhere or introduce an indirection layer. (I"d probably opt for the
latter since it results in cleaner code overall, but it's more work.)

Currently the trickiest aspect of SMTP implementations is client error
handling. I'd like to keep it that way, especially since for the most part
client error handling is a sunk cost.

				Ned