Subject: Re: Finding a working SMTP relayer

From: Keith Moore <[email-masked]>
Date: Wed, 1 Jun 2005 00:40:42 -0400

> So the next question is "is there any serious interest in working  
> on a 'how to find the SMTP relay/submission server appropriate to  
> the local topological environment'?

I'm pretty sure that I wouldn't want to see an SMTP-specific solution  
developed.  As for something more generic, I find myself torn.

On one hand, I doubt that it is wise to endorse an architecture that  
expects apps in general, or SMTP clients in particular, to do  
different things depending on their location in the network  
topology.  I am of course aware that various kinds of proxies/caches  
exist and that they are actually useful in corner cases, and that  
explicit proxies might be better than interception/ 
implicit/"transparent" proxies.  I am also aware that there is  
increasing pressure (from various sources) for applications to choose  
"the right" source address, source interface, and destination address  
(from among one or more of each) in order to operate in the face of  
routing limitations, filtering, sudden renumbering, limited-scope  
addresses, etc.

But this also seems like a way to add a lot of complexity to  
applications, and a lot of additional failure modes, for a marginal  
or dubious gain.

If nothing else there is a conflict between the application user's  
interest and the local network operator's interest.  The application  
user wants to be able to select a submission (or other) server that  
will serve his interests for authentication, robustness, content  
conversion, timeliness, data transparency, privacy, etc.  The  
operator of the network to which the user's computer is connected may  
not serve the sender's interests as well.  As long as the operator is  
only providing pure IP service then there's a limit to how much it  
can affect the sender's email traffic.
But once the operator starts providing other services then the  
separation of functionality is gone.

This intersects with security considerations.  I believe we need to  
be moving toward a layered security architecture, where we don't  
expect either the end systems or the network to be responsible for  
security, but instead the end systems, firewalls, border routers,  
intrusion detectors, etc. all cooperate to implement the intersection  
of the network's security policy and the user's security policy.   
This is because in a world where applications are expected to deal  
with apparently arbitrary constraints on reachability/routing between  
one point and another in the network, they need to be explicitly told  
when it is okay to send traffic between two points and when they need  
to try to overcome limitations in the network (or its addressing,  
whatever).

At any rate, in the near term it seems saner to recommend that  
networks NOT filter outgoing traffic for the submission service, and  
to recommend that submission servers require STARTTLS and  
authentication, than to build a way for a local network to redirect  
SMTP submissions through a proxy.   In the long term a more generic  
application policy specification might emerge if it's found to be  
useful.  An SMTP-specific mechanism strikes me as suboptimal in both  
the near- and long-term scenarios.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 01 Jun 2005 01:07:45 -0400

On 6/1/2005 12:00 AM, Ned Freed wrote:

> The API situation would definitely have to change for serious DHCP
> provisioning of applications to be viable. I do see some small signs of
> hope, however. For example, in Mac OS X the location manager seems to
> be a place where DHCP options and location-specific network application
> settings come together.

HKLM\System\Current ControlSet\Services\DHCP\Parameters\Options is where
the options are stored (and flagged for fetching) in Win2k and higher.
That's not really the problem--the problem is that no apps make use of
this data (to my knowledge).

The only app-level DHCP support I know of was some directory-locator
stuff, and that was usually issued through its own DHCP queries (this was
until the DHCP socket was permanently bound, which stopped that practice
pretty well).

As it stands right now, if you want app-level DHCP support, you basically
need an agent that will go off and update the application configuration
itself. That's mostly on UNIX although there used to be some windows
software that did this too (don't remember its name and its dead anyway).

>> So the next question is "is there any serious interest in working on
>> a 'how to find the SMTP relay/submission server appropriate to the
>> local topological environment'?   This probably generalizes.  E.g.,
>> if one believes in HTTP caching proxies, finding the right local one
>> raises some of the same issues.
> 
> Very good point. These sorts of things definitely belong in the same
> category, some of them more so than submit server addresses. Offhand
> I'd say the place to start would be an option offering  a list of
> services and corresponding servers/ports. Further elaboration would be
> needed, of course.

http://www.ehsco.com/misc/I-Ds/draft-hall-email-srv-01.txt is the latest
version of the SRV draft, which I reference here to point out that it has
some significant client-side sorting and parsing routines that ought to be
considered for reuse in something like DHCP.

Another point here is that DHCP isn't necessarily the right choice, and
that something like SLP may be more useful simply because it isn't as
restrictive to topological considerations. I mean, DHCP isn't just limited
to ~where you are on the current topology, but its limited to a subset of
topology ~types too. No dialup support, and no access from remote anyway,
so you're doubly screwed, and something like SLP could deal with that by
being separated from the topology itself and being somewhat independent of
topology-related stuff anyway.

I've always said that I think it's useful to pursue other kinds of
approaches in parallel to the SRV configuration stuff, and I still believe
that. Mostly I think this because the market seems to have most clearly
said that it wants all of the different configuration services (DHCP, PPP,
SRV, ...), and trying to pick just one is short-sighted. My reason for
doing SRV first is that its the most needed, IMO, given that it is keyed
to user config which seems to be the 80% case.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 1 Jun 2005 16:25:20 +0200

+1 - I use SSL'd IMAP and SMTP, and have never had problems yet.

Note that this may not be the case with TLS, because some providers 
will block port 25. AFAIK there isn't a standard port for 
SMTP-over-SSL; should there be / is there one?

BTW, such a service can be found at:
   http://www.tuffmail.com/
(not affiliated; just a very happy customer)

On Jun 1, 2005, at 6:18 AM, Tim Bray wrote:

> One of the nice things about working for a big company (Sun in my 
> case) is that they operate this big honking SMTP server that's 
> password+SSL-protected and listens on a bunch of different ports.  If 
> I were an indie again, I'd cheerfully pay a couple bucks a month for 
> such a service; put a choke on it allowing no more than a couple 
> hundred emails a day and that would stop the spammers.
>
> I used to do the ssh trick too but this is less work and I've never 
> been locked out of it once in the last year and a bit, and I'm a 
> serious gipsy.
>
> So, is there really a problem? -Tim
>
>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 01 Jun 2005 17:02:15 +0200

> SMTP-over-SSL; should there be / is there one?

There is:
$ grep smtp /etc/services 
smtp		25/tcp		mail
ssmtp		465/tcp		smtps		# SMTP over SSL

Not that the latter is an 'official' port according to IANA, but
everything that does SMTP-SSL uses it.

But as you are only _submitting_ mail, use the Message Submission
standard (RFC2476) which goes over 587 and defaults to requiring
authentication and SSL.

Greets,
 Jeroen

---

From: "RL 'Bob' Morgan" <[email-masked]>
Date: Wed, 1 Jun 2005 08:10:00 -0700 (PDT)

> +1 - I use SSL'd IMAP and SMTP, and have never had problems yet.

And along the lines of mass deployability, we are about to require SMTP 
authentication (with TLS or Kerberos protection) for all submission to our 
main submission server, for use by all our 200,000 users (though in 
fairness many of them do submissions to dept-level servers or use central 
timeshare boxes hence don't worry about config).  We've heard of other 
institutions doing similarly.  So we are making the bet that this is 
usable by all kinds of users.  Discoverability would be a big help, 
though, so I'd really like to see this draft move forward.

On the other side of the question, it seems to me increasingly the case 
that ISPs don't provide SMTP/submission service by default to all those 
clients that they happen to provide IP connectivity service to; probably 
for the same reasons that open SMTP relaying is a thing of the past.  The 
use of tunnels and authenticated SMTP, and web-based email access, lets 
them make this choice.  So, bouncing the rubble on this point, I don't 
think a standard for discovery of topology-local submission service is 
particularly interesting.

> Note that this may not be the case with TLS, because some providers will 
> block port 25. AFAIK there isn't a standard port for SMTP-over-SSL; 
> should there be / is there one?

No, there shouldn't.  Separate-port SSL is simply a bad idea, especially 
bad for SMTP where public availability of services is important for 
providing universal email connectivity (for better and worse, of course). 
We are stuck with it for HTTP, but for all services for which STARTTLS is 
defined it's much the better approach.  The reason in a nutshell is that 
in-band negotiation is better than port-probing.

   - RL "Bob"

---

From: Larry Masinter <[email-masked]>
Date: Wed, 01 Jun 2005 08:29:56 -0700

I will remind you that for "finding HTTP proxy" configuration,
it was (is?) not unusual for people to deploy complex,
dynamic JavaScript functions that would, for every URL,
look at the URL and decide which proxy to use based on the
URL, the local connection spot, etc.

I don't know how widely this kind of nonsense is required
or deployed these days (some of the cases were just
because of bad network planning), but I wonder what the
high end of the requirements might be.

For finding an "appropriate" SMTP relay, might it not
depend on the source email address? I mainly use two
different email accounts (personal and business), and my
choice of SMTP relay depends both on the "From" address 
as well as my current network location. (I don't want
to use my ISP's SMTP server for internal business mail).

I'm not sure a simple SLP or DHCP service would do.

Larry

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 01 Jun 2005 13:59:00 -0400

> I'm not sure a simple SLP or DHCP service would do.

Discovery by email address is the right answer in 80% of the cases, but
that other 20% is pretty important. Some of the places where discovery by
addr falls down can be pretty painful too.

Let's say that you are telco.net with millions of [email-masked]
addresses, all across the planet. It's pretty difficult to map that single
domain to ~closest/best submission server in a way that is meaningful for
dial-up users around the planet, and I can get away with saying that
because I've tried to develop a formula for it. In the best case it will
fall down only some of the time... Most orgs (the 80%) have a handful of
mail servers and users (~90% of orgs are small businesses with less than
100 employees) but that other 20% case is really frightening. They need
some other option.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 01 Jun 2005 14:36:20 -0400

On 6/1/2005 1:59 PM, Eric A. Hall wrote:

> domain to ~closest/best submission server in a way that is meaningful

I should adde here that it's difficult to do this with submission servers
but nearly impossible with retrieval servers. Figuring out that random
user who authenticated on a dial-up pool should be directed towards a
specific pop/imap server on the other side of the planet is not at all
straightforward.

Overcoming the need for static configuration of POP/IMAP servers should be
the real goal--submission is a breeze by comparison.

---

From: Keith Moore <[email-masked]>
Date: Wed, 1 Jun 2005 14:53:37 -0400

> because I've tried to develop a formula for it. 

Divide the problem into two parts:

1. identifying submission servers that will accept mail for someone who 
    has permission to use that sender address
2. identifying a sufficiently close subset of the servers obtained in #1

#1 is reasonable to solve with SRV records 
#2 can be solved with a SONAR-like service that accepts a list of 
server addresses along with some criteria for "close" (e.g. round
trip latency, bandwidth, packet loss) and returns a list of reachable 
servers with proximity metrics

One nice thing about splitting the two parts of the problem is that
both parts then become reusable for other apps, though the other
apps may have different selection criteria or use the information
in different ways.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 01 Jun 2005 14:59:20 -0400

> servers with proximity metrics

That's pretty much what the currently formula says [see section 4.2 of
http://www.ehsco.com/misc/I-Ds/draft-hall-email-srv-01.txt]

That's also what the retrieval algorithm mostly does [see section 4.1] but
it's a lot more fragile because retrieval mail-stores aren't nearly as
location independent as transient submission servers. That's what I was
actually referencing in the above comment.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 1 Jun 2005 21:22:22 +0200

> ssmtp		465/tcp		smtps		# SMTP over SSL

Interesting; I get
   mnot-laptop:~> grep " 465/tcp" /etc/services
   urd             465/tcp     # URL Rendesvous Directory for SSM
on Mac OS 10.3, and that's also what's in 
http://www.iana.org/assignments/port-numbers.

> standard (RFC2476) which goes over 587 and defaults to requiring

Ah, thanks for that reference! /etc/services and IANA only list 
"Submission", which isn't terribly descriptive. I don't see where it 
defaults to SSL, but it does allow for it.

I think this is the way to go. I understand and agree with the 
arguments about not using a port to negotiate a secure transport, but I 
don't think that's what's happening here; ISPs are blocking port 25 
because people are abusing MTAs. By having a separate port for MSAs, 
ISPs have local control over their network, while still allowing 
appropriate functionality on a separate port. The fact that it's SSL'ed 
(etc.) is orthogonal.

Using your own MSA rather than a local one when you are roaming is far 
preferable; you have an ongoing relationship with it, so you know its 
policies on message size, exploding lists, etc. In contrast, you don't 
know any policies on a local, unknown server until you run up against 
them. They also don't know you, so they're less likely to deliver what 
they consider to be questionable mail on your behalf.

---

From: John C Klensin <[email-masked]>
Date: Wed, 01 Jun 2005 15:47:41 -0400

> I'm not sure a simple SLP or DHCP service would do.

Larry,

There are lots of edge cases and cute heuristics that can be
used to figure out when they apply and what values they would
have, but there are, IMO, only three basic cases:

(1) The user knows enough to statically configure the client to
simply make the right choice.  This is the case for all of the
tunnels and other mechanisms for getting back to a "home" server
as well as our classic (pre-open-relay-restrictions)
environment.   The tunnels may or may not need special
configuration, but the mail client does not.

(2) The choice of server is based on the user's semantic or
mail-abstraction environment.  Whether that is the domain name
or the source email address or something else, some solution
along the lines of Eric's (and my) SRV draft addresses these
cases as far as the network is concerned.   Where the domain
name that goes into the DNS query is a local matter.

(3) The choice of server is based on the user's current location
in the network, i.e., on the network environment rather than
that of the mail abstraction.  With all respect to Keith's
concerns about encouraging middleboxes, many of us have found
ourselves seeking the SMTP relay that is network-nearest for
20-odd years, not because of control issues but because of
simple performance over sometimes-dubious links.  There are
still parts of the world where that is important.  Something
DHCP-ish may offer some hope there, as, for some situations, a
list of servers and a way to determine which one is most
available at a given time.

      john

---

From: Keith Moore <[email-masked]>
Date: Wed, 1 Jun 2005 16:33:52 -0400

> still parts of the world where that is important.  

I'm presuming that you were "in the loop" on such decisions, and
had reason to believe you could trust the SMTP relay on those
occasions where you chose to use it.  That's a bit different from
a situation where user agents blindly trust whatever the DHCP
server says to use.

> Something

of these only (3) can be suitable as a default that can be chosen
without the user's explicit consent.  it is also necessary to support
(1) for testing, for use with legacy servers, or to support the
case where a recipient has his mail forwarded from one address
to another but still wishes to use the first address as a return
address.

case (2) is complex, as the network will often provide the "wrong"
information for that particular user and his address.  so it needs 
to be explicitly enabled by the user and not used as a default.   
the choice of whether to use the DHCP-supplied server (or the 
server recommended by the local network via whatver means) 
might even depend on which network the user is connected
to at the moment.  (e.g. use the DHCP server's recommendation 
if I'm at work, my explicit settings if I'm at home, otherwise use 
the DNS).   this problem isn't unique to SMTP, of course, and it
might help to explain why DHCP isn't used more often to configure
apps.

Keith

---

From: Ned Freed <[email-masked]>
Date: Wed, 01 Jun 2005 17:26:23 -0700 (PDT)

> URL, the local connection spot, etc.

Indeed. This has reached absurd heights here at Sun. There are all sorts of
different .pac files floating around that purport to optimize different
things in regards to proxies. There's even a gizmo run by someone that
measures response times for various proxies and dynamically generates
.pac files reflecting these measurements.

I've never found any of this to actually work all that well as compared to a
simple "always use proxy foo, manually switch is down" setting. What I've found
is that these things improve things on average but also increase the number of
bad response outliers significantly. It doesn't help that broswer support for
this stuff is weak and buggy.

> high end of the requirements might be.

There may be an actual requirement lurking under all this, but it is hard to
separate away from the "engineers at play".

				Ned