Subject: [Asrg] 0. General - Administrative - for M. Wild

From: Yakov Shafranovich <[email-masked]>
Date: Wed, 27 Aug 2003 09:24:24 -0400

This message is intended for M Wild ("Mike"):

I have been trying to send an email reply to you but unfortunately it is 
not going through due to the following error:

450 Client host rejected: cannot find your hostname, [xx.xx.xx.xx]

I do not have an rDNS address and use the IP address in the HELO command 
for SMTP. Apparently your server is not accepting that. Please let me know 
an alternative way to contact you.

Yakov 

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Brad Knowles <[email-masked]>
Date: Wed, 27 Aug 2003 22:30:55 +0200

At 9:24 AM -0400 2003/08/27, Yakov Shafranovich wrote:

>  I do not have an rDNS address and use the IP address in the HELO
>  command for SMTP. Apparently your server is not accepting that.
>  Please let me know an alternative way to contact you.

	Hmm.  Grist for the BCP mill.

-- 
Brad Knowles, <[email-masked]>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

---

From: David Wilson <[email-masked]>
Date: 28 Aug 2003 09:00:30 +0100

> an alternative way to contact you.

RFC 2822 specifically allows domain literals in the EHLO/HELO command.

RFC 1123 Section 5.2.5 specific forbids refusing messages if the domain
name in HELO (predating SMTP extensions, there is no mention of EHLO)
"fails verification".

There was general discussion some years ago about the issue of:

- accepting SMTP connections when there is no rDNS for the calling IP
address.

- accepting SMTP connections if the rDNS hostname does not have an A
record which contains the calling IP address.

At that stage there were enough legitimate sites which fail either of
these tests to make rejection on these grounds unacceptable for a
reasonable service.

So, in my opinion M Wild's MTA is not acting reasonably.

cheers

David Wilson                             [email-masked]
Isode Limited                            Tel: +44 (0) 20 8783 2961
http://www.isode.com

---

From: Brad Knowles <[email-masked]>
Date: Thu, 28 Aug 2003 22:29:02 +0200

At 10:54 AM -0400 2003/08/28, Richard Rognlie wrote:

>  And according to RFC2821, putting an IP as the HELO *IS* valid.
>  it is supposed to be an FQDN or an IP (in those cases where a machine
>  has no name [e.g. a dynamic dialup IP])  However, I don't accept
>  mail from HELO ip.ad.re.ss unless it's a LOCAL IP.

	Since RFC2821 explicitly prohibits you from checking the domain 
literal claimed in EHLO/HELO and rejecting the message if the reverse 
DNS doesn't match that domain, then I submit that your tests are at 
least violating the spirit of RFC2821 in a similar manner.

	But this is something that needs to be put in an updated BCP.  We 
should probably also talk to the appropriate IETF group about an 
update/clarification of RFC2821 to be more explicit on these issues.

---

From: Richard Rognlie <[email-masked]>
Date: Thu, 28 Aug 2003 10:54:28 -0400

-- 
 /  \__  | Richard Rognlie / Oracle Prophet / Gamerz.NET Lackey
 \__/  \ | http://www.gamerz.net/rrognlie/    <[email-masked]>
 /  \__/ | I can only please 1 person per day.  Today is not your day.
 \__/    | Tomorrow doesn't look good either.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 28 Aug 2003 17:04:04 +0200

> name in HELO (predating SMTP extensions, there is no mention of EHLO)

RFC 1123 was written well before spam was considered a problem. RFC 2821
also explicitely forbids examination of the mail headers for other
purposes than adding a Received line, but lots of people reject messages
based on the content. 

I think that the receiver doesn't have any obligation to accept mails
(except to "postmaster") and can reject messages on any criteria the
user deems reasonable. 

> these tests to make rejection on these grounds unacceptable for a

Still, I don't think there are many legitimate sites which don't have an
A record.  Requiring the sender to send a FQDN which resolves to the
sender's IP address doesn't seem unreasonable to me (even for dynamic
IP-Addresses, you can use dyndns.net or a similar service).  In fact,
looking at my log files this seems to be a very good indicator of
legitimate mail servers (I checked several weeks of logs some time ago
and only found one legimitate server which identified itself with an
unresolvable name (I think the box is NATted). I am planning to
implement this test, but I didn't get around to it yet. 

	hp

-- 
   _  | Peter J. Holzer    | Humor ohne Emoticons ist trockener Humor.
|_|_) | Sysadmin WSR       | 
| |   | [email-masked]         |	-- Toni Grass in aip
__/   | http://www.hjp.at/ |

---

From: "Hector Santos" <[email-masked]>
Date: Thu, 28 Aug 2003 15:34:05 -0400

David,

We found rDNS checking on HELO/EHLO to be unreliable due to
mis-configuration of smtp servers, in particular those systems who prepare a
send-only or routing server,  which from my last reading of the RFC (a few
years back), need to be prepare as sub-domains.   Because they are not, it
is not possible to do reliable checking.

Recently, we added logic to check for the bracket DOT format,  i.e,.
HELO/EHLO [X.X.X.X]

We found those servers using this format to be spammer servers and they are
using it incorrectly, providing the literal IP without the brackets, i..e,
HELO/EHLO X.X.X.X

So we reject the HELO/ELHO state when

a) The literal IP does not have brackets, or
b) The provided bracket IP does not match the connecting peer IP.

We have rejected on average about 125 per day using this scheme.

Incidentally, before this logic was added,  the average about 80 attempts
per day.  Hence, the rejection is causing some senders to try again more
often.  We are sending a 5XX response code (permanent error, don't try
again) but some are ignoring it of course. :-)

----

Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
http://www.santronics.com

---

From: Yakov Shafranovich <[email-masked]>
Date: Thu, 28 Aug 2003 17:30:32 -0400

[This is relevant to the BCP area, subject has been changed.]

---

From: Yakov Shafranovich <[email-masked]>
Date: Thu, 28 Aug 2003 17:30:32 -0400

---

From: Richard Rognlie <[email-masked]>
Date: Thu, 28 Aug 2003 21:35:43 -0400

> again) but some are ignoring it of course. :-)

Based on the logs I'm maintaining as part of my DRIP Milter development,
I see ~20% of all HELO/EHLO requests are either...

a.  an IP.  But worse... it's *MY* IP. not theirs.
b.  An unqualified "hostname"  (e.g.  BIZ3)

I am preparing to add an option to the DRIP milter which will treat
either of those cases (IP (with or without []s) that does not match
the connecting IP) and unqualified hostname as invalid DRIP records
(vs. unknown DRIP host as the probably do now)

I'm curious about Brad's contention below...

Section 3.2 states

    In the EHLO command the host sending the command identifies itself;
    the command may be interpreted as saying "Hello, I am <domain>" (and,
    in the case of EHLO, "and I support service extension requests").

Section 3.6 states

    The domain name given in the EHLO command MUST BE either a primary
    host name (a domain name that resolves to an A RR) or, if the host
    has no name, an address literal as described in section 4.1.1.1.
            
Section 4.1.1.1 states

    The argument field contains the fully-qualified domain name of
    the SMTP client if one is available.  In situations in which the
    SMTP client system does not have a meaningful domain name (e.g.,
    when its address is dynamically allocated and no reverse mapping
    record is available), the client SHOULD send an address literal
    (see section 4.1.3), optionally followed by information that will
    help to identify the client system.

Section 4.1.4

    An SMTP server MAY verify that the domain name parameter in the
    EHLO command actually corresponds to the IP address of the client.
    However, the server MUST NOT refuse to accept a message for this
    reason if the verification fails: the information about verification
    failure is for logging and tracing only.

It says we can't use the IP as a basis for rejection.  I'm wondering 
why not...?   Yes, if only an IP is presented, I can envision that it
was presented by a machine behind a NAT... so it presents 192.168.x.y
but the connecting IP is God knows what...   But do I want to accept
that mail?

And certainly... if a machine presents *MY* IP addr as the connecting
info... but the connecting IP does not have anything to do with my
network... Bzzzzt!  Next player!

or when they say "EHLO gamerz.net"  but again are NOT on my network.
Bzzzzt!

The DRIP logs have been very interesting in that regard.  I'll post 
them to ftp://ftp.gamez.net/pub/dripmilter.log shortly.

Richard

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 29 Aug 2003 17:34:07 +0200

[changing the subject]

On 2003-08-28 23:59:29 +0200, Brad Knowles wrote:
> At 5:04 PM +0200 2003/08/28, Peter J. Holzer wrote:
> 
> > IP-Addresses, you can use dyndns.net or a similar service).
> 
> 	Check the recent traffic on NANOG.  Because of stupidity on the 
> part of AOL, they've been discussing this subject intensively.  I've 
> been tagging 75-90% of the recent messages as input for the BCP 
> review.

I think you mean the thread "Fun new policy at AOL" starting at
http://www.cctec.com/maillists/nanog/current/threads.html#05061

This is something different. The thread started because AOL blocks
"residential" addresses, which is essentially the same as a DUL. 

It was also mentioned in the thread that AOL blocks connections from IPs
without rDNS. (see http://postmaster.info.aol.com/standards.html for
AOL's policy).

AFAICS, checking the argument to HELO wasn't even mentioned in the
thread.

I'll try to summarize how HELO can be checked and the implications of
various methods.

1) Formal correctness. 

    RFC 2821 specifies that the parameter must be an FQDN or an address
    literal. So you can check against regexps or a grammar.

    This will disallow unqualified host names and random garbage. 
    Unqualified host names are often (legitimately) used by broken MUAs. 
    I think I have also seen it from misconfigured Exchange servers. 
    Some viruses/worms (Sobig) and some spammer software also uses unqualified
    host names - this would be blocked.

2) Check whether the parameter corresponds to the sender IP address.
    I.e., if the parameter is an FQDN, look up its A record(s) and
    compare to the sender IP, if it is an address literal, compare it to
    the sender IP.

    This would in addition catch all the software which lies about its
    own address. Examples are again MUAs (at least some use either the
    server address or the domain name of the sender) and lots of spammer
    software which sends "HELO yahoo.com" or similar.

3) Require the parameter to be an FQDN and check whether its A record
   matches the sender IP.

    This will additionally block all senders which don't have an FQDN or
    don't know it (which currently seems to be true for a lot of open
    proxies, but may change quickly). 

4) Require the parameter to be an FQDN and check whether the PTR record
   of the sender IP matches the parameter.

    This will block all addresses for which rDNS isn't available, and
    those for which rDNS gives a different name than the "canonical"
    name of the host. This includes lots of cable/dsl customers.

RFC 2821 allows checking the HELO parameter only for logging and tracing
purposes. I think that the decision which mails to accept and which to
reject must be made by the administrator of each system (who is
responsible to his employer and/or his customers) not by some RFC. 

I also think that rejecting mail because it fails a single test (like no
rDNS record or being listed in a single BL) is not good practice (I am
still guilty of that, because that's how sendmail handles BLs, but I am
trying to move to more robust schemes without a single point of
failure).

That said, I think that checking the HELO parameter gives valuable
results for a mail relay which recieves mail only from other servers
(MUAs are way too broken in that regard and probably won't be fixed).

I see very little legitimate mail in my log files which doesn't pass
tests 1) to 3). For the few cases which would break on 3), I would
consider it reasonable for the administrator on any legitimate server to
get an FQDN and a matching A record. This is even possible for dynamic
IP addresses via dyndns.net and similar services. 

Test 4 OTOH cannot be passed by many people with "residential" or
"consumer" cable, dsl or dial-up lines, since providers often don't
provide rDNS mapping, or inconsistent mapping (forward and reverse
lookup don't match), and in any case the "canonical" name of the
computer isn't the unwieldy automatically generated name in the rDNS
record (e.g., my laptop (chthon.hjp.at) has IP address 62.99.236.100
when I'm at home, but the PTR for that address points to
62-99-236-100.C-VTreustrasse.Xdsl-line.inode.at.)

(and BTW, last time I looked the ASRG mailing list server wouldn't have
passed test 4 either)

So for a BCP document I would recommend:

1) Combine different tests (for example by summing scores like
   SpamAssassin does) and reject only if some threshold is exceeded.

2) Of the 4 tests I outlined, give the first 3 a high score, and the
   fourth a low to medium score.

> > unresolvable name (I think the box is NATted).
> 
> 	I'm NAT'ed.  Many people on NANOG appear to be in similar 
> situations, or have run into them frequently.

At least in the case I cited the admin knows the address of the gateway
and could easily arrange for his server to send a HELO parameter
resolving to the NATed address. He probably would if I started to block
his mail, but as long as it doesn't matter, why should he fix his
configuration?

Maybe I am overlooking something, but I cannot think of any situation
where tests 1-3 cannot be met, even for NAT and dynamic IPs.

Test 4 is different, because it requires collaboration of your ISP.
"Get a clueful ISP" is not always an option.

	hp

---

From: Brad Knowles <[email-masked]>
Date: Fri, 29 Aug 2003 20:25:20 +0200

At 8:23 PM -0400 2003/08/28, Yakov Shafranovich wrote:

>  If you need to get in touch with the IETF, please let Paul or me know
>  so we can push it on to the IETF through the regular channels.

	I believe that we should plan on updating the relevant RFCs, or 
at least coordinating the updates for them, in addition to the BCPs.

	I'll let you folks know when we've gotten there.

---

From: Brad Knowles <[email-masked]>
Date: Fri, 29 Aug 2003 20:27:00 +0200

>  or when they say "EHLO gamerz.net"  but again are NOT on my network.

	Both good subjects to be discussed for the BCPs.

---

From: Brad Knowles <[email-masked]>
Date: Sat, 30 Aug 2003 17:38:50 +0200

>      host names - this would be blocked.

	Many hosts in the world are configured (misconfigured?) to have 
unqualified hostnames.  Some sites even have an explicit policy that 
all hostnames will be unqualified.

>      software which sends "HELO yahoo.com" or similar.

	I have over a dozen different e-mail addresses that I use.  Only 
one of them might potentially correspond to the server that I use 
over an ssh tunnel as my outbound mail relay.

	If I were to host my own outbound mail relay here in the house, 
because I am behind a NAT you might see my server identify itself as 
10.0.1.5, when the real external IP address is something completely 
different.

>      proxies, but may change quickly).

	This would require a change to RFC 2821.  Moreover, it would 
break anyone who has an IP address assigned to them (maybe in PI 
space), but does not necessarily have an FQDN.  See above items 1 & 2.

>      name of the host. This includes lots of cable/dsl customers.

	And lots of customers or service providers that run their own 
mail servers on valid public IP space, but who don't properly 
understand the concept of reverse DNS or how to implement it properly.

	I'm perfectly happy to mention names of specific individuals 
and/or providers that I have personally encountered, if anyone wants. 
I'd prefer to keep this discussion above the level of name-calling, 
however.

>  trying to move to more robust schemes without a single point of

	Agreed.  A multivariate analysis should be applied before taking 
such a decision.

>     SpamAssassin does) and reject only if some threshold is exceeded.

	Agreed.

>     fourth a low to medium score.

	The benefits and drawbacks of each specific criteria should be 
discussed, with recommendations for higher or lower score values, 
based on expected probability of occurrence and probabilities and 
costs of false positive and false negative matches.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 30 Aug 2003 23:19:28 +0200

> all hostnames will be unqualified.

The name sent in the HELO parameter is not necessarily the same as the
one returned by the "hostname" command (or equivalent). A mail server
may have a hostname of "rumpelstilzchen" and still identify itself as
"mail-out.example.com" in the HELO command.

RFC 2821 requires the parameter to be either an FQDN or an address
literal. A client which sends an unqualified hostname is in violation of
the RFC without any good reason. (broken software and lazy
administrators are not good reasons, IMHO)

(Please note that I'm talking about servers sending to other servers
over the internet. User agents talking to their servers or one
department's server talking to another within a corporation's intranet
are out if scope of this group IMHO).

> over an ssh tunnel as my outbound mail relay.

I wasn't talking about mail addresses, but about IP adresses.

> 10.0.1.5, when the real external IP address is something completely 

Any reason why it must identify itself as [10.0.1.5]? Why not with the
external IP address or as bradknowles.dyndns.net? 

> 	This would require a change to RFC 2821. 

That would be nice, but it's not required. We are working on a BCP for
mail administrators here, not on a technical specification for the SMTP
protocol. 

So we can recommend that a sending MTA SHOULD always send an FQDN which
resolves to its IP address, because a server MAY interprete a failure to
do so as an indication that the sender is not a properly configured MTA
and reject mails while RFC 2821 can still allow address literals for
situations where they are appropriate.

> Moreover, it would 

You can get an FQDN for free in a few minutes. If want to run a properly
configured mail server that is the least of you worries.

(Of course this is also true for spammers - they might adapt quickly if
people start to check helo parameters)

> understand the concept of reverse DNS or how to implement it properly.

One could argue that if they don't understand DNS, they shouldn't run a
mail server. In fact that was the reasoning behind the rDNS checks in
Klaus Assmann's anti-spam hacks for sendmail: If they can't even get DNS
right, how can you expect them to close an open relay (which wasn't easy
at that time)?

I specifically mentioned cable/dsl customers because lack of
understanding isn't their problem. They cannot set up rDNS themselves
because they don't own their namespace in in-addr.arpa. and can't buy it
(unlike a domain name). Changing to a provider which does offer a fixed
IP address and rDNS entries or even delegation for sub-/24 nets (as in
RFC 2317 or similar) is often not possible (there may not be one in your
area, or it may be prohitively expensive).

> costs of false positive and false negative matches.

Agreed. My (entirely subjective) recommendation was based on the log
files of my private server and from looking at Received headers of many
spam- and non-spam messages (usually for other reasons). I don't have
hard figures at this time, only a gut feeling. I plan to switch to
qpsmtpd at the WSR next week, which will write the complete SMTP
conversation (except the contents of the mail, of course) to a log file,
so I will have a bit better statistics in a few weeks. (still only for
about 150 users or so - the results of a large ISP would be
interesting).

	hp

---

From: Brad Knowles <[email-masked]>
Date: Sun, 31 Aug 2003 02:09:34 +0200

>  administrators are not good reasons, IMHO)

	I don't see this requirement in 2821:

4.1.1.1  Extended HELLO (EHLO) or HELLO (HELO)

    These commands are used to identify the SMTP client to the SMTP
    server.  The argument field contains the fully-qualified domain name
    of the SMTP client if one is available.  In situations in which the
    SMTP client system does not have a meaningful domain name (e.g., when
    its address is dynamically allocated and no reverse mapping record is
    available), the client SHOULD send an address literal (see section
    4.1.3), optionally followed by information that will help to identify
    the client system.  y The SMTP server identifies itself to the SMTP
    client in the connection greeting reply and in the response to this
    command.

	Note the qualification of "if one is available."  Also note that 
there is no MUST assigned to this usage.  The client SHOULD otherwise 
use the IP address literal, but again this is not a requirement.  The 
address literal is defined in section 4.1.3:

4.1.3 Address Literals

    Sometimes a host is not known to the domain name system and
    communication (and, in particular, communication to report and repair
    the error) is blocked.  To bypass this barrier a special literal form
    of the address is allowed as an alternative to a domain name.  For
    IPv4 addresses, this form uses four small decimal integers separated
    by dots and enclosed by brackets such as [123.255.37.2], which
    indicates an (IPv4) Internet Address in sequence-of-octets form.

	Note that the RFC explicitly states that the IPv4 address should 
be in square brackets.  Which is precisely the kind of behaviour that 
I believe you said that you were refusing.

>  external IP address or as bradknowles.dyndns.net?

	If it's behind a NAT, how would it know the external DNS name? 
If that IP address on the NAT device is dynamically assigned and the 
machine is not an intelligent host running software capable of 
updating a dynamic DNS record (as 99.999% of all NAT/router devices 
are almost certainly going to be), then how would the internal host 
know what this external DNS name is?

>  mail administrators here, not on a technical specification for the SMTP

	You are correct, we are working on a BCP, and we can go beyond 
the recommendations in the RFCs.

>  people start to check helo parameters)

	Indeed, a lot of spammers do already register valid domains, then 
throw them away in hours or days, after they are done using them.

>  mail server.

	I would like to be able to make such an argument.  I really 
would.  Indeed, I could even see making such an argument part of a 
BCP, but only if the recommended practice did *NOT* involve rejecting 
the connection outright just because they did not have reverse DNS 
properly configured.

>                In fact that was the reasoning behind the rDNS checks in
>  Klaus Assmann's anti-spam hacks for sendmail

	And that's precisely the sort of thing that I believe we should 
recommend against.

	I know, I know.  I used to do it myself.  But my views have 
recently changed.

>                                                         (still only for

	My subjective views have been based on my experience as the Sr. 
Internet Mail Administrator for AOL from '95-97 (up to five million 
users), and the Sr. Systems Architect for Belgacom Skynet from 
'98-2001 (peak of ~one million users).

	I'm not sure if we've got anyone on the list with current 
experience and access to log data from such large ISPs, but I do know 
that we have some people on the list with current experience and 
access to log data from ISPs in the six-figure range.  I would be 
very interested to see their data.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sat, 30 Aug 2003 22:16:23 -0400

At 09:04 PM 8/30/2003, Steven F Siirila wrote:
>On Sun, Aug 31, 2003 at 02:09:34AM +0200, Brad Knowles wrote:
> >
>We have seen about 1,200,000 different IP addresses connecting to us in the
>past six months and we block about 1,000,000 SMTP connections on our MX
>servers every week on average.  What other stats would be of interest?

This would also be relevant to the analysis area. We are definitely 
interested in all the data we can get.

---

From: Brad Knowles <[email-masked]>
Date: Sun, 31 Aug 2003 03:49:24 +0200

>  servers every week on average.  What other stats would be of interest?

	What's your breakdown of why those connections are blocked?  How 
many unique blocks are due to which blacklist?  How many unique 
blocks are from lack of rDNS?  How many unique blocks are there by 
domain-based blacklist?  How many unique blocks are there due to 
other techniques?

	I assume that once you have a positive response on one of those 
checks, you reject the connection and you don't bother trying the 
other checks as well.  Could you re-process your data to apply all 
checks to all IP address/envelope sender/recipient combinations, to 
see where certain blocks might overlap?