Subject: Re: Yahoo Mail Filtering
From: Hector Santos <[email-masked]>
To: Laura Atkins <[email-masked]>
Cc: SMTP Interest Group <[email-masked]>
Date: Sat, 31 Jul 2010 19:10:05 -0400
List-Id: <ietf-smtp.imc.org>

Very odd. I don't see how yahoo's "shred" logic or deferrals promoting 
10+ retries before it finally accepts, and they all seem to finally 
make it, helps anyone.

Thanks for the link.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Laura Atkins wrote:
> 
> On Jul 31, 2010, at 11:22 AM, Hector Santos wrote:
> 
>> J.D. Falk wrote:
>>
>>> It's not the wikipedia definition of greylisting, but they do use 4xx replies to control the flow of inbound mail.
>> Ok, now I am starting to see 45x "grey listings", here is snap shot of a mail queue:
> 
> <snip>
> 
>> There are multiple retries spread across different machines.
>>
>> Do you think they need the same exact IP or they support a class C
>> machine distribution?
> 
> Yahoo isn't greylisting.
> 
> http://blog.wordtothewise.com/2007/11/greylisting-that-which-yahoo-does-not-do/
> 
> "Yahoo is not keeping track of the mail it rejects and is not reliably allowing email through on the second attempt. There are a couple reasons why Yahoo is deferring mail.
> 
> 	• Shedding load in a generic and non-specific way.
> 	• Shedding load by temporarily refusing mail from specific IPs.
> In the first case, the shedding of load means nothing more than Yahoo is shedding load. There is not really anything the sender can do to compensate for this, nor is there any thing the sender is doing (except possibly send mail to Yahoo at the same time as the rest of the world) to precipitate the blocking."
> 
> laura 
> 

Subject: Re: Yahoo Mail Filtering
From: Laura Atkins <[email-masked]>
Date: Sat, 31 Jul 2010 14:13:02 -0700
Cc: SMTP Interest Group <[email-masked]>
To: Hector Santos <[email-masked]>
Sender: [email-masked]

On Jul 31, 2010, at 11:22 AM, Hector Santos wrote:

> 
> J.D. Falk wrote:
> 
>> It's not the wikipedia definition of greylisting, but they do use 4xx replies to control the flow of inbound mail.
> 
> Ok, now I am starting to see 45x "grey listings", here is snap shot of a mail queue:

<snip>

> There are multiple retries spread across different machines.
> 
> Do you think they need the same exact IP or they support a class C
> machine distribution?

Yahoo isn't greylisting.

http://blog.wordtothewise.com/2007/11/greylisting-that-which-yahoo-does-not-do/

"Yahoo is not keeping track of the mail it rejects and is not reliably allowing email through on the second attempt. There are a couple reasons why Yahoo is deferring mail.

	� Shedding load in a generic and non-specific way.
	� Shedding load by temporarily refusing mail from specific IPs.
In the first case, the shedding of load means nothing more than Yahoo is shedding load. There is not really anything the sender can do to compensate for this, nor is there any thing the sender is doing (except possibly send mail to Yahoo at the same time as the rest of the world) to precipitate the blocking."

laura 

-- 
Laura Atkins
Word to the Wise			"The Deliverability Experts!"
Direct: 650 678-3454		Fax: 650 249-1909
AIM: wttwlaura			YIM: wttw_laura
Delivery blog: <http://blog.wordtothewise.com/>

Date: Sat, 31 Jul 2010 14:22:11 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "J.D. Falk" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Yahoo Mail Filtering
Sender: [email-masked]

J.D. Falk wrote:

> It's not the wikipedia definition of greylisting, but they do use 
> 4xx replies to control the flow of inbound mail.

Ok, now I am starting to see 45x "grey listings", here is snap shot of 
a mail queue:

250-Begin Mail Queue Manager
250-att: 4, to: [email-masked], next: 07/31/2010 14:24:33
250-att: 6, to: [email-masked], next: 07/31/2010 14:22:08
250-att: 4, to: [email-masked], next: 07/31/2010 14:28:11
250-att: 4, to: [email-masked], next: 07/31/2010 14:28:20
250-att: 3, to: [email-masked], next: 07/31/2010 14:25:42
250-att: 3, to: [email-masked], next: 07/31/2010 14:24:56
250-att: 2, to: [email-masked], next: 07/31/2010 14:15:27
250-att: 2, to: [email-masked], next: 07/31/2010 14:15:48
250-att: 2, to: [email-masked], next: 07/31/2010 14:15:04
250 End Mail Queue Manager

Except one, all yahoo.

There are multiple retries spread across different machines.

Do you think they need the same exact IP or they support a class C
machine distribution?

I might have to change the software to use the same first machine to 
retry when it sees a 451.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 21 Jul 2010 14:07:37 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "J.D. Falk" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Yahoo Mail Filtering
Sender: [email-masked]

J.D. Falk wrote:

> On Jul 20, 2010, at 7:46 PM, Hector Santos wrote:
> 
> It's not the wikipedia definition of greylisting, but they do use 4xx replies to control the flow of inbound mail.

A SMTP customer I was working, provider a link showing a greylist 
transaction. From my perspective, the user spoke as it was a 
persistent  behavior.

But I have yet to see it in the 2-3 weeks of exploring this YAHOO 
issue with the customer.  So it made me think the user was 
experiencing maybe a YAHOO exploration into greylisting at the time or 
it was done on a per IP basis or something. I have not seen it in all 
our send mail attempts.

>> Absolutely, no pattern, inconsistent "DISCARDING" and delivery of mail.
> 
> Often, filters will be reacting to a system-wide pattern that 

 > an individual user can't see.  To give an overly
 > simplified and surely incorrect example, if a spammer were sending
 > "test #4" all day, then your entirely legitimate "test #4" would
 > get caught too.

> 
> That said, if the mail is being discarded due to system-wide 

 > (rather than recipient-specific) filters, I'd personally prefer
 > to see a 5xx reply instead.

Well, I prefer they follow decades of SMTP practice of mail delivery 
or bounce.  Although new semantics has been written into 5321, 
discarding mail is not recommended behavior, and IMHO, I don't mind 
mentioning the behavior smacks against US ECPA provisions for "user 
expectations."

 > But: their system, their rules.

Sure, but this random "discard" and mail delivery unknown practice, 
rule or "protocol" is costing people time and money in solving mail 
delivery issues when in fact, their are no transport or content issues 
in what is otherwise private 1 to 1 messages.

My daughter (http://jaclynsantos.com) who had a yahoo account for a 
long time has recently started gotten thousands of email because she 
is a controversial contestant on the Bravo reality TV show, "Work Of 
Art: Next Great Artist" and now what was once a reliable occasional 
private sending mail to her by me or her mother, is now not getting 
through - its random.

So I reminded her that she always had the jaclynsantos.com email 
domain and server running with her Wildcat! based web site and can 
avoid YAHOO altogether.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: "J.D. Falk" <[email-masked]>
Subject: Re: Yahoo Mail Filtering
Date: Wed, 21 Jul 2010 10:29:48 -0600
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Jul 20, 2010, at 7:46 PM, Hector Santos wrote:

> J.D. Falk wrote:
> 
>> On Jul 20, 2010, at 8:09 AM, Hector Santos wrote:
>>> It seem so random in what seems to be always accepting a transaction but intermittently delivered or not.
>> Are you talking about the 4xx replies?  If you check the text of the reply, there's usually a URL with a bit more information about the reason.  Same with 5xx, for that matter.
> 
> I have not seen temporary rejects.  All transactions accepted with 250. However, it has been noted by others that YAHOO does employ greylisting (in some form).  But I have not seen the temporary negative reply codes.

It's not the wikipedia definition of greylisting, but they do use 4xx replies to control the flow of inbound mail.

>> If you're saying the mail is just disappearing, it's possible that your users have configured their accounts to delete suspected spam instead of putting it in the spam folder.
> 
> The issue is that its intermittent and not consistent.  Send a test, with the recipient on the phone, he gets it.  Send another test, i.e, same message with "TEST #2" as subject, it never arrives. Send "Test #3" it may or not arrive, and so on. No pattern.

I remember hearing some years ago that Hotmail's SVM filter had learned that an empty message with a subject of "test" was spam.  They've probably fixed it since then, but it's a great example of how machine learning can lead to entirely logical but unexpected results.  (I'm fairly certain Yahoo! Mail doesn't use any Microsoft software, though.)

> However, at some point, we tried the reverse, yahoo to non-yahoo, which arrived each time and then we try sending test messages again. For a moment there, it appeared to work, so we presumed there was a YAHOO user based "auto-white listing" to addresses the user sends mail to.  I jumped on this theory because we have that feature - local users can build an auto-white list just by sending mail to to their "social/business" network.
> 
> But then we try sending "test #4", "test #5" and so on and they done ever arrive or go into any folder.
> 
> Absolutely, no pattern, inconsistent "DISCARDING" and delivery of mail.

Often, filters will be reacting to a system-wide pattern that an individual user can't see.  To give an overly simplified and surely incorrect example, if a spammer were sending "test #4" all day, then your entirely legitimate "test #4" would get caught too.

That said, if the mail is being discarded due to system-wide (rather than recipient-specific) filters, I'd personally prefer to see a 5xx reply instead.  But: their system, their rules.

> I really hope someone with the yahoo story can explain it.

Like everyone else who writes anti-spam software, Yahoo! is not known for divulging their secrets.  Though I did work there for a time, everything I've written here could be gleaned from public statements and general knowledge of the techniques a large mailbox provider is likely to employ.

> I was somewhat watching to see/hear if there was a related to valid DKEY/DKIM Brownie points or something now expected by YAHOO.  But all I know so far about that is that is required by BULK mail senders.  We are not bulking here. just individual single shot messages.

DKIM isn't required to send mail, bulk or otherwise.  Yahoo! does require DKIM for non-ISPs to subscribe to their complaint feedback loop, but doesn't automatically provide any brownie points.

Single shot messages are always the hardest to diagnose, since you can't see a pattern -- and if you send enough to create a statistically significant pattern, the spam filters will notice it too.

Date: Tue, 20 Jul 2010 21:46:24 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "J.D. Falk" <[email-masked]>
CC: [email-masked]
Subject: Re: Yahoo Mail Filtering
Sender: [email-masked]

J.D. Falk wrote:

> On Jul 20, 2010, at 8:09 AM, Hector Santos wrote:
> 
>> It seem so random in what seems to be always accepting a transaction but intermittently delivered or not.
> 
> Are you talking about the 4xx replies?  If you check the text of the reply, there's usually a URL with a bit more information about the reason.  Same with 5xx, for that matter.

I have not seen temporary rejects.  All transactions accepted with 
250. However, it has been noted by others that YAHOO does employ 
greylisting (in some form).  But I have not seen the temporary 
negative reply codes.

> If you're saying the mail is just disappearing, it's possible that your users have configured their accounts to delete suspected spam instead of putting it in the spam folder.

The issue is that its intermittent and not consistent.  Send a test, 
with the recipient on the phone, he gets it.  Send another test, i.e, 
same message with "TEST #2" as subject, it never arrives. Send "Test 
#3" it may or not arrive, and so on. No pattern.

However, at some point, we tried the reverse, yahoo to non-yahoo, 
which arrived each time and then we try sending test messages again. 
For a moment there, it appeared to work, so we presumed there was a 
YAHOO user based "auto-white listing" to addresses the user sends mail 
to.  I jumped on this theory because we have that feature - local 
users can build an auto-white list just by sending mail to to their 
"social/business" network.

But then we try sending "test #4", "test #5" and so on and they done 
ever arrive or go into any folder.

Absolutely, no pattern, inconsistent "DISCARDING" and delivery of mail.

I really hope someone with the yahoo story can explain it.

I was somewhat watching to see/hear if there was a related to valid 
DKEY/DKIM Brownie points or something now expected by YAHOO.  But all 
I know so far about that is that is required by BULK mail senders.  We 
are not bulking here. just individual single shot messages.

Very odd.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Cc: [email-masked]
From: "J.D. Falk" <[email-masked]>
Subject: Re: Yahoo Mail Filtering
Date: Tue, 20 Jul 2010 15:11:05 -0700
To: Hector Santos <[email-masked]>
Sender: [email-masked]

On Jul 20, 2010, at 8:09 AM, Hector Santos wrote:

> It seem so random in what seems to be always accepting a transaction but intermittently delivered or not.

Are you talking about the 4xx replies?  If you check the text of the reply, there's usually a URL with a bit more information about the reason.  Same with 5xx, for that matter.

If you're saying the mail is just disappearing, it's possible that your users have configured their accounts to delete suspected spam instead of putting it in the spam folder.

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Tue, 20 Jul 2010 17:19:29 +0200
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
To: Hector Santos <[email-masked]>
CC: [email-masked]
Subject: Re: Yahoo Mail Filtering
Sender: [email-masked]

Hi,

I can explain how the Yahoo support operates:

Sometimes Yahoo gives to my users smtp temporary error with a hyperlink 
where to report the problem.  When I write the report, Yahoo asks for 
more information A, when I explain A, they ask for B, when I write B, 
they ask about A, and when I write A, they ask for B...

The answers are precomposed, the support is just clicking there what to 
ask as next, everytime I get a different contact person.  When I ask for 
telephone converstation, they say there is no such option...

I am nearly to give up.

Със здраве
   Дилян

Hector Santos schrieb:
> 
> Can someone here explain how YAHOO operates their mail system?
> 
> It seem so random in what seems to be always accepting a transaction but 
> intermittently delivered or not.
> 
> At one point it appears to deliver mail if the yahoo recipient had once 
> sent mail to the sender. But then that theory is shot when mail is not 
> received by the yahoo account on a 2nd or 3rd attempt.
> 
> I'm getting tired of having to service SMTP customers who are asking us 
> why their yahoo recipients are not getting their mail - sometimes.
> 
> There doesn't seem to be a clear patten.
> 
> I'm asking here to see if this is an isolated issue or more common with 
> other SMTP operators and vendors.
> 

Date: Tue, 20 Jul 2010 11:09:48 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Yahoo Mail Filtering
Sender: [email-masked]

Can someone here explain how YAHOO operates their mail system?

It seem so random in what seems to be always accepting a transaction 
but intermittently delivered or not.

At one point it appears to deliver mail if the yahoo recipient had 
once sent mail to the sender. But then that theory is shot when mail 
is not received by the yahoo account on a 2nd or 3rd attempt.

I'm getting tired of having to service SMTP customers who are asking 
us why their yahoo recipients are not getting their mail - sometimes.

There doesn't seem to be a clear patten.

I'm asking here to see if this is an isolated issue or more common 
with other SMTP operators and vendors.

-- 
Sincerely

Hector Santos
http://www.santronics.com