Subject: Filtering on raw (non-decoded) headers?

From: "Rob Mueller" <[email-masked]>
Date: Thu, 14 Feb 2008 07:39:18 +1100

We're a large cyrus installation, and I recently got an email from a 
customer noticing that cyrus doesn't correctly decode MIME encoded headers 
when doing sieve header matches.

I looked through the cyrus code, and I think the fix is actually pretty 
easy, but it occured to me that in fixing this, we completely loose the 
ability to match against the non-decoded headers. That's actually annoying 
because I know a number of people do that to try and block emails with 
particular character sets in the subject line. For instance I know people 
have script tests like this:

if header :contains "subject" "?koir-8?" { ... }

To catch and drop all emails with Russian subjects.

By fixing this bug, it would break those existing scripts, and from what I 
can see in the RFC, there's no way to actually force a match against the 
non-decoded header (eg no :raw option or something like that).

Am I missing something? Is there a way to make a header match against a 
non-decoded header?

Rob

---

From: Дилян Палаузов<[email-masked]>
Date: Wed, 13 Feb 2008 22:19:29 +0100

Hello Rob,

	I see nothing wrong to encode ordinary ASCII headers with ?koir-8? 
style. If you want to check against headers containing Cyrillic letters, 
then
	if header :contains "subject" "е" { ... }
shall do the work, where "e" is Unicode 0435.

	Със здраве,
		Дилян

Rob Mueller wrote:
> customer noticing that cyrus doesn't correctly decode MIME encoded 
> headers when doing sieve header matches.
> 
> ability to match against the non-decoded headers. That's actually 
> annoying because I know a number of people do that to try and block 
> emails with particular character sets in the subject line. For instance 
> I know people have script tests like this:
> 
> By fixing this bug, it would break those existing scripts, and from what 
> I can see in the RFC, there's no way to actually force a match against 
> the non-decoded header (eg no :raw option or something like that).
>

---

From: Дилян Палаузов<[email-masked]>
Date: Wed, 13 Feb 2008 22:25:42 +0100

Post Scriptum:

	Sometimes when I reread the mail, just before sending it out, I correct 
here and there something. Usually, when the mail is finished my keyboard 
layout is in Cyrillic. Sometimes, when I do the corrections, I make the 
edits without changing the layout, thus instead of writing "a" I write 
"а" (these are symbols that have different unicode number), especially 
when I know that the recipient has a UTF-8 capable client. From time to 
time I correct the subject in this way.

	So I consider the policy your users apply as wrong. Better use spam 
filter, instruct it to increase the spammines, when encoded headers 
arrive, and sum the results of the spam tests, before evaluating the 
mail for spam.

Дилян Палаузов wrote:
>> I looked through the cyrus code, and I think the fix is actually 
>> pretty easy, but it occured to me that in fixing this, we completely 
>> loose the ability to match against the non-decoded headers. That's 
>> actually annoying because I know a number of people do that to try and 
>> block emails with particular character sets in the subject line. For 
>> instance I know people have script tests like this:
>>
>> By fixing this bug, it would break those existing scripts, and from 
>> what I can see in the RFC, there's no way to actually force a match 
>> against the non-decoded header (eg no :raw option or something like 
>> that).
>>
>> Am I missing something? Is there a way to make a header match against 
>> a non-decoded header?
>>
>> Rob
>>
>

---

From: "Rob Mueller" <[email-masked]>
Date: Thu, 14 Feb 2008 09:22:37 +1100

> style. If you want to check against headers containing Cyrillic letters,

While that's in theory true, it's in practice rare.

Still maybe they know that they converse with no-one that ever uses any 
Russian encoding system with their email client, so blocking all koir-8 
headers is still a reasonable action.

> shall do the work, where "e" is Unicode 0435.

I presume "e" (unicode 0435) is a common character in Russian text. Still 
there's no guarantee that a Russian header will contain an "e" (unicode 
0435) in it.

It doesn't get past the point that there is still no way to look at the raw 
undecoded header even if you wanted to for some other reason.

Am I missing something? Was that intentional in the spec?

Rob

---

From: Ned Freed <[email-masked]>
Date: Wed, 13 Feb 2008 20:13:47 -0800 (PST)

> Am I missing something? Was that intentional in the spec?

FWIW, I brought the idea of having an option not to decide the header in the
distant past but there was little enthusiasm for it so I dropped it. It would
be an easy extension to specify, of course.

Our implementation has an option to do this that was added because, if memory
serves, a particular customer requested it. I've used it myself in a
couple of cases to catch some bogons. I don't know if any other customers
have used it, but I don't recall anyone else asking for the capbility.

				Ned

---

From: Дилян Палаузов<[email-masked]>
Date: Thu, 14 Feb 2008 15:57:18 +0100

Hello Rob,

	If somebody does not want to accept headers with Cyrillic characters, 
s/he has to list all character sets allowing Cyrillic characters, 
including UTF-8. For me this job is more for a per-user spam filter, 
than to Sieve.

	Nevertheless, I think that including an option raw in test envelope, 
supposes introduction of a similar option in test body, 
draft-ietf-sieve-body-07, Section 4.2. ":content".

	Със здраве,
		Дилян

Rob Mueller wrote:
> I presume "e" (unicode 0435) is a common character in Russian text. 
> Still there's no guarantee that a Russian header will contain an "e" 
> (unicode 0435) in it.
> 
> It doesn't get past the point that there is still no way to look at the 
> raw undecoded header even if you wanted to for some other reason.
>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 14 Feb 2008 16:19:53 +0100

I've been thinking about a langtest extension for a good long while. 
Like this: langtest <language> returns true if the message does or may 
use that language, and returns false if it certainly does not.

This is a fairly easy language test to implement, and I think it could 
be useful in scripts. Language detection in general is very hard, but 
this particular version is easy. And it can be used easily in scripts. 
For me, discarding russian safely would be something like:

   If allof( langtest "ru", not langtest ["no","de","en"] ) ...

Perhaps it ought to take two arguments, and return true if the language 
may be in the first list and is definitely not in the second list.

Comments?

Arnt

---

From: Ken Murchison <[email-masked]>
Date: Thu, 14 Feb 2008 13:40:29 -0500

Ned Freed wrote:

> have used it, but I don't recall anyone else asking for the capbility.

Ned, would you mind sharing the syntax that your implementation uses? 
If we implement this in Cyrus, we might as well use a consistent syntax.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: Ned Freed <[email-masked]>
Date: Thu, 14 Feb 2008 18:07:50 -0800 (PST)

> including UTF-8.

OK, I seem to be missing something basic here, but given that a properly
functioning Sieve implementation will decode all encoded words using
well-known charsets to Utf-8, why would you need to list anything other than
the Cyrillic characters as they appear in UTF-8?

Using :regex with a bracket expression covering the entire Cyrillic range in
Unicode (0400-04FF) strikes me as the obvious way to do this.

> For me this job is more for a per-user spam filter,
> than to Sieve.

I've ignored the question of _why_ you'd want to do this.

> draft-ietf-sieve-body-07, Section 4.2. ":content".

Again I must be missing something - envelope deals in envelope address
information which is currently restricted to ASCII. Perhaps you meant
the header test instead?

				Ned

---

From: Дилян Палаузов<[email-masked]>
Date: Fri, 15 Feb 2008 05:22:56 +0100

Hello Ned,

 > OK, I seem to be missing something basic here, but given that a
 > properly functioning Sieve implementation will decode all encoded
 > words using well-known charsets to Utf-8, why would you need to list
 > anything other than the Cyrillic characters as they appear in UTF-8?

	The point is that Rob asked for properly functioning Sieve 
implementation that rejects mails with KOI8-R encoded subjects, as he 
searches for a way to refuse mails with subjects in Russian language. If 
he wants to refuse mail with subject in Russian language, or any other 
language using Cyrillic alphabet, based on the used charset, then he is 
supposed to list all charsets that can be used for encoding Cyrillic 
characters. To achieve what he wants, it is easier to check for the 
presence of ~ 3 most used characters in Cyrillic languages, regardless 
of the used charset. Using regex with brackets is even better that 
listing the 3 most used characters.

 > information which is currently restricted to ASCII. Perhaps you meant

Here I mean header test, I am sorry for the confusion.

	Със здраве,
		Дилян

---

From: Philip Guenther <[email-masked]>
Date: Thu, 14 Feb 2008 21:22:40 -0700

> Unicode (0400-04FF) strikes me as the obvious way to do this.

It would arguably be better to use a regexp of "\p{Cyrillic}", as there's 
apparently a couple Cyrillic characters outside the Cyrillic block now. To 
quote http://www.unicode.org/Public/UNIDATA/Scripts.txt:
1D2B          ; Cyrillic # L&       CYRILLIC LETTER SMALL CAPITAL EL
1D78          ; Cyrillic # Lm       MODIFIER LETTER CYRILLIC EN

...but those aren't in the koi8-r tables I've seen, so they may be 
irrelevant to the goal.  As always, figuring out what the actual goal is 
is key to finding the optimal implementation.

Philip Guenther

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 15 Feb 2008 09:35:23 +0100

Philip Guenther writes:
> It would arguably be better to use a regexp of "\p{Cyrillic}", as 
> there's apparently a couple Cyrillic characters outside the Cyrillic 
> block now. To quote 
> http://www.unicode.org/Public/UNIDATA/Scripts.txt:
> irrelevant to the goal.

Koi8-r is (almost) just the Russian subset of Cyrillic. The other 
languages using Cyrillic have other subsets.

I think en is for Ukrainian, but don't hold me to that.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 15 Feb 2008 12:08:22 +0100

Ned Freed writes:
>> For me this job is more for a per-user spam filter, than to Sieve.
>

Some people do, including me.  For some reason I get a truckload of 
Turkish spam. That's not common - I don't know anyone else in Munich 
who does.

IMO using regexps to identify Russian mail is too crude. There are too 
many other messages that contain one or five or ten Cyrillic 
characters. Dilyan's messages in this thread would match. If there is a 
significant need to do it (more than just myself and five other 
people), then we need something like the langtest I suggested.

Arnt