Subject: Re: Header-Format: Downgraded (Re: [EAI] CONSENSUS CALL: Removal of Header-Format: marker)

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 01 Feb 2007 21:02:05 -0000

On Wed, 31 Jan 2007 23:14:25 -0000, Chris Newman <[email-masked]>  
wrote:

> Charles Lindsey wrote on 1/31/07 17:08 +0000:
>> BTW, I tried an experiment to see how much overhead this check cost,  
>> and I
>> could not detect any difference,...
>
> On modern hardware, the MTA bottlenecks are either the spam/virus engine  
> or the disk I/O subsystem.  The cost to scan the MIME structure (or  
> scan/verify a DKIM signature) is negligible by comparison.  You'd  
> probably see the same result (MIME scanning has no real cost) on any  
> modern MTA which properly commits messages to disk with fsync/fdatasync  
> or the equivalent.

No, there was no spam/virus engine in my test, and the timing was done  
with /usr/bin/time which reports cpu time separately from real time  - but  
in any case I have enough memory that it was all cached in memory anyway,  
so real time was little more than cpu time.

However, the plot thickens - and then it thins out again. I have spent  
today doing more careful timing tests, and it turns out that sendmail  
actually runs _faster_ when you do the walk of the whole mime tree than  
when you don't! And that eventiually turned out to be becasue it actually  
copies the file to the output as it is doing the walk, and it just happens  
to use slightly faster (but I suspect buggier) code that when it does its  
normal copying.

Anyway, here is how I now see it as regards downgrading (or bouncing) in  
the case of UTF8SMTP.

1. As you read the DATA, you check whether there is an 8th bit anywhere in  
the message. If not, then you have neither 8BITMIME nor UTF8SMTP problems,  
and everything else is simple. Note that this is only done once; what  
follows has to be done separately for each recipient of the message  
(though sensible cacheing of any downgraded form will help).

2. Moreover if, for each recipient you look at, the next MTA in line  
already supports UTF8SMTP (and hence 8BITMIME as well), you are still on  
the simple case and no special measures are needed.

3. Otherwise. if the message turns out to need 8BITMIME or UTF8SMTP, then  
downgrading is needed. If you try to discover this _beforehand_ (by  
walking through the whole tree, assuming we do not have a Header-Type to  
help us out), and then downgrade (or not) depending on the result of that  
test, then you will have a severe performance hit on account of doing the  
walk. So you don't do it that way.

4. Instead of that (and assuming you have already found an 8th bit  
somewhere in step #1, and no UTF8SMTP capability in the next agent in step  
#2), you start doing the full downgrade process, walking through the  
complete MIME tree and looking for an 8th bit in headers (which will need  
a UTF8SMTP downgrade) or an 8th bit in a body (which will need an 8BITMIME  
downgrade unless the nect MTA does 8BITMIME). So, in either case, you do  
each downgrade (or bouce/reject if you don't like downgrading) as you come  
to it.

5. If, in fact, it turns out out that no downgrading was needed at any  
stage, then no change will have been made to the message and no harm will  
have been done, but you will have wasted a bit of time analysing the MIME  
structure as you went throught it; but the imnportant thing is that you  
never had to scan the whole message twice.

Obviously, much the same thing can be done by MDAs and MUAs. What worries  
me is still that people who write ad hoc scripts for processing mail are  
unlikely to go through all that properly, and that will lead to errors. So  
I would still prefer to see a Header-Type header, but it looks as though I  
am going to be outvoted on that one :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 02 Feb 2007 13:28:57 +0100

Charles Lindsey wrote:

> am going to be outvoted on that one :-(

I think you're confusing 8BITMIME and UTF8SMTP.  For the latter you have
only to look at the header, check that it's either pure ASCII (including
2047/2231 encodings resulting in pure ASCII), or valid UTF-8, or invalid.

Especially the presence of some non-ASCII octets isn't the same as UTF-8,
you MUST NOT claim to have UTF8SMTP just because there's a non-ASCII byte.

And as far as "message/utf822" (or similar) is concerned the body of the
message, multipart, message, text, or what else, is irrelevant, it's the
job of the producer to get this right, not the job of a "downgrade" from
UTF8SMTP to ESMTP.

Parsing the complete MIME structure is necessary for 8BITMIME to 7bit
gateways, but not for a UTF8SMTP to 8BITMIME "downgrade" gateway.  Or do
I miss some odd scenarios here ?

Frank

---

From: Randall Gellens <[email-masked]>
Date: Fri, 2 Feb 2007 01:04:56 -0800

At 11:13 AM +0100 1/26/07, Harald Tveit Alvestrand wrote:

>  1. I agree with this resolution

I support this view, since from an SMTP server point of view the 
header is useless.  However, I agree that practically speaking, there 
is a need for the POP or IMAP server to know if the message contains 
non-ASCII.  But there must be other ways to do so, that would be more 
reliable.  For example, the delivery agent could scan the message and 
add something to the already horrible to parse "From " separator. 
Or, the POP or IMAP server could do the scan itself.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Maybe Computer Science should be in the College of Theology.  -
                                              -R. S. Barton

---

From: Kari Hurtta <[email-masked]>
Date: 02 Feb 2007 20:56:22 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> On Wed, 31 Jan 2007 23:14:25 -0000, Chris Newman
> <[email-masked]>  wrote:
> 
> >> BTW, I tried an experiment to see how much overhead this check
> >> cost,  and I

> actually runs _faster_ when you do the walk of the whole mime tree
> than  when you don't! And that eventiually turned out to be becasue it
> actually  copies the file to the output as it is doing the walk, and
> it just happens  to use slightly faster (but I suspect buggier) code
> that when it does its  normal copying.
> 
> Anyway, here is how I now see it as regards downgrading (or bouncing)
> in  the case of UTF8SMTP.
> 
> 1. As you read the DATA, you check whether there is an 8th bit
> anywhere in  the message. If not, then you have neither 8BITMIME nor
> UTF8SMTP problems,  and everything else is simple. Note that this is
> only done once; what  follows has to be done separately for each
> recipient of the message  (though sensible cacheing of any downgraded
> form will help).
> 
> already supports UTF8SMTP (and hence 8BITMIME as well), you are still
> on  the simple case and no special measures are needed.
> 
> 3. Otherwise. if the message turns out to need 8BITMIME or UTF8SMTP,
> then  downgrading is needed. If you try to discover this _beforehand_
> (by  walking through the whole tree, assuming we do not have a
> Header-Type to  help us out), and then downgrade (or not) depending on
> the result of that  test, then you will have a severe performance hit
> on account of doing the  walk. So you don't do it that way.
> 
> somewhere in step #1, and no UTF8SMTP capability in the next agent in
> step  #2), you start doing the full downgrade process, walking through
> the  complete MIME tree and looking for an 8th bit in headers (which
> will need  a UTF8SMTP downgrade) or an 8th bit in a body (which will
> need an 8BITMIME  downgrade unless the nect MTA does 8BITMIME). So, in
> either case, you do  each downgrade (or bouce/reject if you don't like
> downgrading) as you come  to it.
> 
> stage, then no change will have been made to the message and no harm
> will  have been done, but you will have wasted a bit of time analysing
> the MIME  structure as you went throught it; but the imnportant thing
> is that you  never had to scan the whole message twice.

That is just that stragegy, where sendmail runs problems when it detects
that some body part can not 8BITMIME downgraded. Because output is not
stored but instead directly written to socket, it can not cancel it (at
least without just closing socket.)  I suspect that UTF8SMTP downgrade
will run to similar problems.

----------------------------------------------------------------------
From: Kari Hurtta <[email-masked]>
Subject: Re: [EAI] Re: Sendmail message/TBD handling
Newsgroups: gmane.ietf.ima
Date: 14 Oct 2006 18:58:04 +0300

Frank Ellermann <[email-masked]> writes in gmane.ietf.ima:

> Kari Hurtta wrote:
>  
> > Sendmail will strip message/TBD to 7-bit if next host do not
> > announce 8BITMIME.
> 
> Odd, does it try that stunt with any 8bit content ?  1652 says
> "it must cause no loss of information; MIME transport encodings
> must be employed as needed to insure this is the case".  Losing
> one of eight bits is FUBAR.
> 
> > I suggested that mail should be bounced, but it is not
> > implemented. 
> 
> We can't freeze this WG until sendmail is upgraded to support 
> RFC 1652 published more than 12 years ago.  Maybe all poor old 
> pre-1652 MTAs (are forced to) duck behind non-sendmail MXs.

I just warn :-)

> Are you really saying that sendmail announces 8BITMIME support,
> and then forwards 8bit mails to non-8BITMIME hops by stripping
> this bit ?

Only for types where quoted-printable or base64 is not allowed
and type can not handled recursively.   

That means message/* other that message/rfc822

( 8bit is also stripped from multipart preamble, I think.
  But that 8bit on multipart preamble is more or less undefined
  area. )

quoted-printable or base64 are applied to leaf mime types,
where these encodings are allowed.

> [2821 2.4] An originating SMTP client which has not successfully
> | negotiated an appropriate extension with a particular server
> | MUST NOT transmit messages with information in the high-order
> | bit of octets.
> 
> That certainly doesn't mean "just strip the 8th bit on the side
> of the client if the server doesn't offer 8BITMIME".  
> 
> Frank

/ Kari Hurtta

KNOWNBUGS file from sendmail distribution:

* 8->7 bit MIME conversion

  When sendmail is doing 8->7 bit MIME conversions, and the message
  contains certain MIME body types that cannot be converted to 7-bit,
  sendmail will strip the message to 7-bit.

doc/op/op.me from sendmail distribution:

      $=s  contains the set of subtypes of message that  can
           be  treated  recursively.  By default it contains
           only "rfc822".  Other "message/*" types cannot be
           8->7  bit encoded.  If a message containing eight
           bit data is sent to a seven bit  host,  and  that
           message  cannot  be  encoded  into seven bits, it
           will be stripped to 7 bits.

So at least documentation say so.

8BITMIME downgrade happens on sendmail/mime.c but it
is difficult to give meaningfull quote.

---

From: Kari Hurtta <[email-masked]>
Date: 02 Feb 2007 22:54:42 +0200

> you MUST NOT claim to have UTF8SMTP just because there's a non-ASCII byte.

So question: 
        When message is UTF8SMTP and therefore candinate of UTF8SMTP downgrading ?

        Possibilities

        1) Some top level header fields with UTF8 sequences and otherwise ASCII only 
           header fields

        2) Some top level header fields with UTF8 sequences and some top 
           level header fields with non-ASCII byte which don't form UTF-8

        3) Some top level header fields with UTF8 sequences  and some MIME header 
           fields with UTF8 sequences

        4) Top level header fields ASCII only and some MIME header fields with
           UTF8 sequences

        5) Some top level header fields with UTF8 sequences and some MIME header 
           fields with  non-ASCII byte which don't form UTF-8

        1') Some top level header fields with non-ASCII bytes and otherwise ASCII only 
           header fields

I think that there is no good answer to that question.

So perhaps just downgrade these header fields with are UTF-8.

Other header fields are garbage in, garbage out. Alternatively
they can be downgraded as UNKNOWN-8BIt charset.

That part "you MUST NOT claim to have UTF8SMTP just because there's a non-ASCII byte"
will not be problem when there is Header-Format: -header field and no ESMTP option
for that.  So there is no claim that message is UTF8SMTP.

It is true that there is message header fields which are not pure ASCII and not 
either UTF-8.  Specially Subject: -header field with random non-ASCII charset
(without 2047 encoding) are seen daily. Other header fields seen are:
phrases on From:, To:, CC, Content-Description:, filename on Content-Disposition:
-header field.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Feb 2007 20:55:06 +0100

> Harald Tveit Alvestrand wrote:
 
>> 3. I have another opinion, which is:
 
> I can't judge it at the moment.
[...]
> I'd feel far more confident that the WG is on the right track if we
> nail the message/rfc822 vs. message/utf8smtp (or similar) question.

Strike that, I hadn't seen the message/utf-8 draft when I posted that.

Removing the "header-format" header field should now work, "top level"
messages can be identified (and if necessary downgraded) by looking
at their header, and a "nested" message/utf-8 is already tagged with
an explicit Content-type in its container.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 22:21:42 +0200

> an explicit Content-type in its container.

You still not take account Content-Type and  Content-Disposition header
fields on MIME structure which are allowed include utf-8 on UTF8SMTP
message.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 03 Feb 2007 22:43:58 +0200

> fields on MIME structure which are allowed include utf-8 on UTF8SMTP

That of course is not problem. You need just also look all MIME headers
on MIME structure, before you decide is that UTF8SMTP message.

/ Kari Hurtta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 04 Feb 2007 13:37:02 -0800

--On 30. januar 2007 17:33 +0000 Charles Lindsey <[email-masked]> 
wrote:

> On Mon, 29 Jan 2007 19:18:50 -0000, Harald Tveit Alvestrand
> <[email-masked]> wrote:
>
>> --On 29. januar 2007 11:13 +0000 Charles Lindsey <[email-masked]>
>> wrote:
>
>>> But I suspect those languages mostly do not need the language
>>> information
>>> to assist in the display process, whereas I would imagine that it is
>>> useful for a user agent to know whether it is displaying Chinese or
>>> Japanese, even though those languages use essentially the same "CJK"
>>> parts of Unicode.
>>
>> Have you read RFC 2277?
>
> Yes, the relevant bit seems to be:
>
>     Many operations, including high quality formatting, text-to-speech
>     synthesis, searching, hyphenation, spellchecking and so on benefit
>     greatly from access to information about the language of a piece of
>     text. [WC 3.1.1.4].
>
> Which seems to cover the sort of situation that I mentioned.

And where in that text did you find any justification for assuming that 
this information is required for UTF-8 headers and not required for ASCII 
headers?

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 04 Feb 2007 13:41:07 -0800

The responses to this consensus call are summarized as follows:

Tony Finch
Bill McQuillan
Chris Newman
Yangwoo Ko
Kari Hurtta
Yao Jankang
John Klensin
Jeff Yeh
Kazunori Fujiwara
Yoshiro Yoneya
Randall Gellens
Frank Ellermann

2. I disagree with this resolution, for the following technical reasons:
Charles Lindsey - multiple reasons

This issue is now settled. Further discussion is off-topic for the WG, 
unless new technical information can be brought to the table.

I would STRONGLY recommend that dissenting voices not attempt to reopen the 
issue without having checked with at least a few other particpiants that 
they will support reopening the issue based on that new technical 
information.

                Harald Alvestrand

The original call:

--On 26. januar 2007 11:13 +0100 Harald Tveit Alvestrand 
<[email-masked]> wrote:

> The discussion of whether or not we need a header marker for marking
> UTF8SMTP messages that actually use UTF-8 characters has reached a point
> where it seems that no new technical information is being provided.
>
> At this point, the chairs wish to see if the group is near a possible
> consensus on the issue.
>
> PROPOSED RESOLUTOIN:
>
> There is no significant operational or implementation benefit in having a
> marker in the headers of UTF8SMTP messages, and significant additional
> complexity. Therefore, the proposed "Header-Type" section,
> draft-ietf-eai-utf8headers-02 section 5, will be removed.
>
> Note that this is not a call on the proposal to have a similar marker in
> the SMTP protocol; that's a separate issue.
>
> POSSIBLE RESPONSES:
>
> 2. I disagree with this resolution, for the following technical reasons:
> RESPOND:
> - To the list, if you want to present an argument or just want to make
> your position visible at once
> - To the chairs, if you don't want to respond on-list.
>
> The deadline for responses is THURSDAY, FEBRUARY 1, 23:59 GMT
> The chairs will prepare a summary, with names listed for each position,
> hopefully on Friday, Feb 2.
>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 05 Feb 2007 19:18:30 -0000

> this information is required for UTF-8 headers and not required for  
> ASCII headers?

Nowhere. My point was that the need for such language information was  
greater in the case of Non-Ascii texts. Currently, we do not have any  
mechanism for indicating language in headers beyond the much-disliked RFC  
2231. One would hope that one eventual byproduct of our endeavours would  
be that RFC 2047 and RFC 2231 could eventually be consigned to history.  
But that would require a Header-Language header (which might then even be  
useful in pure ASCII messages).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 05 Feb 2007 21:48:39 -0000

On Sat, 03 Feb 2007 22:01:51 -0000, Kari Hurtta  
<[email-masked]> wrote:

NOT on the list (he wrote a similar message to the list, but this version  
raises more issues)

> That walking and downgrading need to be done whole before outputting
> DATA command to next MTA.  This is fine, if you can keep result on  
> memory.
> You will hit on perfomance if you need write it to disk.

But, in practice, even my 22MB attachment (30MB after Base64) happily sat  
in memory while sendmail fiddled with it, and few messages get that big.

OTOH, even if you do a separate scan through the message prior to deciding  
whether it needs to be downgraded anywhere, you still have the problem  
that it may not still be in memory when you eventually do go back to do  
the actual downgrade.

But I take your point that a separate pre-scan may indeed be necessary,  
unless you can be sure in advance that your donwgrade cannot fail.
>
> Sendmail gets away with this by never failing on downgrading (of  
> 8BITMIME).
> That leads that some standard is always violated on message/{unknown}  
> case :-)

Yes, like leaving 8bit stuff around even after downgrading from 8BITMIME  
:=( .

---

From: Randall Gellens <[email-masked]>
Date: Wed, 7 Feb 2007 23:42:00 -0800

At 5:02 PM +0100 1/27/07, Frank Ellermann wrote:

>  Chris Newman wrote:
>
>>  A mbox-style file which contains UTF8SMTP messages would be a different
>>  file format from today's mbox format.  There are multiple ways to
>>  distinguish the two file formats, including by file extension,
>  [...]
>>  A final delivery agent which put a UTF8SMTP message in a vanilla mbox
>>  file would have a software bug that needs to be corrected.
>
>  Yes, and that's an important statement, we've to make sure that the
>  affected drafts (e.g., maybe the mailinglist I-D wrt "digest mode")
>  clearly say so.

mailinglist -01 doesn't mention digests, and if it did, I think it 
would discuss MIME multiparts and not mbox format.

I do think the POP and IMAP drafts might want to mention it.  Perhaps 
the mailinglist draft, in a parenthetical aside about list agents 
that read from mbox format files.  Also perhaps, the SMTP extension 
document might have an aside about final delivery to vanilla mbox 
spools.

-------------- Randomly-selected tag: ---------------
Note that although IBM 'invented' virtual memory in 1972 it had
been available in a multi-processor multi-tasking environment on
the Burroughs B5000 line since 1963 and the B6000 line since 1968.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 07 Feb 2007 15:12:13 -0800

--On 5. februar 2007 19:18 +0000 Charles Lindsey <[email-masked]> 
wrote:

> greater in the case of Non-Ascii texts.

Good, then we have made it sufficiently clear that the RFC you are citing 
gives no backing for such a claim, and the claim is entirely your own.
I, for one, don't believe it.

> Currently, we do not have any

Actually, the semantics of 2231 may have the right level of granularity. If 
I reply to a message, and the reply passes through a mailing list, the 
subject line will be in a language chosen by the original sender, the 
free-form text part of my email address will be in a language chosen by me, 
and the List-* headers will be in a language chosen by the list maintainer.

But (speaking with my charter-guardian chair cap firmly pulled down over my 
ears) I believe you have sufficiently demonstrated that the need for 
"header-language" is an orthogonal issue to what EAI is trying to solve, 
and is therefore off-charter for this group.

                 Harald

---

From: Randall Gellens <[email-masked]>
Date: Thu, 8 Feb 2007 00:41:48 -0800

At 9:12 PM +0200 1/29/07, Kari Hurtta wrote:

>   > Please sunnarize again what the UTF8-required-fields would do.
>>
>
>  Downgrade needs able to delete headers (when it do not
>  know syntax of headers).  Content of these headers are
>  stored to some encapsulation format (for example Downgraded
>  -header).
>
>  These headers may be necessary for some protocols.
>  That header field makes possible to sender specify that.
>
>  UTF8-required-fields sets requirements, which header fields
>  downgrading process must know. It says that these headers
>  must exists even when message is on downgraded form.

Is the intent to force the presence of headers listed as being 
required even when they contained non-ASCII and were hence removed 
during downgrade?  Or is it to force bounce instead of downgrade when 
headers listed as being required contain non-ASCII?
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
It is impossible to make anything foolproof because fools are so
ingenious.

---

From: Kari Hurtta <[email-masked]>
Date: 08 Feb 2007 18:02:59 +0200

> headers listed as being required contain non-ASCII?

Force bounce when headers listed as being required contains non-ASCII
and downgrading is not available (Downgraded: -header field is encapsulation
and do not qualify. )

Sender do not know which all header syntax downgrading gateway knows,
but it knowns protocol which it is using for signaling.

/ Kari Hurtta