Subject: Re: Status of UTF-8 headers?
From: Russ Allbery <[email-masked]>
To: [email-masked]
Date: 29 Apr 2001 15:48:33 -0700
List-Id: <ietf-822.imc.org>

Keith Moore <[email-masked]> writes:

> I can't imagine that we should try to do anything regarding utf-8 in
> message headers until the IDN work is finalized.

Thanks for the pointer; that had been mentioned earlier on USEFOR as well
but I for some reason didn't think through the connection.  Yes, that
makes sense.

Hm.  IDN also seems to be working to solve the same canonicalization
problems that we'll need to solve for newsgroup names.  Excellent.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked]
Subject: Re: Status of UTF-8 headers? 
Date: Sun, 29 Apr 2001 18:16:16 -0400
Sender: [email-masked]

I can't imagine that we should try to do anything regarding utf-8
in message headers until the IDN work is finalized.  

Keith

To: [email-masked]
Subject: Status of UTF-8 headers?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: 29 Apr 2001 13:04:27 -0700
Lines: 17
Sender: [email-masked]

Some two years ago, there was a discussion on this mailing list of adding
UTF-8 character set support in headers to e-mail, and an expressed
consensus that this was where everyone was wanting to go in the long term.
This was stated to be a post-DRUMS project, which at the time was expected
to conclued in a few months.

DRUMS unfortunately took much longer than anyone wanted, but it's now
complete.  I'd really like to see unified handling of UTF-8 in headers
between e-mail and netnews, and as the USEFOR draft is making very little
progress, it seems likely to me that this is feasible.  Are there any
plans in the works for starting the UTF-8 header work for e-mail (probably
after a few months off for everyone involved in DRUMS to catch their
breath), and if so, would the people working on that be interested in
finding solutions that would also work for netnews?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: [email-masked]
To: [email-masked]
Date: Thu, 26 Apr 01 16:20:43 EST
Subject: Prize Notification
Sender: [email-masked]

                                      ---NOTICE---
This message is to notify you of an investigation on you by an "iDetect Me" user. We
Have Blocked The "iDetect ME" user's Identity for Security reasons. Don't Be Scared,
You are not going to jail. This just means that someone saw you online and wanted to
find out just who you are and maybe a little more. 

You can do a little investigating for yourself by getting iDetect ME. You will also get
A list of people who have done Investigations on you and your family.
                                     ---Read Below---

Now you can learn EVERYTHING about your friends, neighbors, enemies, 
employees, co-workers, your boss, even yourself!
 
Is there someone online that you want to find out more about?
 
Are they exploring areas of the internet that they shouldn't be?
 
It's time for you to see for yourself and find out what they are really doing
online!
 
Don't let suspicions become paranoia, uncover the truth now before it's too
late!
 
Many people hire a private investigator to observe, track and record what
their husband, wife, boyfriend/girlfriend and children are doing.  Be your
own private investigator. Get the full details of the hidden truth. Now you
can  investigate anyone secretly, and in the privacy of your own home! 
 
Get your hands on the most powerful IDetect ME ware available today! With it,
you can find out who they are talking to and more importantly what they are
talking about!
 
See and record every word typed in:
 
 EMails
 Private Messages
 Chat Rooms
 Message Boards
 ICQ & IRC
 
Find out who they are talking to and what they are saying.  Find out what type
of chatrooms they are going in.  Record and check out all the web sites they
are visiting or searching for!  Best of all, they don't have to be using the same
computer as you!  IDetect ME can find all this out by using just their IP Address!
And with all the easy to use tools, finding out an IP is one of the easiest features!
 
Need to find out about someone who's not online?  With Spy-Soft, you can locate anyone
by their name, phone number, state license plate number, social security number, even 
e-mail address.  You can even find unlisted phone numbers!
 
Credit checks are easy and free with IDetect ME.  Never pay again just to see a copy of
your's or anyone else's credit report! 
 
Take a look at some of the other features packed into IDetect ME's powerful tools...
 
 - Look at anyone's criminal records
 - Examine their driving record
 - Locate people who have changed their name
 - Locate birth parents and adoptees
 - Anonymously surf the internet
 - Locate long lost friends  
 - Locate e-mails, phone numbers, or addresses. 
 - Get a Copy of Your FBI file. 
 - FIND DEBTORS and locate HIDDEN ASSETS. 
 - Check DRIVING and CRIMINAL RECORDS. 
 - Locate old classmates, missing family member, or a LONG LOST LOVE. 
 - Do Background Checks on EMPLOYEES before you hire them. 
 - Investigate your family history, birth, death and SOCIAL SECURITY RECORDS! 
 - Discover how UNLISTED phone numbers are located. 
 - Check out your new or old LOVE INTEREST. 
 - Verify your own CREDIT REPORTS so you can correct WRONG information. 
 - Track anyone's Internet ACTIVITY; see the sites they visit. 
 - Explore SECRET WEB SITES that conventional search engines have never found. 
 - Discover little-known ways to make UNTRACEABLE PHONE CALLS. 
 - Check ADOPTION records; locate MISSING CHILDREN or relatives. 
 - Dig up information on your FRIENDS, NEIGHBORS, or BOSS! 
 - Discover EMPLOYMENT opportunities from AROUND THE WORLD! 
 - Locate transcripts and COURT ORDERS from all 50 states. 
 - CLOAK your EMAIL so your true address can't be discovered. 
 - Find out how much ALIMONY your neighbor is paying. 
 - Discover how to check your phones for WIRETAPS. 

WARNING:  You may be shocked or overwhelmed by the results of what you
find!!!  
 
IDetect ME is simple to install and use and will work on any PC or
Macintosh that has an internet connection.

To order IDetect ME, send in the order form below.
  
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 
       R-ID#:   154

          Send To.......
                            vCash
                            7013 Foothill Blvd
                            Tujunga Ca, 91042

 
Your Name________________________________________________
 
Address___________________________________________________
 
City_____________________________State_________Zip__________       
 
[ ] Please rush me IDetect ME for just...$29.99.
 
[ ] Include $8.00 for EXPRESS Shipping and handling.

» Please make checks and money orders payable to vCash.
 
Important! Please circle one:
 
Is your computer a   [PC]   or a   [Macintosh]?
 
*Note*  After purchasing our program, we are not responsible
for your use of it.     

Date: Wed, 25 Apr 2001 21:39:43 -0400
To: [email-masked]
From: Harald Tveit Alvestrand <[email-masked]>
Subject: Re: Content-MD5
Cc: [email-masked], [email-masked]
Sender: [email-masked]

At 19:11 12.04.2001 -0700, [email-masked] wrote:
>I believe this is done in anticipation of agents that can deal with 8bit but
>not MIME encodings. And regardless of whether or not you or I think this is a
>good thing, it apparently happens fairly often. I see messages like
>"autoconverted from quoted-printable to 8bit" in message headers associated
>with text parts all the time. (Such messages are especially amusing when the
>message is subsequently reencoded, as sometimes happens.) Of course it is only
>a problem if the text wasn't in canonical form to begin with.

Changing the "9" flag on Sendmail's mailer descriptions into an "8" seems 
to increase the sanity of the process (and the likelyhood of verifying 
PGP/MIME signed messages).

From: "Son Ho Jin" <[email-masked]>
To: "[email-masked]"[email-masked]
Subject: I got it
Date: Wed, 25 Apr 2001 03:56:46 +0900
Sender: [email-masked]

Hey, lu

Sorry, it took longer than i expected but I found the site, it's

 http://www.multiopen.com
 
 the site will make your web surfing very convenient.

And here goes one more, it's

 http://www.mysimon.com 
 
 this one will help your online shopping    
 
 Get to the site and mail me after

 bye~

From: [email-masked]
To: <[email-masked]>
Subject: Customer Care Center                          22227
Date: Fri, 13 Apr 2001 06:05:59 -0800
Sender: [email-masked]

<HTML>
<BODY>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><HTML><HEAD>=
<TITLE>Take Control Of Your Conference Calls</TITLE><META http-equiv=3DCon=
tent-Type content=3D"text/html; charset=3Dwindows-1252"><META content=3D"M=
SHTML 5.50.4134.100" name=3DGENERATOR></HEAD><BODY vLink=3D#c0c0c0 link=3D=
#c0c0c0 bgColor=3D#000000 leftMargin=3D0><PRE><FONT face=3Darial,helvetica=
><HEAD><META content=3DFrontPage.Editor.Document name=3DProgId><DIV align=3D=
center><CENTER><TABLE height=3D789 cellSpacing=3D0 cellPadding=3D0 width=3D=
602 border=3D0><TBODY><TR vAlign=3Dtop><TD width=3D602 height=3D452><DIV a=
lign=3Dcenter><TABLE width=3D470 border=3D0><TBODY><TR><TD><P align=3Dcent=
er><FONT color=3D#0000ff size=3D7><B><FONT color=3D999999 size=3D6>Why Pay=
 More For Your </FONT></B></FONT><FONT color=3D#999999 size=3D6><B>Confere=
nce Calls?</B></FONT></P></TD></TR></TBODY></TABLE><TABLE width=3D352 bord=
er=3D0><TBODY><TR><TD><P align=3Dcenter><FONT color=3D#ff0000 size=3D5>Onl=
y <U><B>.18 Cents </B></U>per minute (Including long distance!)</FONT></P>=
</TD></TR></TBODY></TABLE></DIV><UL><UL><UL><UL><UL><LI><DIV align=3Dleft>=
<FONT color=3D999999 size=3D3><B>No setup fees</B></FONT></DIV><LI><DIV al=
ign=3Dleft><FONT color=3D999999 size=3D3><B>No contracts or monthly fees</=
B></FONT></DIV><LI><DIV align=3Dleft><FONT color=3D999999 size=3D3><B>Call=
 anytime, from anywhere, to anywhere</B></FONT></DIV><LI><DIV align=3Dleft=
><FONT color=3D999999 size=3D3><B>International Dial In 18 cents per minut=
e</B></FONT></DIV><LI><DIV align=3Dleft><FONT color=3D999999 size=3D3><B>C=
onnects up to 100 participants</B></FONT></DIV><LI><DIV align=3Dleft><FONT=
 color=3D999999 size=3D3><B>Operator Help available 24/7</B></FONT></DIV><=
/LI></UL></UL></UL></UL></UL><DIV align=3Dcenter><TABLE width=3D424 border=
=3D0><TBODY><TR><TD><DIV align=3Dcenter><FONT color=3D#ff0000 size=3D6><B>=
<FONT size=3D5>Get the best quality, the easiest to use,</FONT></B></FONT>=
 <FONT color=3D#ff0000 size=3D5><B>and lowest rate in the industry.</B></F=
ONT></DIV></TD></TR></TBODY></TABLE><TABLE width=3D300 border=3D0><TBODY><=
TR><TD height=3D73><DIV align=3Dcenter><P align=3Dcenter><FONT color=3D#99=
9999 size=3D4>If you like saving money, fill out the form below and one of=
 our consultants will contact you.</FONT></P></DIV></TD></TR></TBODY></TAB=
LE></DIV></BLOCKQUOTE></BLOCKQUOTE><DIV align=3Dcenter><P align=3Dcenter><=
FONT color=3D#ff0000 size=3D2>Required Input Field</FONT><FONT color=3D#ff=
0000 size=3D2>*</FONT></P><TABLE cellSpacing=3D0 borderColorDark=3D#333300=
 cellPadding=3D3 width=3D600 borderColorLight=3D#ffffcc><TBODY><TR><TD><FO=
RM action=3D"mailto:[email-masked]?subject=3DConference_Inquiry" meth=
od=3Dpost encType=3Dtext/plain><DIV align=3Dcenter><TABLE width=3D"100%"><=
TBODY><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helvet=
ica, sans-serif" color=3D#ff0000 size=3D2>Name*</FONT></DIV></TD><TD width=
=3D"51%"><FONT color=3D#ff0000><INPUT name=3DNAME> </FONT></TD></TR><TR><T=
D width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helvetica, sans-se=
rif" color=3D#ff0000 size=3D2>Web Address*</FONT></DIV></TD><TD width=3D"5=
1%"><FONT color=3D#ff0000><INPUT value=3Dhttp:// name=3DURL> </FONT></TD><=
/TR><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helvetic=
a, sans-serif" color=3D#ff0000 size=3D2>Company Name*</FONT></DIV></TD><TD=
 width=3D"51%"><FONT color=3D#ff0000><INPUT name=3DCOMPANY_NAME> </FONT></=
TD></TR><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helv=
etica, sans-serif" color=3D#ff0000 size=3D2>State</FONT></DIV></TD><TD wid=
th=3D"51%"><FONT color=3D#ff0000><INPUT size=3D2 name=3DSTATE> </FONT></TD=
></TR><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helvet=
ica, sans-serif" color=3D#ff0000 size=3D2>Business Phone*</FONT></DIV></TD=
><TD width=3D"51%"><FONT color=3D#ff0000><INPUT name=3DBUS_PHONE> </FONT><=
/TD></TR><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Hel=
vetica, sans-serif" color=3D#ff0000 size=3D2>Home Phone</FONT></DIV></TD><=
TD width=3D"51%"><FONT color=3D#ff0000><INPUT name=3DHOME_PHONE> </FONT></=
TD></TR><TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helv=
etica, sans-serif" color=3D#ff0000 size=3D2>E-mail*</FONT></DIV></TD><TD w=
idth=3D"51%"><FONT color=3D#ff0000><INPUT name=3DEMAIL> </FONT></TD></TR><=
TR><TD width=3D"49%"><DIV align=3Dright><FONT face=3D"Arial, Helvetica, sa=
ns-serif" color=3D#ff0000 size=3D2>Type of Business</FONT></DIV></TD><TD w=
idth=3D"51%"><FONT color=3D#ff0000><INPUT name=3DTYPE_OF_BUSINESS> </FONT>=
</TD></TR></TBODY></TABLE><P><BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;<DIV align=3Dcenter><INPUT type=3Dsubmit value=3D"Submit In=
formation" name=3Dsubmit><P align=3Dcenter></BR><B></BR><FONT color=3D9999=
99 face=3D"Arial, Helvetica, sans-serif" size=3D5><P align=3Dcenter>This c=
ould be your ad!</FONT></B><FONT face=3D"Arial, Helvetica, sans-serif" siz=
e=3D2><BR><A href=3D"mailto:[email-masked]?subject=3DDirect Marketin=
g"><FONT color=3Dff0000>Click here to e-mail us your contact info</A>.</FO=
NT></P><P align=3Dcenter><FONT face=3D"Arial, Helvetica, sans-serif" color=
=3D#999999 size=3D1>This email is to those persons that have contacted Con=
ference Calls for Less regarding available services or product information=
.  If this email is reaching you in error and you feel that you have not c=
ontacted us, <A href=3D"mailto:[email-masked]?subject=3DRemove_Confere=
nce">click here</A>. We apologize and will gladly remove you from our mail=
ing list.</FONT></P></DIV></DIV></BODY>

</BODY>
</HTML>

From: [email-masked]
Date: Fri, 13 Apr 2001 01:11:29 -0700 (PDT)
Subject: Re: Content-MD5
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> > > But what is textual data?

> >The basic rule I use is anything under the top-level text type or anything
> >that's encoded using 7bit or 8bit. (The full set of rules I use is actually a
> >lot more complex, but this is to deal with operating systems that support more
> >complex file organizations that simple streams.)

> OK, I see that makes sense. Presumably anything sent (or to be sent) Q-P
> or Base64 is computed as it decodes (whether that turns out to be
> canonical or not). Anything that is sent 8bit, and then encoded into
> something different en route will presumably have been computed and sent
> canonically, so the encoding will be on the canonical form and should be
> checked as such. The only problem I can foresee is if some intermediate
> agent decides to decode the base64, then canonicalises it before passing
> it onwards. That might cause an MD5 falure at the ultimate destination,
> but I cannot understand why an intermediate site should think of doing
> such a thing, even if it could see Content-Type text/plain (but, you never
> know :-( ).

I believe this is done in anticipation of agents that can deal with 8bit but
not MIME encodings. And regardless of whether or not you or I think this is a
good thing, it apparently happens fairly often. I see messages like
"autoconverted from quoted-printable to 8bit" in message headers associated
with text parts all the time. (Such messages are especially amusing when the
message is subsequently reencoded, as sometimes happens.) Of course it is only
a problem if the text wasn't in canonical form to begin with.

> I can also see a problem if my postscript with naked NLs is decoded from
> base64 at the far end, and then immediately canonicalized (because that's
> what the remote postscript interpreter expects).

Alas, any attempt to automatically canonicalize Postscript as if it were text
is doomed to failure. Even examination of the source to determine what
terminators to use doesn't work: Since Postscript is a general-purpose
programming language with the ability to completely replace or augment the
process of reading program data, figuring out what line terminators are "right"
for a given Postscript object is equivalent to the halting problem. And as a
practical matter real-world Postscript programs can be so obfuscatory in nature
that approximate solutions don't work all that well either.

The common practice of embedding one PostScipt object inside of another also
leads to situations where different line terminators are needed at different
points in the same document. There are standards for doing this correctly
(so-called encapsulated Postscript) but in practice they often aren't
followed.)

> One needs to be sure that
> the MD5 checking is done by the mail agents before that happens, and not
> by any postscript agent. But that is as it should be.

Yes.

> However, all this ought really to have been spelled out explicitly in the
> RFC, which certainly seems to be ambiguous as written, though your
> interpretation would seem to be the only one which makes sense.

Probably so.

> > Hardly. In general Postscript is NOT text. It can contain arbitrary binary
> > sequences and even the parts that look to you like text can be sensitive to
> > what line terminators are used. (The format includes multiline byte counted
> > strings.) Unless PostScript is being carried around as 7bit or 8bit text you
> > have to treat it as binary.

> Hmmm! All the postscript binary I have ever seen has been textualized in
> hex or base64 or something, and provided with a postscript procedure to
> read it in. I grant that one could write postscript that included pure
> binary (and the means to read it in), but I have never seen any like that.

Well, all I can say is that you've been lucky. I encounter such usage
regularly. It is especially common when dealing with Postscript that contains
large, detailed images.

> > This is described in the MIME RFCs, BTW.

> I saw nothing relevant in RFC2046 where application/postscript is
> described.

Section 4.5.2, bullet item (7) calls out the possible binary nature of
Postscript explicitly.

> BTW, and off topic, there are dire warnings in there about nasty
> side effects of blindly obeying postscript sent by mail. Is it the case
> that anything sent as PDF rather than postscript is immune from that?

I believe PDF is supposed to be immune to this sort of nonsense, however, I've
never done a careful inventory of the PDF format like the one done I did for
Postscript. And FWIW, PDF uses internal offsets heavily, which unless
interpreters are coded to carefully range check offset values could result in
security problems.

				Ned

Newsgroups: local.mime
Date: Thu, 12 Apr 2001 12:40:01 -0500
To: [email-masked] (Charles Lindsey)
From: Steve Dorner <[email-masked]>
Subject: Re: Content-MD5
Cc: [email-masked]
Sender: [email-masked]

At 9:28 AM +0000 4/12/01, Charles Lindsey wrote:
>However, all this ought really to have been spelled out explicitly in the
>RFC, which certainly seems to be ambiguous as written, though your
>interpretation would seem to be the only one which makes sense.

The canonical form of a media type should be described in the 
registration for that type.  This should include instruction on 
whether text canonicalization should be performed.  Most type 
registrations are woefully inadequate in this regard, alas.

It's perfectly possible to have an application/* type that requires 
newline canonicalization.  It's possible to have a text/* type that 
forbids it.

The only way to handle it correctly is to maintain a "table" of known 
types and whether or not to apply text canonicalization on them, and 
for types not in the table to apply heuristics such as Ned mentions.

Oh well.

To: [email-masked]
Xref: clerew local.mime:558
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Content-MD5
Date: Thu, 12 Apr 2001 09:28:20 GMT
Lines: 85
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> But what is textual data?

>The basic rule I use is anything under the top-level text type or anything
>that's encoded using 7bit or 8bit. (The full set of rules I use is actually a
>lot more complex, but this is to deal with operating systems that support more
>complex file organizations that simple streams.)

OK, I see that makes sense. Presumably anything sent (or to be sent) Q-P
or Base64 is computed as it decodes (whether that turns out to be
canonical or not). Anything that is sent 8bit, and then encoded into
something different en route will presumably have been computed and sent
canonically, so the encoding will be on the canonical form and should be
checked as such. The only problem I can foresee is if some intermediate
agent decides to decode the base64, then canonicalises it before passing
it onwards. That might cause an MD5 falure at the ultimate destination,
but I cannot understand why an intermediate site should think of doing
such a thing, even if it could see Content-Type text/plain (but, you never
know :-( ).

I can also see a problem if my postscript with naked NLs is decoded from
base64 at the far end, and then immediately canonicalized (because that's
what the remote postscript interpreter expects). One needs to be sure that
the MD5 checking is done by the mail agents before that happens, and not
by any postscript agent. But that is as it should be.

However, all this ought really to have been spelled out explicitly in the
RFC, which certainly seems to be ambiguous as written, though your
interpretation would seem to be the only one which makes sense.

>This is basically just common sense: Agents routinely mess with
>the line terminators of text subtypes and things encoded as 7bit or
>8bit, so these are the cases that need to be canonicalized.

>> Now I can see that Content-Type:
>> text/plain is textual, and doubtless text/html likewise. And
>> application/some-binary-executable us clearly not textual (and arbitrary
>> changes of CFLF to LF, or whatever the local notation demanded would be
>> disastrous).

>> But what about application/postscript? That is certainly readable as
>> text,

>Hardly. In general Postscript is NOT text. It can contain arbitrary binary
>sequences and even the parts that look to you like text can be sensitive to
>what line terminators are used. (The format includes multiline byte counted
>strings.) Unless PostScript is being carried around as 7bit or 8bit text you
>have to treat it as binary.

Hmmm! All the postscript binary I have ever seen has been textualized in
hex or base64 or something, and provided with a postscript procedure to
read it in. I grant that one could write postscript that included pure
binary (and the means to read it in), but I have never seen any like that.

>This is described in the MIME RFCs, BTW.

I saw nothing relevant in RFC2046 where application/postscript is
described. BTW, and off topic, there are dire warnings in there about nasty
side effects of blindly obeying postscript sent by mail. Is it the case
that anything sent as PDF rather than postscript is immune from that?

>> There are two attachments to this message. One is that postscript in
>> base64, and the other is exactly the same file without encoding (I
>> may have some difficulty is persuading my system to send it without
>> encoding,

>Sounds like your system knows what it is doing.

Well almost. I had to hack the message manually and pipe it into sendmail
to make sure it went out as I intended :-(. I would still be curious to
know if anyone found that the two versions did or did not satisfy the MD5
check, and did or did not display correctly (esp. on windoze boxes).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: Charles Lindsey <[email-masked]>
cc: [email-masked], Ian Bell <[email-masked]>
Subject: Re: Content-MD5 
Date: Wed, 11 Apr 2001 14:33:47 -0400
Sender: [email-masked]

I think this is actually simpler than you are making it out to be, though
the wording in the MIME spec can be confusing in this regard.

for each content-transfer-encoding method there is a decoding algorithm
that unambiguously produces canonical form.  for quoted-printable, 7bit,
and 8bit this means that (non-escaped) line endings are ALWAYS decoded
as CR LF, even if it happens that the local mailbox format uses bare LF
as a line terminator.  

in the mail tools that I've written the decoding and the conversion
to local formats are separate steps.   thus encoded form is first
translated to canonical form, and (when necessary) that is then converted
to local form.  if the body part is quoted-printable, an unescaped
end-of-line in encoded form becomes CR LF in canonical form, which might
then be translated to LF in local form.

this works for all body parts regardless of content-type.

if you're generating a MIME object you should compute content-MD5 over
canonical form - i.e. over what the encoded form will produce after 
decoding.  if you happen to have botched the translation from source-local
to canonical form, it doesn't matter for the purposes of content-MD5
calculation - you need to compute MD5 over whatever octet string you 
have encoded in the body part, even if it was improperly translated
from source-local form.

Keith

From: [email-masked]
Date: Wed, 11 Apr 2001 07:35:22 -0700 (PDT)
Subject: Re: Content-MD5
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], Ian Bell <[email-masked]>
Sender: [email-masked]

> I have been trying to write a program to generate Content-MD5 headers
> for Mime objects, and find some difficulty in interpreting RFC1864.

> What it says there is that I am to compute the MD5 algorithm on "the
> canonical form of the MIME entity's object", which means the form
> before any Content-Transfer-Encoding (or after decoding same, if at the
> receiving end). So far so good.

> It then says:

> "For textual data, this means the MD5 algorithm must be computed on data
> in which the canonical form for newlines applies, that is, in which each
> newline is represented by a CR-LF pair."

> But what is textual data?

The basic rule I use is anything under the top-level text type or anything
that's encoded using 7bit or 8bit. (The full set of rules I use is actually a
lot more complex, but this is to deal with operating systems that support more
complex file organizations that simple streams.)

This is basically just common sense: Agents routinely mess with
the line terminators of text subtypes and things encoded as 7bit or
8bit, so these are the cases that need to be canonicalized.

> Now I can see that Content-Type:
> text/plain is textual, and doubtless text/html likewise. And
> application/some-binary-executable us clearly not textual (and arbitrary
> changes of CFLF to LF, or whatever the local notation demanded would be
> disastrous).

> But what about application/postscript? That is certainly readable as
> text,

Hardly. In general Postscript is NOT text. It can contain arbitrary binary
sequences and even the parts that look to you like text can be sensitive to
what line terminators are used. (The format includes multiline byte counted
strings.) Unless PostScript is being carried around as 7bit or 8bit text you
have to treat it as binary.

This is described in the MIME RFCs, BTW.

> and there is no special need to encode it as base64.

Sometimes there most certainly is such a need on anything short of a
binary transport.

> Or image/fig
> (don't know whether that is a recognised application type, but fig is
> certainly a way of specifying images, and it comes as text). Or any
> application/foo which the recipient might not understand, but would at
> least like to check that the MD5 agrees?

In this case it would again depend on whether or not image/fig is sent
as 7bit or 8bit.

> So I did some experimentation with Sun's dtmail (which has known bugs in
> its Content-MD, but at least seems to get it right for attachments). I
> gave it a shell script: it decided it was text plain, and put the CRs in
> before computing the MD5. I then constructed a postscript file (draws
> a little red circle) which, of course, on my Solaris system has lines
> terminated with LF only. It recognised that application/postscript was
> needed, it computed the MD5 on the LF version, and then encoded it in
> base64 (which seems a neat way to pass the problem onto someone else).
> But is it correct?

Yes it is.

> And what if I choose to leave the encoding at 7bit?

Then you risk the Postscript content being damaged during transport.

> Or if I receive an application/postscript in 7bit and want to check the
> MD5?

Canonicalize and see if it matches, but understand that in this case the
content may have been damaged as the document was sent (and effectively before
the content-md5 field was added).

> There are two attachments to this message. One is that postscript in
> base64, and the other is exactly the same file without encoding (I
> may have some difficulty is persuading my system to send it without
> encoding,

Sounds like your system knows what it is doing.

				Ned

Date: Wed, 11 Apr 2001 14:28:10 +0100 (BST)
From: Charles Lindsey <[email-masked]>
Subject: Content-MD5
To: [email-masked]
Cc: Ian Bell <[email-masked]>
Status:  O
Sender: [email-masked]

--Volery_of_Birds_372_000
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: wicf+7Ukwr6dlCwaebUckg==

I have been trying to write a program to generate Content-MD5 headers
for Mime objects, and find some difficulty in interpreting RFC1864.

What it says there is that I am to compute the MD5 algorithm on "the
canonical form of the MIME entity's object", which means the form
before any Content-Transfer-Encoding (or after decoding same, if at the
receiving end). So far so good.

It then says:

"For textual data, this means the MD5 algorithm must be computed on data
in which the canonical form for newlines applies, that is, in which each
newline is represented by a CR-LF pair."

But what is textual data? Now I can see that Content-Type:
text/plain is textual, and doubtless text/html likewise. And
application/some-binary-executable us clearly not textual (and arbitrary
changes of CFLF to LF, or whatever the local notation demanded would be
disastrous).

But what about application/postscript? That is certainly readable as
text, and there is no special need to encode it as base64. Or image/fig
(don't know whether that is a recognised application type, but fig is
certainly a way of specifying images, and it comes as text). Or any
application/foo which the recipient might not understand, but would at
least like to check that the MD5 agrees?

So I did some experimentation with Sun's dtmail (which has known bugs in
its Content-MD, but at least seems to get it right for attachments). I
gave it a shell script: it decided it was text plain, and put the CRs in
before computing the MD5. I then constructed a postscript file (draws
a little red circle) which, of course, on my Solaris system has lines
terminated with LF only. It recognised that application/postscript was
needed, it computed the MD5 on the LF version, and then encoded it in
base64 (which seems a neat way to pass the problem onto someone else).
But is it correct? And what if I choose to leave the encoding at 7bit?
Or if I receive an application/postscript in 7bit and want to check the
MD5?

There are two attachments to this message. One is that postscript in
base64, and the other is exactly the same file without encoding (I
may have some difficulty is persuading my system to send it without
encoding, and even if I do some intermediate transport agent might try
to munge it). Both have the same MD5, computed on the LF only. Assuming
that they arrive at your end without further encoding changes, I would
be interested to see whether you all agree on whether the MD5 is correct
in both cases, and also what happens if you save them both to a file, or
even try to display them.

Cc to Ian Bell at Turnpike, where they take Content-MD5 seriously.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

--Volery_of_Birds_372_000
Content-Type: APPLICATION/postscript; name="red-base64.ps"
Content-Transfer-Encoding: BASE64
Content-Description: red-base64.ps
Content-MD5: 6IAJtpF8/4UKYH/rdKw+Ig==

--Volery_of_Birds_372_000
Content-Type: APPLICATION/postscript; name="red-7bit.ps"
Content-Description: red-7bit.ps
Content-MD5: 6IAJtpF8/4UKYH/rdKw+Ig==

%!PS-Adobe-2.0
%%Title: /tmp/xfig-exp018065
%%Creator: fig2dev Version 3.2 Patchlevel 3c
%%CreationDate: Wed Apr 11 11:59:56 2001
%%For: chl@clerew (Charles Lindsey)
%%Orientation: Landscape
%%Pages: 1
%%BoundingBox: 0 0 595 842
%%BeginSetup
%%IncludeFeature: *PageSize A4
%%EndSetup
%%Magnification: 1.0000
%%EndComments
%%Ps2ps-Initialization
/@@initmatrix initmatrix matrix currentmatrix def
@@initmatrix setmatrix
save
newpath
0 842 moveto
0 0 lineto
595 0 lineto
595 842 lineto
closepath
clip
newpath
0.0 0.0 translate
90 rotate
1 -1 scale
save
%%Page: 1 1
10 setmiterlimit
0.06 0.06 scale
%%Page: 1 1
7.5 setlinewidth
newpath
900 825 translate
541 541 scale
0 0 1 0 360 arc
closepath
[0.0 -0.06 0.06 0.0 0.0 0.0 
] setmatrix [1 0 0 -1 0 0] concat @@initmatrix concat
gsave
1.0 0.0 0.0 setrgbcolor
1.0 0.0 0.0 setrgbcolor
eofill
grestore
gsave
1.0 0.0 0.0 setrgbcolor
stroke
grestore
restore
restore
showpage

--Volery_of_Birds_372_000--

From: "Rita de Groot" <[email-masked]>
To: <[email-masked]>
Subject: huidziekte
Date: Thu, 5 Apr 2001 11:55:51 +0200
Sender: [email-masked]

Vijf jaar voordat deze foto's werden genomen werd bij mij diagnose 
reumatische artritis gesteld en als zodanig ook behandeld. Niets hielp. 
Toen de ziekte zich zo manifesteerde zoals u hieronder kunt zien..........
http://www.naardedokter.com/testimonials/sys_lup_eryth.htm