Subject: Mangled Internet-Drafts

From: Albert Vezza <[email-masked]>
Date: Mon, 1 Nov 2004 19:09:16 -0500

Dear IETF Participants:

The Secretariat has received complaints that Internet-Drafts are 
being posted with formatting errors.  One problem involves the 
concatenation of files, which sometimes occurs when multiple files 
are submitted as attachments to a single email message.  In some 
cases, the attached files include two versions of the same I-D.  In 
other cases, the attached files include two different I-Ds.  Another 
problem involves the insertion of blank lines within an I-D.

The Secretariat has been trying to understand why these problems are 
occurring.  The insertion of blank lines is still under 
investigation.  However, we have performed some experiments that we 
believe shed some light on the concatenation issue.  Based on the 
results of these experiments, it appears that the concatenation of 
files stems from an incompatibility between the email composer 
software and the email reader software.

In our experiments, we sent and received test messages using various 
combinations of the email clients Eudora, Pine, Netscape, and Apple's 
Mail.  Each of the test messages included two text attachments. 
Please note that all four email clients use mime 1.0 and appear to 
correctly specify boundaries between body parts.

The results of these experiments are summarized below.  The number 
"1" indicates that both of the attachments were placed in-line, i.e., 
in the body of the email message, by the email reader software and 
were concatenated when viewed by the recipient.  The number "2" 
indicates that the email reader software identified the attachments 
by name.  And finally, the number "3" indicates that the email reader 
software created files of the attachments and placed them in an 
appropriate directory.  (The last issue is a receiver-side action 
that has no bearing on this issue.  It was included here only for 
completeness.)

Eudora to Eudora		2, 3
Eudora to Netscape		1, 2
Eudora to Pine		2
Eudora to Mail		2

Pine to Pine		2
Pine to Eudora		2, 3
Pine to Netscape		1, 2
Pine to Mail		2

Netscape to Netscape	1, 2
Netscape to Eudora		1
Netscape to Pine		1, 2
Netscape to Mail		1

Mail to Mail		2
Mail to Eudora		2, 3
Mail to Pine		1, 2
Mail to Netscape		1, 2

When a message was sent from Eudora to Eudora, from Pine to Eudora, 
or from Mail to Eudora, the attachments were placed in the 
recipient's "Attachments" directory, and the recipient was notified 
that the message contained two attachments.  Therefore, Eudora 
unwraps the files from the body when it is able to do so.

When a message was sent from Eudora to Netscape, from Pine to 
Netscape, from Netscape to Netscape, from Netscape to Pine, from Mail 
to Pine, or from Mail to Netscape, the attachments were concatenated 
within the body of the message.  However, the recipient was notified 
that the message contained two attachments and could easily find them 
within the message and save them.

When a message was sent from Eudora to Pine, from Eudora to Mail, 
from Pine to Pine, from Pine to Mail, or from Mail to Mail, the 
recipient was notified that the message contained two attachments and 
could save them.

But as you can see, when a message was sent from Netscape to either 
Eudora or Mail, the attachments were concatenated within the body of 
the message and the recipient was *not* notified that the message 
contained attachments.  Therefore, there was no easy way to identify 
and save the attachments other than by scanning the message and 
screen-scraping the relevant parts.

It would be interesting to see if the I-Ds that were concatenated 
were sent by a Netscape email composer, or by some other email 
composer that causes Eudora or Mail to not understand that some body 
parts should be unwrapped and identified as an attachment.  (One 
possible clue...and we say possible because we have been unable to 
prove it...is the fact that Netscape uses 12 leading dashes "--" as 
part of the boundary specification.)  If you have submitted multiple 
I-Ds that have been posted as a concatenation of I-Ds, then we would 
appreciate your letting us know what email composer you used when you 
submitted the I-Ds to the Secretariat.

The I-D administrator at the Secretariat uses Eudora as an email 
client.  When multiple I-Ds are placed in the body of a message, the 
only way that she can know that the submission contains more than one 
draft is if the sender indicates this in the message, or if the email 
reader software notifies her that there are multiple attachments. 
Until now, if neither of these events occurred, then the 
administrator assumed that the message contained a single I-D, and 
copied the entire contents of the message to a single file for 
posting.  We considered having the I-D administrator switch email 
clients from Eudora to Netscape.  However, there may be other email 
composers that are incompatible with Netscape and may cause similar 
problems in the future.

Suggestions for how to solve this problem are welcome.  Until we find 
a solution that corrects the problem without introducing other 
problems, the I-D administrator will manually check for multiple 
files within email messages whenever attachments are not specified. 
Needless to say, this is a time-consuming task, and will slow the 
processing of I-Ds.

Albert Vezza and Barbara Fuller
-- 
Albert Vezza
President
Foretec Seminars, Inc
1895 Preston White Drive
Suite 100
Reston, VA 20191-5434

Internet mail: [email-masked]
Tel: (703) 620-9053
Fax: (703) 620-9071

Natick, MA Office
Tel:  (508) 650 4020
Fax:  (508) 650 4639

---

From: Paul Hoffman / VPNC <[email-masked]>
Date: Tue, 2 Nov 2004 09:53:03 -0800

At 7:09 PM -0500 11/1/04, Albert Vezza wrote:
>Suggestions for how to solve this problem are welcome.

A couple of things come to mind:

- Keep an exact bit-copy of everything sent in on the server before 
delivering it to the mailbox; this is fairly easy with almost any 
mail server software running on a Unixy system. This is good practice 
for doing forensics after a mangle.

- Process the attachments with a Perl script, not with a mail client. 
There are a zillion Perl modules for this, and your script could go 
ahead and change the text into the desired format (proper line 
endings, form feeds, and so on). Heck, it could even run the ID-nits 
tool! :-) Use the mail client to see what the Perl script caught and 
check for sanity (does the file name match what they said they were 
submitting, and so on), but don't expect any mail client to not 
mangle incoming text files.

--Paul Hoffman, Director
--VPN Consortium

---

From: Rohan Mahy <[email-masked]>
Date: Tue, 2 Nov 2004 13:40:50 -0800

Hi,

My proposal is to use a web form to submit internet-drafts instead of 
an email.  The form can automatically check for appropriate formatting, 
boilerplate, and other ID nits, and return an immediate response 
indicating the success or failure of the submission. The form could 
also optionally check for WG chair approval for new WG IDs.  The drafts 
that pass the pre-check could be automatically included in the 
repository, or quarantined for inspection and release by the 
secretariat staff.

I realize that this approach has been discussed before and rejected for 
historical reasons.  I do not intend to start a debate on the merits of 
using web pages instead of email, merely to publicly indicate my 
support for the approach.

thanks,
-rohan

On Nov 1, 2004, at 16:09, Albert Vezza wrote:

> Dear IETF Participants:
>
> The Secretariat has received complaints that Internet-Drafts are being 
> posted with formatting errors.  One problem involves the concatenation 
> of files, which sometimes occurs when multiple files are submitted as 
> attachments to a single email message.  In some cases, the attached 
> files include two versions of the same I-D.  In other cases, the 
> attached files include two different I-Ds.  Another problem involves 
> the insertion of blank lines within an I-D.
>
> occurring.  The insertion of blank lines is still under investigation. 
>  However, we have performed some experiments that we believe shed some 
> light on the concatenation issue.  Based on the results of these 
> experiments, it appears that the concatenation of files stems from an 
> incompatibility between the email composer software and the email 
> reader software.
>
> Mail.  Each of the test messages included two text attachments. Please 
> note that all four email clients use mime 1.0 and appear to correctly 
> specify boundaries between body parts.
>
> The results of these experiments are summarized below.  The number "1" 
> indicates that both of the attachments were placed in-line, i.e., in 
> the body of the email message, by the email reader software and were 
> concatenated when viewed by the recipient.  The number "2" indicates 
> that the email reader software identified the attachments by name.  
> And finally, the number "3" indicates that the email reader software 
> created files of the attachments and placed them in an appropriate 
> directory.  (The last issue is a receiver-side action that has no 
> bearing on this issue.  It was included here only for completeness.)
>
> Eudora to Eudora		2, 3
> Eudora to Netscape		1, 2
> Eudora to Pine		2
> Eudora to Mail		2
>
> Pine to Pine		2
> Pine to Eudora		2, 3
> Pine to Netscape		1, 2
> Pine to Mail		2
>
> When a message was sent from Eudora to Eudora, from Pine to Eudora, or 
> from Mail to Eudora, the attachments were placed in the recipient's 
> "Attachments" directory, and the recipient was notified that the 
> message contained two attachments.  Therefore, Eudora unwraps the 
> files from the body when it is able to do so.
>
> When a message was sent from Eudora to Pine, from Eudora to Mail, from 
> Pine to Pine, from Pine to Mail, or from Mail to Mail, the recipient 
> was notified that the message contained two attachments and could save 
> them.
>
> It would be interesting to see if the I-Ds that were concatenated were 
> sent by a Netscape email composer, or by some other email composer 
> that causes Eudora or Mail to not understand that some body parts 
> should be unwrapped and identified as an attachment.  (One possible 
> clue...and we say possible because we have been unable to prove 
> it...is the fact that Netscape uses 12 leading dashes "--" as part of 
> the boundary specification.)  If you have submitted multiple I-Ds that 
> have been posted as a concatenation of I-Ds, then we would appreciate 
> your letting us know what email composer you used when you submitted 
> the I-Ds to the Secretariat.
>
> Until now, if neither of these events occurred, then the administrator 
> assumed that the message contained a single I-D, and copied the entire 
> contents of the message to a single file for posting.  We considered 
> having the I-D administrator switch email clients from Eudora to 
> Netscape.  However, there may be other email composers that are 
> incompatible with Netscape and may cause similar problems in the 
> future.
>
> problems, the I-D administrator will manually check for multiple files 
> within email messages whenever attachments are not specified. Needless 
> to say, this is a time-consuming task, and will slow the processing of 
> I-Ds.
>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 03 Nov 2004 00:15:21 +0100

Rohan,

can you please check out draft-ietf-tools-draft-submission-06?

I think other people have come to the same conclusion.... this is a topic 
for the General Area meeting at the upcoming IETF meeting.

                         Harald

--On tirsdag, november 02, 2004 13:40:50 -0800 Rohan Mahy 
<[email-masked]> wrote:

> Hi,
>
> My proposal is to use a web form to submit internet-drafts instead of an
> email.  The form can automatically check for appropriate formatting,
> indicating the success or failure of the submission. The form could also
> optionally check for WG chair approval for new WG IDs.  The drafts that
> pass the pre-check could be automatically included in the repository, or
> quarantined for inspection and release by the secretariat staff.
>
> using web pages instead of email, merely to publicly indicate my support
> for the approach.
>
> thanks,
> -rohan
>
>
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 03 Nov 2004 01:14:35 +0100

Hi Rohan,

    The tools team has been working on specifying such a web interface.
Maybe you'd like to review the draft:

http://www.ietf.org/internet-drafts/draft-ietf-tools-draft-submission-06.txt

	Henrik

---

From: Paul Hoffman / VPNC <[email-masked]>
Date: Tue, 2 Nov 2004 17:09:53 -0800

At 7:12 PM -0500 11/2/04, Albert Vezza wrote:
>Thanks we will look into that.  We need to be a little careful as 
>the ID administrator is not a technical person and needs standard 
>type tools to process IDs.

That's fine, but the documents can be pre-processed for her before 
her "standard type tools" even see the email message. That way, the 
Secretariat's tools are in charge of capture and conversion, not the 
mail program she happens to be running.

--Paul Hoffman, Director
--VPN Consortium

---

From: Albert Vezza <[email-masked]>
Date: Tue, 2 Nov 2004 19:12:11 -0500

Thanks we will look into that.  We need to be a little careful as the 
ID administrator is not a technical person and needs standard type 
tools to process IDs.

At 9:53 -0800 11/2/04, Paul Hoffman / VPNC wrote:
>mail server software running on a Unixy system. This is good 
>practice for doing forensics after a mangle.
>
>- Process the attachments with a Perl script, not with a mail 
>client. There are a zillion Perl modules for this, and your script 
>could go ahead and change the text into the desired format (proper 
>line endings, form feeds, and so on). Heck, it could even run the 
>ID-nits tool! :-) Use the mail client to see what the Perl script 
>caught and check for sanity (does the file name match what they said 
>they were submitting, and so on), but don't expect any mail client 
>to not mangle incoming text files.
>
>
>--Paul Hoffman, Director
>--VPN Consortium

-- 
Albert Vezza
President
Foretec Seminars, Inc
1895 Preston White Drive
Suite 100
Reston, VA 20191-5434

Natick, MA Office
Tel:  (508) 650 4020
Fax:  (508) 650 4639

---

From: Albert Vezza <[email-masked]>
Date: Wed, 3 Nov 2004 17:53:16 -0500

Thanks to Bill Fenner and Tony Hansen who both hit the nail on the 
head.  We will look at putting a filter in-line with the mail stream 
to change "Content-disposition: inline" to "Content-disposition: 
attachment".  This should fix the concatenated ID problem for email 
composers that use the "Content-disposition:" field but specify 
"inline".  We will also have to figure out how to add 
"Content-disposition: attachment" to MIME parts that don't have any 
Content-disposition: header.

Fortuitously, this has a high probability of also fixing the added 
linefeed and missing ^L problem, because we have only been able to 
observe the phenomena when the ID administrator gets the attachment 
in-line instead of as an attachment.  But, we will have to wait and 
see if it crops up after we have fixed the multiple attachment 
problem to be sure.  In any case, this problem seems to be caused by 
some interplay between Eudora and Notepad.

  Al

At 8:41 -0700 11/3/04, Bill Fenner wrote:
>My guess is that the problematic messages use
>"Content-disposition: inline" for the attachments.
>If it's possible to insert a filter into the message
>stream, I'd bet that changing "Content-disposition: inline"
>to "Content-disposition: attachment" (and perhaps adding
>"Content-disposition: attachment" to MIME parts that
>don't have any Content-disposition: header) would fix
>the problem.
>
>   Bill

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 04 Nov 2004 11:41:42 +0100

--On 3. november 2004 17:53 -0500 Albert Vezza <[email-masked]> wrote:

> Thanks to Bill Fenner and Tony Hansen who both hit the nail on the head.
> We will look at putting a filter in-line with the mail stream to change
> "Content-disposition: inline" to "Content-disposition: attachment".  This
> should fix the concatenated ID problem for email composers that use the
> "Content-disposition:" field but specify "inline".

Good!

> We will also have to
> figure out how to add "Content-disposition: attachment" to MIME parts
> that don't have any Content-disposition: header.

You don't have to do that - most readers assume "attachment" when there is 
no content-disposition header.

                Harald