Subject: [marf] [MARF] Additional MIME subparts

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Tue, 02 Feb 2010 07:00:17 -0500

I'd like to see provision for additional MIME parts of multipart/report;
report-type=feedback-report. Specifically, I'd like to relegate supporting
documentation[1] to a separate MIME part. I'd also like for those
additional part to be identified by, e.g., their MIME type, rather than by
their position withing the multipart/report. Finally, I'd like an IETF
registry of the defined additional parts. If the group concurs, "4. 
Extensibility" should discuss the new registry.

[1] E.g., whois data. Note that while in theory the provider could do
    his own, e.g., traceroute, relevant data might have changed in
    the intervening period.

-- 
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2        <http://patriot.net/~shmuel>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

---

From: Steve Atkins <[email-masked]>
Date: Wed, 3 Feb 2010 05:11:21 -0800

>    his own, e.g., traceroute, relevant data might have changed in

If you want to add unstructured data (with the understanding that
it's fairly unlikely it'll ever be read) there's already a place for that -
the first, probably text/plain, MIME part.

Cheers,
  Steve

---

From: Yakov Shafranovich <[email-masked]>
Date: Wed, 3 Feb 2010 10:47:09 -0500

This standard is based off RFC 3462 which is limited to three MIME
parts: first for human readable, second for machine readable, and
third for the actual message. This keeps the standard simple. Before
unraveling that, perhaps you can provide some use cases where it would
be necessary to have more than one MIME subpart?

Additionally, as stated in the draft, the standard is focused on one
message/one report paradigm. It does not support reporting for
multiple messages in a single ARF report, this is out of scope.

Third, as I mentioned once before, the logic behind the MIME fields in
the middle section is as follows - the enclosed spam message
constitutes prima facie or self evident evidence of the spam or virus
occurrence. The fields in the middle section contain data that cannot
be easily derived from the original spam message such as "Source-IP"
which may not be included in the "Received" headers,
"Original-Mail-From" which is part of the SMTP transaction. The fields
"Reported-Domain" and "Reported-URI" are used to indicate to the
receiver which specific data within the message triggered the report.
Again, this is not data easily derived from the message itself since
it requires knowledge of what the originating system did.

Yakov

On Tue, Feb 2, 2010 at 7:00 AM, Shmuel  Metz
<[email-masked]> wrote:
> _______________________________________________
> marf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/marf
>

---

From: "J.D. Falk" <[email-masked]>
Date: Wed, 3 Feb 2010 13:18:05 -0700

> the first, probably text/plain, MIME part.

Agreed.  I don't see much value in adding a new part for data that can't be automatically parsed when we already have one designated for that purpose.

--
J.D. Falk <[email-masked]>
Return Path Inc

---

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Wed, 03 Feb 2010 20:07:06 -0500

In <[email-masked]>, on
02/03/2010
   at 10:47 AM, Yakov Shafranovich <[email-masked]> said:

>message/one report paradigm.

As was my suggestion.

>The fields in the middle section contain data that cannot

Traces and such at the time of the spam may not be reproducible if there
are, e.g., subsequent DNS changes. The intent was to provide a mechanism
for giving point in time evidence. The second section of the report does
not provide for that.

---

From: Yakov Shafranovich <[email-masked]>
Date: Wed, 3 Feb 2010 20:07:09 -0500

> In <[email-masked]>, on

What sort of evidence do you have in mind?

---

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Fri, 05 Feb 2010 11:00:57 -0500

In <[email-masked]>, on
02/03/2010

>What sort of evidence do you have in mind?

Output from dig, nslookup, tracerte and whois. Redirection. I've seen
posts in the past about situations where the evidence at the time of abuse
was not reproducible even a day later.

---

From: "John R Levine" <[email-masked]>
Date: 5 Feb 2010 17:26:28 -0500

> was not reproducible even a day later.

Why wouldn't you put that in the initial plain text section?

I don't think it's plausible to try to define a parsed container format 
for every possible bit of evidence.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Steve Atkins <[email-masked]>
Date: Fri, 5 Feb 2010 14:33:12 -0800

On Feb 5, 2010, at 2:26 PM, John R Levine wrote:

> I don't think it's plausible to try to define a parsed container format for every possible bit of evidence.

Or send it manually.

ARF is intended for reports that are likely to be processed mechanically,
resulting from reports that are generated mechanically.

If your report requires all that data to be of value, you're not providing
much added value by sending it in ARF format. If it doesn't, you shouldn't
be including that data.

ARF isn't... optimal for unsolicited reports to role accounts in many
respects. This is one of them.

Cheers,
 Steve

---

From: Steve Atkins <[email-masked]>
Date: Fri, 5 Feb 2010 14:31:50 -0800

Or send it manually.

Cheers,
  Steve

---

From: Yakov Shafranovich <[email-masked]>
Date: Sat, 6 Feb 2010 19:22:52 -0500

> every possible bit of evidence.

I think he is referring to a different issue - encapsulation. Being
limited to three MIME parts, does not allow inclusion of files or
binary data as an additional MIME part - like we do with regular email
messages.

---

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Sun, 07 Feb 2010 07:23:50 -0500

In <[email-masked]>, on
02/06/2010
   at 07:22 PM, Yakov Shafranovich <[email-masked]> said:

>I think he is referring to a different issue - encapsulation. 

Indeed. The results of, e.g., whois, often include non-ASCII data. Similar
issues may exist for other types of evidence. I hadn't thought of the
possible requirement to provide binary files, but that might also be an
issue.

---

From: "John R Levine" <[email-masked]>
Date: 7 Feb 2010 17:15:48 -0500

> possible requirement to provide binary files, but that might also be an

ARF reports are in most cases handled by software, with no human ever 
seeimg them.  Can you describe the scenario in which all of this extra 
stuff would be useful?

Regards,

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 7 Feb 2010 17:51:34 -0500

> In <[email-masked]>, on

I don't think this is a problem at all. The following is a quote from
RFC 3462, which defines multipart/report used by ARF:

" The text in the first section may be in any MIME standards-track
   content-type, charset, or language.  Where a description of the error
   is desired in several languages or several media, a
   Multipart/Alternative construct may be used.
"

Note the last sentence - multipart/alternative can be used to embed
additional MIME parts inside the first part of a multipart/report. I
don't see a reason why multipart/mixed cannot be used as well which
will give you ability to embed files. I would expand the MARF draft,
in section 2 as follows:

"b.  The first MIME part of the message contains a human readable
description of the report and MUST be included. As described in
section 1 of RFC 3462, the human readable section of a
multipart/report may be set to any valid MIME type. Implementors also
have ability to embed additional information and binary files if
necessary by using the multipart/mixed MIME type. The format and
interpretation of data in this section is for intended for human
operators as per RFC 3462 and is not defined in this standard.
"

Yakov

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 7 Feb 2010 18:13:06 -0500

> operators as per RFC 3462 and is not defined in this standard.

One other thing - this may be a very obscure way to use
multipart/report and is probably that did not come up with DSNs and
MDNs. We might want to look into the possibilities of writing up a
"best practices" document after ARF is standardized.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 8 Feb 2010 08:49:56 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Yakov Shafranovich
> Sent: Sunday, February 07, 2010 3:13 PM
> To: [email-masked]
> Subject: Re: [marf] [MARF] Additional MIME subparts
> 

That's an explicit part of this working group's charter (item #2).

---

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Mon, 08 Feb 2010 08:29:47 -0500

In <[email-masked]>, on 02/07/2010
   at 05:15 PM, "John R Levine" <[email-masked]> said:

>To: "Shmuel (Seymour J.) Metz"
><[email-masked]> Cc: [email-masked]

Please send only a single copy. If there is a reason that you must send an
off-list copy, please annotate it as a duplicate so that I will know to
respond to the list rather than only to you. Thanks.

>seeimg them.  Can you describe the scenario in which all of this extra 

I see a MIME part for evidence as useful only when the report is routed to
a human being. I don't see it as directly relevant to automated
processing.

---

From: "John R Levine" <[email-masked]>
Date: 8 Feb 2010 13:25:20 -0500

> a human being. I don't see it as directly relevant to automated

ARF is intended for automated processing, so it seems out of scope.

If you want to send the WHOIS dump, you can always mail a message in 
whatever form you want to abuse@wherever.

Regards,

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 8 Feb 2010 10:30:48 -0800

> John R Levine
> Sent: Monday, February 08, 2010 10:25 AM
> To: Message Abuse Report Format working group

By that logic, shouldn't we mandate that the first (text/plain) MIME part always be empty?

---

From: Steve Atkins <[email-masked]>
Date: Mon, 8 Feb 2010 10:43:41 -0800

> By that logic, shouldn't we mandate that the first (text/plain) MIME part always be empty?

While ARF is intended for automated processing, we don't want
to lock out tiny ISPs who might get 5 FBL reports a week from being
able to do something useful with them using a regular MUA.

Also even if an ARF consumer is handling inbound ARF format reports
automatically, there's always going to be the case where a human needs
to put eyes on them, for diagnostics.

But as <hand wave> 99% of ARF reports will likely be handled entirely
automatically, there should not be anything in the human readable
section that's essential for handling the report.

c.f. Received headers or MIME preamble or the Organization or
User-Agent headers. They're useful if you need them for diagnostics,
but they're almost never looked at, and don't contain anything needed
for reading the message.

That means that if you're adding information that needs to be looked
at by a human in order for a FBL report to be actionable you're doing
it wrong. But dropping a brief message or a URL or somesuch in the
human readable part such that it can be handled without automation
causes no harm and will be occasionally helpful to a FBL consumer.

Cheers,
 Steve

---

From: Steve Atkins <[email-masked]>
Date: Mon, 8 Feb 2010 10:41:43 -0800

Cheers,
  Steve

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 8 Feb 2010 10:53:04 -0800

> Sent: Sunday, February 07, 2010 2:52 PM
> To: [email-masked]
> will give you ability to embed files.

[as participant]

I don't think this conforms to the general intent of RFC3462, which to me fairly clearly intended the first MIME part to contain a human-readable description of what's in the other parts.  Using multipart/alternative there is consistent with that, but using multipart/mixed to pack parts of the actual report into it seems to deviate from that intent.

The part where we can define extensions of the nature we're discussing, within the RFC3462 framework, is the second part.  If we want to accommodate additional data outside the "name: value" format we have now, we'd have to say that message/feedback-report is itself a multipart and then establish the requisite structure around that.

Given that one of the chartered goals is to maintain backward compatibility with existing deployments, I suspect we can't accommodate anything that complex until we're discussing ARF extensions in later draft work.  But this is something we should revisit at that time.

---

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 8 Feb 2010 19:14:32 -0500

> I don't think this conforms to the general intent of RFC3462, which to me fairly clearly intended the first MIME part to contain a human-readable description of what's in the other parts.  Using mu…

I would have to say that multipart/mixed is how regular email messages
right now relay attachments, and I don't think it is far fetched to
allow this in ARF.

> Given that one of the chartered goals is to maintain backward compatibility with existing deployments, I suspect we can't accommodate anything that complex until we're discussing ARF extensions in…

Perhaps. I do wonder if multipart/mixed has ever been used with DSNs.

---

From: "J.D. Falk" <[email-masked]>
Date: Tue, 9 Feb 2010 13:39:07 -0700

> In <[email-masked]>, on

Are there any ARF recipients interested in receiving "attached" metadata like this?  Perhaps you could define a local extension with them, see how it works out, and report back to the working group.