Subject: [ietf-smtp] Proper definition of the term "email payload".

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Sun, 31 Mar 2019 23:27:30 +0530

Hello IETF,

I need some clarification about the term "email payload".

Wikipedia <https://en.wikipedia.org/wiki/Payload_(computing)> says

*In computing and telecommunications, the payload is the part of
transmitted data that is the actual intended message. Headers and metadata
are sent only to enable payload delivery*

Python email library documentation
<https://docs.python.org/3/library/email.message.html> says this.

*An email message consists of headers and a payload (which is also referred
to as the content). Headers are RFC 5322 or RFC 6532 style field names and
values, where the field name and value are separated by a colon. The colon
is not part of either the field name or the field value. The payload may be
a simple text message, or a binary object, or a structured sequence of
sub-messages each with their own set of headers and their own payload. The
latter type of payload is indicated by the message having a MIME type such
as multipart/* or message/rfc822.*

It looks like Python email library
<https://github.com/python/cpython/blob/3.7/Lib/email/message.py>author "Barry
Warsaw <https://barry.warsaw.us/>" followed similar definition found in
wikipedia when defining his library functions. But I feel like calling ONLY
the email "Body Part" as "payload" is wrong. The term "payload" should
refer to the entire "Message Part" in Email. i.e. Both Headers and Body.

When you place an order for a "box of beer", you are not paying only for
the "beer cans", but also paying for the "container box". So the payload
here is the entire box.

HTTP Example:

HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Content-Type: text/html
Content-Length: 1234

<html>

<head>
<title>Hello World!</title>
</head>

<body>
(more contents)
  .
  .
  .
</body>
</html>

If you take a closer look at this HTTP example, the headers are only just
instructions for the client. The end user doesn't need to worry about any
piece of information found in those headers. So wikipedia definition
perfectly suited for applications like HTTP.

But in Email, When a mail get transferred from Hop A to Hop C via Hop B,
the user in Hop A actually wants to deliver the whole "message part" to Hop
C. If Hop B, strips the headers and transfer only the "Body" part, then it
becomes an "Anonymous" message. So the end user requires the information
found in the "Headers" too. e.g. From, Subject, Date etc. [In HTTP, title
tag is equivalent to Subject and it's found in the "head" Markup, not in
the HTTP Headers]

As you can see, the user is interested in the "entire message". So the term
"actual intended message" should refer to the "whole message" extracted
from the DATA command. The "actual intended message" should be pictured like
this <https://www.dropbox.com/s/2i6mec5fzgu1v6n/messagepart1.png?dl=0> in
email.

Also note that, when you migrate your mails to another mail service, you
need the whole message with Headers, not just the body.

Based on my points, I believe calling only the "Body" part as "Payload" is
wrong. I would love to hear your thoughts on this. If Barry Warsaw is here,
would love to know your opinion too.

PS: I did actually ask this question 2 years back in a stackexchange website
<https://softwareengineering.stackexchange.com/questions/342184/what-is-an-email-payload>.
I wasn't satisfied with the answer I got there. I don't want to use the
term incorrectly in my application. That's why I'm posting it here.

Thanks
-- 
Best Regards,

Viruthagiri Thirumavalavan
Dombox, Inc.

---

From: Keith Moore <[email-masked]>
Date: Sun, 31 Mar 2019 14:10:08 -0400

> I need some clarification about the term "email payload".

IMO, the term "email payload" is not well-defined, and is therefore 
ambiguous and should not be used whenever precision is needed.

The terms header and body are well-defined.

Keith

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Sun, 31 Mar 2019 23:55:00 +0530

Thanks Mr. Moore for the input.

The well known Python software already using that term. I don't want others
like me to follow the wrong terminology.

If I'm wrong that's fine. But If I'm right, we should find a way to define
the term properly. e.g. Editing that wikipedia article.

Thanks

On Sun, Mar 31, 2019 at 11:40 PM Keith Moore <[email-masked]>
wrote:

> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

-- 
Best Regards,

---

From: Keith Moore <[email-masked]>
Date: Sun, 31 Mar 2019 14:30:15 -0400

On 3/31/19 2:25 PM, Viruthagiri Thirumavalavan wrote:

> The well known Python software already using that term. I don't want 
> others like me to follow the wrong terminology.
>
> If I'm wrong that's fine. But If I'm right, we should find a way to 
> define the term properly. e.g. Editing that wikipedia article.
>
> Thanks

Somebody can edit the Wikipedia article if they want.   But we in IETF 
don't have the power to make everyone use terms consistently.  It's hard 
enough to get people to read our protocol specifications and implement 
them correctly.

Ketih

p.s. I still remember being part of a crowd insisting that the term 
"hacker" referred only to a particular style of working and implied no 
malice at all.  But the media insisted on using the term "hacker" to 
mean "someone who tries to break into someone else's computers" and they 
effectively changed the meaning of the term.

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Mon, 1 Apr 2019 00:02:17 +0530

Ok, thanks for the clarification.

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Mon, 1 Apr 2019 05:57:39 +0530

Thanks Mr. Warsaw for taking your time and explaining this. Yes I would
love to hear RDM and Mark opinions.

On Mon, Apr 1, 2019 at 5:39 AM Barry Warsaw <[email-masked]> wrote:

> Hi, I hope you (and they!) don’t mind me CCing two other people who have
> worked extensively on Python’s email library, and in fact much more than
> myself in the recent years.  RDM has done the bulk of the work on the
> new-in-Python-3 APIs, and Mark is a long time core developer on GNU Mailman
> (the project that spawned Python’s email library).
>
> There are two ways I think about this, and I’ll use the original RFC
> numbers to clarify.  There’s RFC 821, which describes the on-the-wire
> protocol for SMTP transfers, embodied in Python’s smtplib library.  Then
> there’s RFC 822, which describes the format of the content of that SMTP
> transfer, but not the protocol itself.  Of course there are lots of
> developments along the way, but that’s unimportant for the way I think
> about these things.
>
> What I think you are describing, where the headers are part of the
> payload, is more akin to RFC 821.  That’s the payload as far as the actual
> bytes-on-the-wire are concerned.  Python’s email library is for RFC 822
> (and the many, many elaborations thereof), so in that case, the payload is
> the body of the message.  On more practical terms, the implementation makes
> this clear, and the APIs you use to change headers are different in form
> and function than the ones you use to change the body of the message.
>
> I think the Python documentation is fairly clear about this distinction.
> At least, I don’t remember seeing any feedback to the contrary, although
> RDM may have a better sense of that.  Of course, we are always open to
> improvements in Python’s documentation.
>
> Cheers,
> -Barry
>
> > On Mar 31, 2019, at 10:57, Viruthagiri Thirumavalavan <[email-masked]>
> wrote:
> >
> > Hello IETF,
> >
> > Wikipedia says
> > In computing and telecommunications, the payload is the part of
> are sent only to enable payload delivery
> >
> > Python email library documentation says this.
> >
> > An email message consists of headers and a payload (which is also
> referred to as the content). Headers are RFC 5322 or RFC 6532 style field
> names and values, where the field name and value are separated by a colon.
> The colon is not part of either the field name or the field value. The
> payload may be a simple text message, or a binary object, or a structured
> sequence of sub-messages each with their own set of headers and their own
> payload. The latter type of payload is indicated by the message having a
> MIME type such as multipart/* or message/rfc822.
> >
> > It looks like Python email library author "Barry Warsaw" followed
> similar definition found in wikipedia when defining his library functions.
> But I feel like calling ONLY the email "Body Part" as "payload" is wrong.
> The term "payload" should refer to the entire "Message Part" in Email. i.e.
> Both Headers and Body.
> >
> > If you take a closer look at this HTTP example, the headers are only
> just instructions for the client. The end user doesn't need to worry about
> any piece of information found in those headers. So wikipedia definition
> > As you can see, the user is interested in the "entire message". So the
> term "actual intended message" should refer to the "whole message"
> extracted from the DATA command. The "actual intended message" should be
> pictured like this in email.
> >
> > Based on my points, I believe calling only the "Body" part as "Payload"
> is wrong. I would love to hear your thoughts on this. If Barry Warsaw is
> here, would love to know your opinion too.
> >
> > PS: I did actually ask this question 2 years back in a stackexchange
> website. I wasn't satisfied with the answer I got there. I don't want to
> use the term incorrectly in my application. That's why I'm posting it here.
> >
> > Thanks
> > --
> > Best Regards,
> >

-- 
Best Regards,

---

From: Dave Crocker <[email-masked]>
Date: Sun, 31 Mar 2019 17:42:46 -0700

> I need some clarification about the term "email payload".

The term 'payload' is not a formal term of art in email.  The word 
appears only once in RFC 5598, and not in a context that concerns your 
topic of interest.  It does not appear at all in RFC 5321 or RFC 5322 or 
RFC 3501.

Considering the message body as the payload is reasonable in some cases, 
but not in others, because there is quite a bit of user-to-user 
information in some of the message header fields.

In SMTP context, it's probably safe to say that everything after the 
DATA command is payload.  Certainly in terms of typical use of the word 
payload in protocol discussions, I think that's a straightforward 
assessment.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Mon, 1 Apr 2019 06:17:15 +0530

Thanks for the clarification.

-- 
Best Regards,

---

From: Barry Warsaw <[email-masked]>
Date: Sun, 31 Mar 2019 17:09:30 -0700

Hi, I hope you (and they!) don’t mind me CCing two other people who have worked extensively on Python’s email library, and in fact much more than myself in the recent years.  RDM has done the bulk of the work on the new-in-Python-3 APIs, and Mark is a long time core developer on GNU Mailman (the project that spawned Python’s email library).

There are two ways I think about this, and I’ll use the original RFC numbers to clarify.  There’s RFC 821, which describes the on-the-wire protocol for SMTP transfers, embodied in Python’s smtplib library.  Then there’s RFC 822, which describes the format of the content of that SMTP transfer, but not the protocol itself.  Of course there are lots of developments along the way, but that’s unimportant for the way I think about these things.

What I think you are describing, where the headers are part of the payload, is more akin to RFC 821.  That’s the payload as far as the actual bytes-on-the-wire are concerned.  Python’s email library is for RFC 822 (and the many, many elaborations thereof), so in that case, the payload is the body of the message.  On more practical terms, the implementation makes this clear, and the APIs you use to change headers are different in form and function than the ones you use to change the body of the message.

I think the Python documentation is fairly clear about this distinction.  At least, I don’t remember seeing any feedback to the contrary, although RDM may have a better sense of that.  Of course, we are always open to improvements in Python’s documentation.

Cheers,
-Barry

> On Mar 31, 2019, at 10:57, Viruthagiri Thirumavalavan <[email-masked]> wrote:
> 
> Hello IETF,
> 
> In computing and telecommunications, the payload is the part of transmitted data that is the actual intended message. Headers and metadata are sent only to enable payload delivery
> 
> An email message consists of headers and a payload (which is also referred to as the content). Headers are RFC 5322 or RFC 6532 style field names and values, where the field name and value are separated by a colon. The colon is not part of either the field name or the field value. The payload may be a simple text message, or a binary object, or a structured sequence of sub-messages each with their own set of headers and their own payload. The latter type of payload is indicated by the message having a MIME type such as multipart/* or message/rfc822.
> 
> It looks like Python email library author "Barry Warsaw" followed similar definition found in wikipedia when defining his library functions. But I feel like calling ONLY the email "Body Part" as "payload" is wrong. The term "payload" should refer to the entire "Message Part" in Email. i.e. Both Headers and Body.
> 
> When you place an order for a "box of beer", you are not paying only for the "beer cans", but also paying for the "container box". So the payload here is the entire box.
> 
> HTTP Example:
> 
> HTTP/1.1 200 OK
> If you take a closer look at this HTTP example, the headers are only just instructions for the client. The end user doesn't need to worry about any piece of information found in those headers. So wikipedia definition perfectly suited for applications like HTTP.
> 
> But in Email, When a mail get transferred from Hop A to Hop C via Hop B, the user in Hop A actually wants to deliver the whole "message part" to Hop C. If Hop B, strips the headers and transfer only the "Body" part, then it becomes an "Anonymous" message. So the end user requires the information found in the "Headers" too. e.g. From, Subject, Date etc. [In HTTP, title tag is equivalent to Subject and it's found in the "head" Markup, not in the HTTP Headers]
> 
> As you can see, the user is interested in the "entire message". So the term "actual intended message" should refer to the "whole message" extracted from the DATA command. The "actual intended message" should be pictured like this in email.
> 
> Also note that, when you migrate your mails to another mail service, you need the whole message with Headers, not just the body.
> 
> Based on my points, I believe calling only the "Body" part as "Payload" is wrong. I would love to hear your thoughts on this. If Barry Warsaw is here, would love to know your opinion too.
> 
> PS: I did actually ask this question 2 years back in a stackexchange website. I wasn't satisfied with the answer I got there. I don't want to use the term incorrectly in my application. That's why I'm posting it here.
> 
> Thanks
> --
> Best Regards,
>