Subject: Re: [Last-Call] New Version Notification for draft-crocker-inreply-react-07.txt

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 18:41:06 +0100

> On 25 Feb 2021, at 17:59, Dave Crocker <[email-masked]> wrote:
> 
> Sorry, no.  Copying specification details invites divergence.  And the style of inclusion by reference done in this specification is not that unusual.

Experience with Unicode have shown that is exactly what we need.

What I am after is not *copying* from UT#51. I want the subset of what is in UT#51 that suits this spec be expressed in ABNF like the rest of the specification. This to ensure interoperability to the degree IETF feel comfortable with.

My point is that IETF will not be able to make that determination unless the specification of what is valid and not is expressed in the way IETF can evaluate it. I specifically think about sequences including ZWJ, different directionality in contexts of multiple emoji etc.

This is also why I suggest starting with a very limited set of code points and then later opening up. So not the same mistake is made as we made with domain names, which are as you say a different animal. But where issues with sequences still exists.

  Patrik

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 09:47:29 -0800

> Experience with Unicode have shown that is exactly what we need.

Patrik,

Please do not use an 'appeal to authority' approach for this exchange.

Please substantiate your assertions. What experience, exactly? Where is 
it documented?

> What I am after is not *copying* from UT#51. I want the subset of what is in UT#51 that suits this spec be expressed in ABNF like the rest of the specification. This to ensure interoperability to t…

Please explain why a subset is necessary and appropriate.

> My point is that IETF will not be able to make that determination unless the specification of what is valid and not is expressed in the way IETF can evaluate it. I specifically think about sequence…

The current draft specifies what is valid.  So far, I do not understand 
your belief that it doesn't.

> This is also why I suggest starting with a very limited set of code points and then later opening up. So not the same mistake is made as we made with domain names, which are as you say a different…

The context of use for reactions is entirely different from domain names.

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

Volunteer, Silicon Valley Chapter
American Red Cross
[email-masked]

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 18:54:27 +0100

> On 25 Feb 2021, at 18:48, Dave Crocker <[email-masked]> wrote:
> 
> Please substantiate your assertions. What experience, exactly? Where is it documented?

This is one example of a description where versioning in Unicode is described being different than versioning in IETF.

https://datatracker.ietf.org/doc/draft-faltstrom-unicode12/

This will be updated as soon as I have some more time.

This is why I recommend having one ietf version separated from the Unicode version of “the same thing”, although of course I from a general stand point agree with you on the risk of divergence.

    Patrik

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 10:06:00 -0800

On 2/25/2021 9:54 AM, Patrik Fältström wrote:
>
> This is one example of a description where versioning in Unicode is 
> described being different than versioning in IETF.
>
> <https://datatracker.ietf.org/doc/draft-faltstrom-unicode12/?include_text=1>
>

Patrik,

You are claiming there are serious problems likely with this draft.

You appear to feel it is essential to make this specification 
substantially more complex in content, more complex in administration, 
and more complex to use.

However you appear to be relying on your own, personal draft that hasn't 
been updated for two years, as the sole authority for this concern.

In spite of the 6-week period, since the Last Call was issued, no one 
else has expressed similar (or even related) concerns.

Also, the fact that your draft has lingered for so long does not 
encourage a view that the community deems it essential.

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 19:14:34 +0100

> On 25 Feb 2021, at 19:06, Dave Crocker <[email-masked]> wrote:
> 
> You appear to feel it is essential to make this specification substantially more complex in content, more complex in administration, and more complex to use.

The draft will be harder to administrate, yes, but it will be much easier to implement as people do not have to parse and understand the ins and out of UT#51.

Regarding the comments of my draft, as you very obviously have not followed the Unicode work in ietf, I do not find it worth addressing thorn. They also have nothing to do with this draft of yours.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 19:20:17 +0100

Now, we have a consensus process in ietf. And now I have stated my view. I understand many issues with cultural /technical tests will be clarified in a future draft and I hope that’s excellent.

The reference to UT#51...well I have said my view. Maybe a minority view.

   Patrik

---

From: Adam Roach <[email-masked]>
Date: Thu, 25 Feb 2021 12:28:06 -0600

> The context of use for reactions is entirely different from domain names. 

Without wading into the other points being made: I agree, and I think 
this is the winning argument. The complications that arise from Unicode 
in domain names are almost exclusively due to the fact that domain names 
are used as *identifiers* that are input to a lookup system instead of 
simply being displayed.

In the context that inreply-react describes, Unicode is used exactly and 
precisely the same way that Unicode is used in normal email bodies: 
these are glyphs that are meant to be displayed to a user. They're not 
database keys; lookalike attacks are of no consequence; normalization 
forms on the wire are fungible. None of the IDN issues arise, and so 
analogy to them does not hold. The considerations for these display 
elements need to be exactly as complex as the display considerations for 
Unicode in email bodies, and no more.

/a

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 19:34:16 +0100

> On 25 Feb 2021, at 19:29, Adam Roach <[email-masked]> wrote:
> 
> The considerations for these display elements need to be exactly as complex as the display considerations for Unicode in email bodies, and no more.

Then why does it not have the same characteristics as any mime body part with Unicode content?

It is exactly where the boundary is between code points that are allowed and not you have the key issues here.

The draft DO refer to very specific rules that are to be followed.

If it was not, I would not be as worried.

   Patrik

---

From: Adam Roach <[email-masked]>
Date: Thu, 25 Feb 2021 12:44:19 -0600

> If it was not, I would not be as worried.

So you'd be okay if the document just removed the codepoint restrictions 
and instead limited it to one line which is intended to convey a 
pictographic reaction? That seems like a reasonable compromise, as it 
would allow popular reactions like "¯\_(ツ)_/¯" and "(╯°□°)╯︵ ┻━┻" 
without needing a later revision.

/a

---

From: Barry Leiba <[email-masked]>
Date: Thu, 25 Feb 2021 14:11:16 -0500

Patrik, as I understand our off-list conversation, a couple of minor
text changes would satisfy you that the document is being clear about
these issues.  Is it possible for you to suggest specific text that
the authors can consider?  I'm sure the authors will be happy to add
some text that clarifies things, and I'll hold the final approval on
the document briefly, to allow for that.

Barry

On Thu, Feb 25, 2021 at 1:35 PM Patrik Fältström
<paf=[email-masked]> wrote:
> last-call mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/last-call

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 20:31:37 +0100

> On 25 Feb 2021, at 19:45, Adam Roach <[email-masked]> wrote:
> 
> So you'd be okay if the document just removed the codepoint restrictions and instead limited it to one line which is intended to convey a pictographic reaction? That seems like a reasonable compromise, as it would allow popular reactions like "¯\_(ツ)_/¯" and "(╯°□°)╯︵ ┻━┻" without needing a later revision.

Yes. Given you say what you said, same issues as any text (directionality, zwj, ...).

It’s the reference to a limitation of what can go in that field and not which I claim (and others obviously disagree) is neither easy to parse, nor stable enough for IETF.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 25 Feb 2021 21:12:04 +0100

I am told by Dave the old and new version is addressing my comments.

I am not judging that myself.

My comment as liaison to Unicode is addressed by Adam (or similar fix). The issue I saw is understood.

I am now not going to comment more on this document.

   Patrik

> On 25 Feb 2021, at 20:11, Barry Leiba <[email-masked]> wrote:
>

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 12:29:01 -0800

> I am told by Dave the old and new version is addressing my comments.

I did?

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 25 Feb 2021 21:34:29 +0100

>> I am told by Dave the old and new version is addressing my comments.

That the text in existing or coming document is already dealing with what I brought up.

I am no longer having any comments on the document, and will work on other things.

   Patrik

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 25 Feb 2021 22:05:28 +0100

On Thu, 2021-02-25 at 12:44 -0600, Adam Roach wrote:
> > It is exactly where the boundary is between code points that are
> > allowed and not you have the key issues here.
> > 

This was partially one of the points I made earlier.  The draft is
eerily silent on what to do with a reaction like "J".  It is not an
unallocated code point, but it is not a valid emoji either.  When I
brought it up, Dave seemed to expect it to be presented as the plain
"J" it is.  (Also consider the draft explicitly accepts single-byte
emojis, even though this is at odds with Emoji-Seq.) 

I do not want some clients presenting the "J" as a "J" and some as a
smiley (think Wingdings) and some as a Unicode replacement character.

The easy way out is to not restrict the allowable set of codepoints,
which means allowing the shrug sequence and the table rage sequence
above.  I will note that the draft's grammar allows whitespace between
each "emoji", or let's call them individual emotions, which means
"Great Job" could parse as two emotions.  Or not.  My preferred
solution is still that all non-emoji (according to TR51) should be
presented as if they were unallocated code points.

Speaking of whitespace, the grammar uses LWSP = *(WSP / CRLF WSP)
This is IMHO at odds with "The content of this part is restricted to
single line of emoji."  Why allow CRLF if only a single line is
allowed?  Why restrict to a single line?

Another thing which could be spelled out in the draft is whether
multiple parts with "Content-disposition: reaction" (at the same level
in the mime structure) is allowed.  Per normal MIME behaviour it should
be.

Suggestion:

-A message sent as a reply MAY include a part containing:
+A message sent as a reply MAY include one or more parts containing:

Using multiple reaction parts can be used to circumvent the restriction
of a single line of emojis.

-- 
venleg helsing,
Kjetil T.

---

From: John C Klensin <[email-masked]>
Date: Thu, 25 Feb 2021 19:03:27 -0500

Hi. Barry asked that I, as well as Patrik, expand a bit on what
I said, off-list, to the IESG. First, let me stress that I
think this is a worthwhile experiment and should go forward. I
also believe it is entirely appropriate to include things about
which we are uncertain in an Experimental spec. However, if we
do, I think there is at least an ethical obligation for us to
be clear about known areas of uncertainty and to consider them
part of the experiment. The issues I see are with matters of
definition and what the experiment is about. I see the document
as fine on the definition of the MIME extension (plus or minus
questions Ban Kaduk has raised in his IESG comments). Where I
think there is a problem is that the message body content
(<part-content> in the text) is a small subset of what we
usually think of as plain text in UTF-8 but is not adequately
defined (or the challenges with doing so identified).

Patrik and I are focused on slightly different issues as most
important but I should stress that we are, at least AFAICT, in
complete agreement on what the issues are. I'm going to try to
summarize the key ones for those who dislike long messages; I
will follow up if needed with an inevitably-longer one with
more details.

The document's definition of the allowed characters
incorporates Unicode UTS#51 (which the I-D references through
[Emoji-Seq]) by reference. That is problematic in multiple ways
that go beyond our criteria for normative references to
documents produced by other bodies and how relaxed we should be
about those criteria for Experimental specification. The I-D
even indicates that the syntax for its key content syntax rule
is inherited from UTS#51, but UTS#51 does not contain (or use)
ABNF. The syntax definition language it uses is defined
elsewhere in the family of Unicode specifications; someone
trying to understand the syntax that the I-D supposed imports
will need to understand much of Unicode. I think that is an
unreasonable requirement but, if others disagree, the I-D
should at least warn people about what they are getting into
rather than saying "The ABNF rule Emoji-Seq is inherited from
[Emoji-Seq]" and stopping.

Another issue with UTS#51 is that the document, and the other
documents and tables on which it depends, change with each
version of Unicode to the extent that it is possible for a
sequence to be invalid in one (annual or more frequent) version
of Unicode and valid in another. That further complicates our
long-standing problem (discussed in IDNA2008 and PRECIS) with
how to handle unassigned code points. More on that, and
especially about its rendering implications, if a longer note
is needed.

In general, while I believe it is reasonable to be more relaxed
about definitions and other things because this is an
Experimental spec [1], it seems to me that, if we do so and
know those are problem areas, we are under some obligation to
identify those areas and ask for feedback on whether they were
problems in practice as part of the experiment.

I see several possible ways forward but the most plausible one
-- if we are to avoid going off in directions we have not
explored before (and likely introducing significant delay) --
is, approximately:

Leave the spec more or less as is but identify the issues with
dependency on UTS#51 as an evolving document that is long
(especially when the data files on which most of its
definitions rest are included) and complex. Also identify the
issues with alternate renderings and cultural interpretations
that the new text in draft-crocker-inreply-react-09 makes a
start on (but which I believe is too abstract to be useful
without examples or more description). In addition, make it
clear that part of the experiment is to gain some understanding
of whether the use of emoji (or other types of reactions) cause
cross-cultural problems (especially with multi-recipient
messages with recipients in widely different locales), more
complex features are used in practice, whether MUAs run into
rendering problems, and whether the UTS#51 dependencies are
problematic in practice.

An alternative, the one I think Patrik is now suggesting after
today's discussion, would be to limit the body content to a
line consisting of one or more emoji drawn from an enumerated
list of single-code-point emoji. That would considerably
simplify the spec but might reduce the feedback we would get
about interest and usability.  Or, as I think Adam is
suggesting, we could also allow symbol sequences, possibly
drawn only from the ASCII symbol repertoire. If we were to do
that, the question of whether emoji sequences and combinations
were needed (i.e., whether there was significant user demand
for them) should be explicitly part of the experiment. That
approach 
would have the advantages of both eliminating the need for a
reference to UTS#51 except for emoji definitions more specific
than those in the I-D text and of expanding the possible range
of "reactions". UTS#51 would become a relatively ordinary
normative reference, one consistent with what it actually
specifies. However, it would require us to agree on which code
points are allowed and which are not and would not eliminate
the need to discuss the cultural issues (particularly with
regard to the hand gestures that are now included in the
<base-emojis> set) and, if more than one code point is allowed
on the line, raises possible sequencing, delimiter, and Bidi
issues that we would need to at least consider and possibly
sort out. 

And, before someone says "send text", I'm willing to do so but
only when it is clear what the community consensus is terms of
strategy.

best,
   john

[1] Thanks to Barry for a comment in an offlist note that
helped me focus on this.

---

From: Adam Roach <[email-masked]>
Date: Thu, 25 Feb 2021 18:26:01 -0600

On 2/25/21 18:03, John C Klensin wrote:
> Or, as I think Adam is
> for them) should be explicitly part of the experiment.

My examples were decidedly not limited to those symbols that ASCII can 
represent. :)

I'll also concede that I was going fast and read:

part-content =  emoji *(lwsp emoji) CRLF

as:

part-content =  emoji *(*lwsp emoji) CRLF

...so I was under the impression that the sequences you describe were 
already part of the proposal. :)

/a

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 17:48:38 -0800

> allowed?  Why restrict to a single line?

Good catch.  It should be WSP and not LWSP.

Thanks!

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 17:55:43 -0800

On 2/25/2021 4:03 PM, John C Klensin wrote:
>      the I-D

At Barry's suggestion earlier today the next version of the draft 
already contains:

             <t>The ABNF rule emoji_sequence is inherited from <xref 
target="Emoji-Seq"/>. It defines a set of octet
                 sequences, each of which forms a single pictograph. The 
BNF syntax used in [Emoji-Seq] differs
                 from <xref target="ABNF"/>, and MUST be interpreted as 
used in Unicode documentation.</t>

>      it seems to me that, if we do so and

Send text.

d/

-- 

Dave Crocker
[email-masked]
408.329.0791

---

From: John C Klensin <[email-masked]>
Date: Thu, 25 Feb 2021 22:09:25 -0500

Kjetil,

Disclaimer: I read your note and started this response just
after sending my long one but was interrupted by a meeting/ call
I had to take.  So, if there are messages that have arrived
since mine was sent, I have not read them yet.  If they bear on
this part of the conversation, my apologies.

The much longer note I just posted covers some of this and I
will try to not repeat myself, but a few comments: 

First and most important, thanks for the reminder that issues of
this general family were raised much earlier and not really
addressed and this is not just Patrik and myself having late
attacks of Unicode- or 118n-anxiety.

More specific comments in line below (with some trimming)...

--On Thursday, February 25, 2021 22:05 +0100 Kjetil Torgrim
Homme <[email-masked]> wrote:

>...
> This was partially one of the points I made earlier.  The
> draft is eerily silent on what to do with a reaction like "J".
> It is not an unallocated code point, but it is not a valid
> emoji either.  When I brought it up, Dave seemed to expect it
> to be presented as the plain "J" it is.  (Also consider the
> draft explicitly accepts single-byte emojis, even though this
> is at odds with Emoji-Seq.) 

Actually, that is an odd near-ambiguity in the document that I
think should be corrected as other things are sorted out (and,
if needed, flagged to the RFC Editor before publication).  If
one puts the URL aside as a convenience, the reference for
[Emoji-Seq] points to UTS#51, which clearly allows single-code
point [1] emoji and even some traditional symbols.  On the other
hand, the link is to
http://www.unicode.org/reports/tr51/#def_emoji_sequence, which,
at least today, is rule ED-17 of the Version 13.1, 2020-09-18
version of UTS#51.  It points back to ED-15 for
<emoji_core_sequence> which rather clearly (at least IMO) allows
a single emoji character, which leads to ED-3 and single code
point emoji.    So, probably that is consistent in the document.
Whether it is reasonable to expect anyone implementing the I-D
to search through that is another question but is part of my
concern about incorporating UTS#51 by reference and moving on.

On the other hand, because "J" (I'm getting U+0022 from your
message -- see below-- and not a special symbol) does not appear
to have the emoji property, I believe the I-D forbids it
entirely and having it appear as "J" would violate the spec.  If
the intention is that non-emoji appear as themselves, then
either the <part-content> in the spec is wrong or, IMO, the spec
needs some words about how receivers are expected to handle
content that lies outside the specification.

> I do not want some clients presenting the "J" as a "J" and
> some as a smiley (think Wingdings) and some as a Unicode
> replacement character.

It the "J" you are using is actually a character outside the
ASCII repertoire, then either your mail system, the IETF's, or
mine did use a replacement character and that should be a
warning to all of us (and not only in this particular context).  

However, even for "normal" emoji, there are risks if you, as the
sender, are expecting a particular grapheme to be delivered.
For example, if one has "thumbs-up" (U+1F44D) (cited several
times in discussion in the I-D and on the <base-emojis> list),
there is no way to guarantee whether you will get something that
looks like the left hand with the thumb up, the right hand with
the thumb up, or neither.  And, further to Patrik's point, at
least one of those is an obscenity in some cultures.

> The easy way out is to not restrict the allowable set of
> codepoints, which means allowing the shrug sequence and the
> table rage sequence above.  I will note that the draft's
> grammar allows whitespace between each "emoji", or let's call
> them individual emotions, which means "Great Job" could parse
> as two emotions.  Or not.  My preferred solution is still that
> all non-emoji (according to TR51) should be presented as if
> they were unallocated code points.

Well, that takes out Adam's example and your "J".  It (and the
current spec) also allow some interesting sequences, such as the
"police shoot crocodile" (or vice versa) example in UTS#51.
But I have a different concern that I have not raised
(recently):  my expectation is that, at least unless the spec
warns them to do something differently (and I don't think it
should or that they would if we told them to), the typical MUA
is going to take whatever appears in <part-content> and pass it
off as a string or unexamined Unicode code points to whatever it
uses to render strings and put them on the screen (or whatever).
Now suppose that the rendering routine receives an emoji
sequence or emoji-containing sequence that is not allowed by
UTR#51.  Does it not even bother with UTS#51's rule and try
rendering whatever it gets anyway (as it might if it didn't know
that emoji rendering was in any way special)?  Does it guess at
what was intended, perhaps more or less the way font
substitution works?  Does it get confused and display
semi-random garbage?  Or does it give the user, or arrange for
the user to be given, a clear error message about being passed
an invalid or nonsense string?

And, in terms of the I-D --whether your proposal is adopted or
we stay with UTS#51's definition of emoji sequence-- do we give
any advice to receiving systems about those cases, strengthen
the wording about "operational problems" in Section 7, or just
assume that it, like other bad stuff, will be reported.
Personally, I'd prefer to see some advice or warning because, if
the complex or silly cases blow up in the faces of users, it
could get this whole idea and undeserved bad reputation.

> Speaking of whitespace, the grammar uses LWSP = *(WSP / CRLF
> WSP) This is IMHO at odds with "The content of this part is
> restricted to single line of emoji."  Why allow CRLF if only a
> single line is allowed?  Why restrict to a single line?
>...

I'm going to confine myself to Unicode and i18n issues (and
disclaim ABNF expertise as a matter of habit) and let the
authors respond to that, and your remaining, comments -- I hope
not just by telling you its very late to be raising those issues.

best,
   john

[1] I'm avoiding "byte" because the vast number of code points
people think of as emoji, even the assorted faces in other
symbols in the "Miscellenous Symbols" block at U+2600-U+26FFm
cannot be represented in a single octet in any Unicode encoding
scheme.  The exceptions (#, *, 0-9, ©, and ® ) are, I believe,
not what most people think of when they think of emoji.  See
https://www.unicode.org/Public/UCD/latest/ucd/emoji/emoji-data.txt.
I assume you know that already, but some people reading this may
not and it emphasizes the importance of being very precise when
talking about these things.

---

From: John C Klensin <[email-masked]>
Date: Thu, 25 Feb 2021 23:07:11 -0500

--On Thursday, February 25, 2021 17:55 -0800 Dave Crocker
<[email-masked]> wrote:

> At Barry's suggestion earlier today the next version of the
> draft already contains:
> 
>              <t>The ABNF rule emoji_sequence is
> inherited from <xref target="Emoji-Seq"/>. It defines a set of
> octet
>                  sequences, each of which forms
> a single pictograph. The BNF syntax used in [Emoji-Seq] differs
>                  from <xref target="ABNF"/>,
> and MUST be interpreted as used in Unicode documentation.</t>

An improvement, but does not quite work.  First of all, IIR, the
only Unicode materials that talk about octet sequences are those
that deal specifically with Encoding Forms.  UTS#51 is not one
of them.  It mentions only "code points" and "characters" in the
context of what you/Barry are getting out.  The I-D can talk
about octets if it very careful to confine itself to characters
on the wire, where a Content-type of text/plain; charset=utf8 is
specified, but the string might not be in UTF-8 by the time the
MUA gets it.   Even then, it seems to me the above is really
talking about code points.  Second, my biggest concern about the
original language about inheriting ABNF isnt the difference
between the two syntax notations themselves.  It is that the
notation used in UTR#51 involves operators that are vital to
understanding what is going on but than are not defined (or the
definition referenced) from within it.  "interpreted as used in
Unicode documentation" is true, but is handwaving that will be
of no help to someone trying to use UTR#51 to figure out what is
being specified.

That gets back to emoji_sequence and the incorporation of
[Emoji-Seq]/UTS#51 by reference.   If our intent is to say
"you'd better understand Unicode and its documentation
conventions and then study UTS#51 in particular", I'm probably
ok with that but my experience suggests that very few
implementers or designers --at least those similar to the subset
who participate in IETF -- will bother they will just guess what
UTS#51 say and move on (some may look quickly at UTS#51 first;
others won't bother even trying.  Otherwise, we should be, IMO,
providing pointers or hints.  I won't feel that way if it were
straightforward, but UTS#51 uses \p() and \P() notation.  Do you
know what they mean? Are they case-sensitive (i.e., the two are
different) or not?  And do you know where to find the
definitions (hint: the index to TUS 13.0 does not contain any of
the terms "syntax", "metalanguage", "operator", or "expression")?

>> problems in practice as part of the experiment.

As I indicated in my note, I believe this is a case in which we
need to reach some sort of consensus about what we want to do
first.  Text that is linked to specific suggestions or issues is
likely to be an invitation for tours of assorted bikesheds (and
discussions of the kind of details of terminology that appears
above) if it is supplied before there is agreement about what
problems are worth solving.

    john

---

From: Dave Crocker <[email-masked]>
Date: Thu, 25 Feb 2021 20:28:51 -0800

> problems are worth solving.

John, since you are pressing to up-level the entire topic, I'd be 
interested in your explaining a bit about the way you are pursuing your 
concern.

The draft was discussed in September and October on the 1etf-822 mailing 
list, with significant activity and revision.  You posted one note, 
offering multiple points, but not what you are raising now.

IETF Last Call went for a month, but again, nothing on the last-call 
mailing list until now, after the IESG has reviewed the document.

And for all that, you are pressing to re-open first-principles on what 
otherwise seems to be and is intended to be quite a simple 
specification.  And equally pressing for refraining from offering 
concrete solutions to resolve your concerns.

I'll also offer that besides some focused issue with the choice of 
details for referencing Unicode, I really can't claim to have a concrete 
-- or for that matter, even vague -- understanding of what your larger 
concerns are, with respect to this specification and with its use in 
email, in spite of your posting multiple notes today.

I'm curious how this can be viewed as productive.

d/

ps. I do understand that you raised the concern privately with the 
IESG.  That doesn't seem all that relevant, given the above.

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 26 Feb 2021 13:53:57 +0100

> concern about incorporating UTS#51 by reference and moving on.

I think that is entirely reasonable.  The rule references are collected
in one place (except the \P{} mechanism) and the syntax used is close
enough to ABNF to be easy to follow for anyone working with IETF
standards before.

> entirely and having it appear as "J" would violate the spec.

Yes, it is indeed a plain "LATIN CAPITAL LETTER J", and yes it violates
spec, but so does any of the unallocated code points.  Why spell out
how to handle one illegal character sequence and not the other?

> If

Exactly.  The Postel Principle is both a blessing and a curse, but more
often than not the latter, IMHO.  I want more strictness in general. 

Existing text + suggested clarification:

   Reference to unallocated code points SHOULD NOT be treated as an
   error; associated bytes SHOULD be processed using the system default
   method for denoting an unallocated or undisplayable code point.
+  Code points from the private use area MUST NOT be used.
+  Other violations of the grammar SHOULD cause the part to be
+  discarded.

and in section 3, inject a new step 4 like:

+ 4. Reaction parts which are not associated with a valid In-Reply-To
+
   SHOULD be discarded.

> warning to all of us (and not only in this particular context).

It used to be very common to get mail from Microsoft users with "J"
strewn across the message - in old Wingdings, the code point "J" was
replaced by a smiley face.  For users without the Wingdings font, a
normal font was substituted, revealing its true code point.  See? J

> least one of those is an obscenity in some cultures.

Sure, but this is impossible to safeguard against.  E.g., I myself like
to use

  U+1F64F   🙏   PERSON WITH FOLDED HANDS

as a symbol for thanks, since it looks like the gesture of namaste.  I
was mulling over using textual labels to reduce the chance of this
happening, but in reality there *is* a textual label already, and a
":pray:" textual label (like in Slack or Mattermost) would not be
visible to end users any more than the Unicode label is.

> Well, that takes out Adam's example and your "J".

I don't actually have a strong preference for disallowing them or
presenting them as undefined.  But yes.  If we are going to be lenient,
let's be explicitly lenient and allow any single line response, e.g.,
"Bravo!" as well as "<U+1F44D>"

> not just by telling you its very late to be raising those issues.

I understand where he's coming from.  I am following this from "last-
call" only, for good or for bad.  Perhaps there should be pointers to
where to go for archived discussion when posting to last-call.  I don't
actually know what mailing list this draft was discussed on.

> talking about these things.

Actually I did not know or glossed over it, since I consulted "full-
emoji-list" instead.  So yes, #*0123456789 are all plain ASCII
characters that have the Emoji property set.  In other words, I was
wrong when I said a single byte emoji was impossible.  I do however
still think it is unfortunate terminology.  Better to use "character"
or "code point" as you too indicated elsewhere.

-- 
venleg helsing,
Kjetil T.

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Feb 2021 20:05:14 -0500

--On Thursday, February 25, 2021 18:26 -0600 Adam Roach
<[email-masked]> wrote:

> My examples were decidedly not limited to those symbols that
> ASCII can represent. :)
> 
> ...so I was under the impression that the sequences you
> describe were already part of the proposal. :)

Adam,

As Dave has sort of pointed out, turning the <part-content> of
the response into "anything that can be expressed in UTF-8" or
"anything that can be expressed in UTF-8 with emoji and emoji
sequences constrained by UTS#51" or something similar, would be
a rather different proposal.  My recollection is that something
closer to that was discussed during the (Northern Hemisphere)
fall and that it obviously did not get enough traction to make
it into the document.  I find it easy to think of cases where
other strings, or textual descriptions of "reactions" would be
useful.  For example, of course without using the specific body
part structure proposed by the I-D, I've responded to a couple
of messages within the last 48 hours by top-posting "Brilliant!"
and "Great work" and to some other messages with brief
disparaging "reactions".  None of those comments has obvious
corresponding emoji, which may be why they are seen frequently
on social media and instant messages as well the the emoji
themselves.

In that context, if a non-trivial number of people believes that
other symbols, free text, or even fewer restrictions would make
this extension significantly more useful (or useful at all), I
hope the authors and the IESG would be receptive to adding a
bullet to Section 7 of the I-D asking if the constraint to emoji
made the extension significantly less useful and/or less used,
in practice.  In particular, if an MUA provides special
arrangements for a "reaction" as the I-D seems to suggest and
one can send <frowning-face> using the facility but not "Snark!"
or "Bleech" I think experience with user interfaces predicts
that users will become frustrated with the distinction and
either not use the reaction-specific mechanism or whine a lot.
However, that makes the question a legitimate part of the
experiment; the only important question about it, IMO, is
whether it should be called out explicitly in Section 7.

While a change to expand the repertoire of what can occur in
<part-content> might be desirable, it feels to me late enough in
the process that those who want to open that issue should either
generate an I-D to modify/update this I-D to change the
<part-content> definition to something significantly different
and seek sponsorship for it or should ask the IESG to pull
draft-crocker-inreply-react until the body part syntax can be
sorted out and expanded.   I have no idea how the IESG would
react to such a request.  If they asked me for advice (which I
don't imagine they would do), I have no idea how I'd respond.

While I have to admit that I find these more substantive
discussions about allowed characters and interpretation more
interesting, I want to try to concentrate right now (and urge
others to do so) on a fairly narrow issue: getting the
definition of <emoji> in the current I-D close enough to right
that there is a reasonable expectation of implementers, senders,
and receivers reaching the same conclusions about what they
should do and not do (including about how responses will be
interpreted) given the <part-content> production you quote
above.  And, if there are places where we don't know how to do
that or cannot agree, let's explicitly identify those places in
the document where problematic definitions can be found and then
identify figuring out whether there are problems in practice
(and, if so, what they are), as part of the experiment.

best,
   john

---

From: Dave Crocker <[email-masked]>
Date: Fri, 26 Feb 2021 17:16:00 -0800

On 2/26/2021 5:05 PM, John C Klensin wrote:
> In particular, if an MUA provides special

On the average, specifications that start simpler, allowing user 
experience pressure to prompt adding more features, fare far better than 
specifications that try to be all-inclusive at the start, with every 
feature anyone can think of. (cf, Internet vs OSI, at every level.)

Simpler to understand.  Simpler to implement.  Simpler to use.

In this case, there also is established experience for more constrained 
reaction mechanisms.  All of the more flexible cases that have been 
cited here occur within unbounded text, not within a reaction mechanism.

d/

---

From: "Ricardo Signes" <[email-masked]>
Date: Fri, 26 Feb 2021 21:29:48 -0500

On Fri, Feb 26, 2021, at 7:53 AM, Kjetil Torgrim Homme wrote:
> > [...] If

For what it's worth, I'm the one who provided the reference to UTS51, and did so only after carefully making sure it allowed a single-code-point emoji, after a citation in a previous version (possibly only mentioned on list) pointed to something that only allowed multi-code-point sequences (to my bafflement).

Then there is the discussion of "what if somebody sticks `J` into the reaction part".

> +  Other violations of the grammar SHOULD cause the part to be

I agree with the spirit of this change:  supply a simple "what if this part doesn't fits its constraints" case.  I think the language you provide is adequate.  The other behavior that comes to mind is to only discard part-contents in places where "emoji" is expected, but that does not meet the criterion of matching the emoji token.  That is, given the part content:

  \N{FOX FACE}   J   \N{DISGUISED FACE}

Discard the J, and not the rest.  (Actually, "discard" here is probably better as "treat as an invalid codepoint".)

-- 
rjbs

---

From: John C Klensin <[email-masked]>
Date: Sat, 27 Feb 2021 16:54:06 -0500

Dave,

I think there is some misunderstanding about my concerns, so I'd
like to back up a bit and review...

To be clear upfront where I am going with this, I believe it
would be acceptable (especially given that the issues did not
arise during normal Last Call or earlier) for the document to
be modified to indicate that concerns have been raised that the
definition of emoji_sequence is problematic, with an example or
two about why, and then to extend Section 7 to make clear that
we are looking for input on the definition: whether it is
comprehensible and allows too much or too little, and about any
interoperability or inter-cultural problems it causes. I'd
prefer to go a bit further (but only a bit); the background and
reasons are discussed below.

I believe that it has been generally understood since September
or October that this draft is about expressing "reactions"
(whatever that means) in emoji. I would have a good deal of
sympathy if you took the position that the several suggestions
to allow code points other than emoji and/or to allow verbal
descriptions or keywords that could be turned into graphics by
the receiving MUA are too late at this stage, but that is a
topic that you might take up with the IESG either as a general
issue or if a DISCUSS emerges asking that those proposals be
considered. I think those late comments may be a sign that our
Last Call process is failing for AD-sponsored documents and
will probably take that up with the IESG or on the IETF List as
soon as I figure out how to do better (and you or others should
do so if you have good ideas first). But those issues are very
different from my concern and Patrik's main ones: that
<part-content> and <emoji_sequence> are not defined in a way
that is likely to be helpful to the reader or implementer of
the spec, that they will be interpreted differently (in part
because UTS#51 is hard to understand and many implementers will
guess rather than bothering), and that the current definition
without some modification or qualification will weaken the
experiment the I-D proposes. It is not really a separate matter
because emoji and the definition leave everything entangled,
but the possibility that the receiver may interpret emoji or
strings of them very differently than the sender might intend,
is at least a different part of the problem.

I have tried to make clear in both my correspondence with the
IESG and in my first note on this list in this thread that,
while I would far prefer to get the definition nailed down to
the extent that people know what to implement (including what
should be sent and how strings that careful readings of UTS#51
would make clear are invalid are to be handled). A reasonable
person could argue that even an IETF Experimental specification
should not be adopted unless it is as clear as we know how to
make it about what should be implemented and how odd cases
should be handled but I have carefully avoided taking that sort
of showstopper position.

The difference between replacing the definition with a better
one, one that would presumably depend on UTS#51 only for
additional information, and handwaving our way around those
issues by making them part of the experiment are great enough
to motivate my resistance to writing text ... text that would
almost certainly turn the discussion from that choice (even if
non-emoji changes were excluded from the former option) into
discussion of the text itself.

Patrik and I have written a great deal more text in these
messages but, at least for me, the motivation was to explain
and document that the issues associated with the current
definition are significant enough that it would be
inappropriate to dismiss them as just not worth bothering with.
Should the community decide that actually fixing the
definition, or even supplying examples to explain open issues,
is needed, much of the text can probably be taken from those
notes, but, again, the motivation for them so far has been to
demonstrate that the topic really is important.

best,
   john

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Sat, 27 Feb 2021 23:48:29 +0100

> Good catch.  It should be WSP and not LWSP.

Thanks!  This has the added benefit that emojis are required to be
whitespace separated (whereas LWSP has an empty expansion).  Well, I
call it a benefit because I think it gives a better hint about each
emoji being a separate reaction, for MUAs which want to do tallies and
summaries.

-- 
venleg helsing,
Kjetil T.

---

From: Dave Crocker <[email-masked]>
Date: Sat, 27 Feb 2021 15:04:04 -0800

> emoji being a separate reaction, for MUAs which want to do tallies and

Exactly.  It's allows a very simple parser to distinguish emoji octets 
and evaluate them separately.

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

---

From: John C Klensin <[email-masked]>
Date: Sat, 27 Feb 2021 19:33:11 -0500

--On Saturday, 27 February, 2021 23:48 +0100 Kjetil Torgrim
Homme <[email-masked]> wrote:

>> > Speaking of whitespace, the grammar uses LWSP = *(WSP /
>> > CRLF WSP) This is IMHO at odds with "The content of this
>> > part is restricted to single line of emoji."  Why allow
>> > CRLF if only a single line is allowed?  Why restrict to a
>> > single line?
>> 
> Thanks!  This has the added benefit that emojis are required
> to be whitespace separated (whereas LWSP has an empty
> expansion).  Well, I call it a benefit because I think it
> gives a better hint about each emoji being a separate
> reaction, for MUAs which want to do tallies and summaries.

I don't want to spend a lot of time on this unless we are at the
point that people have concluded it is time to call off the Last
Call and start reviewing the document (something I, for one,
have absolutely not concluded), but...

(1) If support for tallies and summaries, presumably including a
message being sent out in bulk and having recipients respond to
it in a way that is equivalent to voting, is part of the
intended use of this feature, why is that not called out in the
document?  Unless I have missed something even after reading it
several times in the last few days, he document now seems to be
focused on interpersonal communications and not such tasks,
tasks that might call for a review of security considerations
and other text.

In addition and at least equally important, unless either the
sending and receiving systems are dedicated application rather
than conventional MUAs, or the vocabularies are extremely
limited, any notion of tabulating responses requires begin able
to determine whether two emoji or emoji sequences are "the
same".  That need to do comparison, especially in combination
with there now being unitary (since code point) emoji whose most
likely symbolic representations can also be represented by
non-trivial sequences  involves hard problems, with the absence
of the normalization techniques that help with text making them
harder.

For one of the most obvious examples (although probably not the
most difficult one) consider whether (all of the words below
representing emoji and skipping over combining and qualifier
issues) are the following three and other permutations
equivalent?
    man-woman-child
    man-child-woman
    woman-child-man  
are any or all of them equivalent to the "family" emoji
character?

So, if the authors (or, now, anyone else) contemplate using
reactions in a way that would lend itself to tabulation or other
operations involving comparisons, text and a health warning are
needed for that too.

(2) When I see multi-grapheme (as distinct from
multi-code-point) emoji in contexts that might be described as
responses, among the most common ones I see are sequences
intended to convey
   hug-hug-hug 
using the :TEDDY-BEAR: emoji.  But
   hug hug hug 
which the above change appears to force, could easily have
rather different semantics especially if used in conjunction
with other emoji characters, e.g., of a teddy bear picnic.  And,
coming back the first point, if the spaces are allowed and
comparison testing is a requirement, are the spaces significant?

Both of the above lead us away from the simplicity and principle
of starting out as simple as possible and extending only when
the need is clear that Dave eloquently argued for and which I
strongly support.  Much as I hate the idea, they also seem to me
to be taking us close to the line beyond which a new version of
the spec would represent a great enough change to require an
additional Last Call.

best,
    john

---

From: "Ricardo Signes" <[email-masked]>
Date: Sat, 27 Feb 2021 21:57:43 -0500

> tasks that might call for a review of security considerations

I believe the focus is interpersonal communication.  In the long run, I would not be surprised were someone to want to summarize the count of reactions of each type to a message, but it is not the given intent or focus of this proposal.  Any means of aggregation other than canonical equivalence seems like it would be a question of context and semantics, unable to be generically specified… but we don't even suggest aggregation over compatible emoji, here.

> intended to convey…

Thanks, this is a great point.  I believe we must specify that the emoji provided in the reaction are not a string, but a set (or multiset, I suppose).  Without this, the advice to drop individual illegal emoji candidates is dangerous!

Imagine that Emoji 20 adds a "NOT" emoji (ignoring that "NO ENTRY SIGN" would work in this story).  The reply "NOT {BEATING HEART}", received by a system with only an Emoji 19 repertoire would drop the NOT.  I'm not sure what this means, other than the opposite of what was intended.  No matter the presentation layer, if we allow some components to be dropped while others are displayed, we must be clear that the line is not a string.

Alternatively, if the line must be a string, then we must not keep any portion while discarding another.  But I favor adding text indicating that it's a multiset.

-- 
rjbs

---

From: Dave Crocker <[email-masked]>
Date: Sat, 27 Feb 2021 19:02:42 -0800

On 2/27/2021 6:57 PM, Ricardo Signes wrote:
> Thanks, this is a great point.  I believe we must specify that the 
> emoji provided in the reaction are not a string, but a set (or 
> multiset, I suppose).  Without this, the advice to drop individual 
> illegal emoji candidates is dangerous!

And, indeed, the even-more-recent draft that will be submitted how has 
added to Section 2:

> Note:
>     The part-content can first be parsed into candidate reactions,
>     separated by wsp. Each candidate reaction that does not constitute
>     a single emoji-sequence (as per [Emoji-Seq]) is invalid. Invalid
>     candidates can be treated individually, rather than affecting the
>     remainder of the part-content's processing . The remaining
>     candidates form the set of reactions to be processed. This
>     approach assumes use of a mechanism for emoji sequence validation
>     that is not specified here.
>
d/