Subject: [media-types] Early review request: draft-ietf-appsawg-text-markdown

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 14 Apr 2015 12:25:41 -0700

Hi there,

Could we get an early review of this document?  It contains a media type
registration.  The document is approaching Working Group Last Call in
APPSAWG and I wanted to see if there's anything he should fix before the
formal request comes through.

https://datatracker.ietf.org/doc/draft-ietf-appsawg-text-markdown/

It's a good opportunity for me to train on filtering new requests too.

Thanks,
-MSK

---

From: Chris Lilley <[email-masked]>
Date: Tue, 14 Apr 2015 22:14:11 +0200

Hello Murray,

> Could we get an early review of this document?  It contains a media
> type registration.  The document is approaching Working Group Last
> Call in APPSAWG and I wanted to see if there's anything he should
> fix before the formal request comes through.

> https://datatracker.ietf.org/doc/draft-ietf-appsawg-text-markdown/

Putting Markdown in the text tree seems reasonable; it is probably
more easily readable in source form than text/html.

"Many Markdown processors will get along just fine by operating on
character codes that lie in printable US-ASCII, blissfully oblivious
to coded values outside of that range."

That seems to imply that encodings where US-ASCII does not encode as
itself (such as UTF-16) should be avoided.

"Encoding considerations: Text."

That doesn't really say anything.

" Fragment identifier considerations:

     See Section 4."

Fragment Identifiers is actually section 3.

-- 
Best regards,
 Chris  Lilley
 Technical Director, W3C Interaction Domain

---

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Apr 2015 12:04:44 -0700

Thanks for the feedback! My responses are below.

> itself (such as UTF-16) should be avoided.

Not exactly.

Many programming ecosystems have a text-oriented stream or processing 
pipeline that abstracts the (usually octet based input) into a sequence 
of non-negative integers. I am referring in vague terms to C/C++ 
stdlibs, PHP, Perl, POSIX, Python, etc.

Someone can write a Markdown processor using these programming 
libraries, in such a way that the code operates on the abstract 
non-negative integers, and sequences out another set of abstract 
non-negative integers, which get converted into octets in accordance 
with the codepage or other settings in the environment or programming 
library. (This is in fact how several Markdown processors are written, 
such as Markdown.pl.)

Therefore if the programming library recognizes UTF-16 input (with or 
without BOM, and with or without external metadata) and turns them into 
abstract non-negative integers for the program to consume, there is no 
reason to avoid (or promote) UTF-16 encoding over other forms. The 
program itself just looks for things like "<" which maps to the 
non-negative integer 0x3C (decimal 60). It is of course possible that 
EBCDIC is used, but I am not aware of popular Markdown processors that 
work specifically on EBCDIC values at the application level (obviously, 
such an application can consume and produce EBCDIC content if the 
programming library does the mapping from and to underneath), hence why 
the text says "will get along just fine by operating on character codes 
that lie in printable US-ASCII".

> That doesn't really say anything.
Well it is supposed to imply all of the stuff above.

Nevertheless I did a search and lo-and-behold, I haven't found other 
registrations that say "Text" for this section.

So, how about:

*****
Encoding considerations:
Markdown content is plain text content; any octet sequence is valid as 
long as it conforms to the character codes of the charset parameter. See 
[RFC2046]. Markup characters in Original Markdown [MDSYNTAX] are limited 
to printable US-ASCII; however other variants can define markup 
characters outside this range (including control characters such as NUL 
and characters encoded in multiple bytes).
*****

> Fragment Identifiers is actually section 3.

Thanks. I will change that reference to Section 3 in the next draft.

Looking at the text, it appears that the Original variant is not 
actually registered, even though it is used in the example. I will draft 
a registration in the text-markdown-07 document.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Apr 2015 22:39:15 -0700

On 4/15/2015 12:04 PM, Sean Leonard wrote:
> [THINGS]

draft-07 has been posted. 

http://tools.ietf.org/html/draft-ietf-appsawg-text-markdown-07

The registration template follows below.

**************

   This section provides the media type registration application for the
   text/markdown media type (see [RFC6838], Section 5.6).

    Type name: text

    Subtype name: markdown

    Required parameters:

     charset: Per Section 4.2.1 of [RFC6838], charset is REQUIRED. There
       is no default value. [MDSYNTAX] clearly describes Markdown as a
       writing format; its syntax rules operate on characters
       (specifically, on punctuation) rather than code points. Neither
       [MDSYNTAX] nor many popular implementations at the time of this
       registration actually require or assume any particular character
       set. Many Markdown processors will get along just fine by
       operating on character codes that lie in printable US-ASCII,
       blissfully oblivious to coded values outside of that range.

    Optional parameters:

     variant: An optional identifier that serves as a "hint" to the
       recipient of the specific Markdown variant that the author
       intended. When omitted, there is no hint; the interpretation is
       entirely up to the receiver and context. This identifier is plain
       US-ASCII and case-insensitive. To promote interoperability,
       identifiers MAY be registered in the registry defined in Section
       6. If a receiver does not recognize the variant identifier, the
       receiver MAY present the identifier to a user to inform him or
       her of it.
 
     Other parameters MAY be included with the media type. The variant
     SHOULD define the semantics of such parameters. Additionally, the
     variant MAY be registered under another media type; this
     text/markdown registration does not preclude other registrations.

    Encoding considerations:

     Markdown content is plain text content; any octet sequence is valid
     as long as it conforms to the character codes of the charset
     parameter. See [RFC2046]. Markup characters in [MDSYNTAX] are
     limited to printable US-ASCII; however, other variants can define
     markup characters outside this range (including control characters
     such as NUL and characters encoded in multiple bytes).

    Security considerations:

     Markdown interpreted as plain text is relatively harmless. A text
     editor need only display the text. The editor SHOULD take care to
     handle control characters appropriately, and to limit the effect of
     the Markdown to the text editing area itself; malicious Unicode-
     based Markdown could, for example, surreptitiously change the
     directionality of the text. An editor for normal text would already
     take these control characters into consideration, however.

     Markdown interpreted as a precursor to other formats, such as HTML,
     carries all of the security considerations as the target formats.
     For example, HTML can contain instructions to execute scripts,
     redirect the user to other webpages, download remote content, and
     upload personally identifiable information. Markdown also can
     contain islands of formal markup, such as HTML. These islands of
     formal markup may be passed as-is, transformed, or ignored (perhaps
     because the islands are conditional or incompatible) when the
     Markdown is processed. Since Markdown may have different
     interpretations depending on the tool and the environment, a better
     approach is to analyze (and sanitize or block) the output markup,
     rather than attempting to analyze the Markdown.

   Interoperability considerations:

     Markdown variations (some might say "innovations") are designed to
     be broadly compatible with humans ("humane"), but not necessarily
     with each other. Therefore, syntax in one Markdown derivative may
     be ignored or treated differently in another derivative. The
     overall effect is a general degradation of the output, proportional
     to the quantity of variant-specific Markdown used in the text. When
     it is desirable to reflect the author's intent in the output, stick
     with the variant identified in the variant parameter.

   Published specification: This specification; [MDSYNTAX].

   Applications that use this media type:

     Markdown conversion tools, Markdown WYSIWYG editors, and plain text
     editors and viewers; markup processor targets indirectly use
     Markdown (e.g., web browsers for Markdown converted to HTML).

   Fragment identifier considerations:

     See Section 3.

   Additional information:

     Magic number(s): None
     File extension(s): .md, .markdown
     Macintosh file type code(s):
       TEXT. A uniform type identifier (UTI) of
       "net.daringfireball.markdown", which conforms to "public.plain-
       text", is RECOMMENDED [MDUTI]. See [MDMTUSES] for other
       considerations.

   Person & email address to contact for further information:

     Sean Leonard <[email-masked]>

   Restrictions on usage: None.

   Author/Change controller: Sean Leonard <[email-masked]>

   Intended usage: COMMON

   Provisional registration? No

   Implementations SHOULD record the value of the variant parameter (and
   other parameters if defined by the variant) along with the Markdown
   content when the content leaves the domain of Internet media type-
   capable formats. Strategies for doing so are discussed in [MDMTUSES].

   The Content-Disposition header (particulalry the preview-type
   parameter) can be used with Markdown content. See Section 4.

*END*

---

From: Graham Klyne <[email-masked]>
Date: Fri, 17 Apr 2015 17:59:25 +0100

I think the template is generally OK, but I spot a couple of nits:

[[

]]

The last sentence here caused me some confusion.  I think you're trying to say 
something like:  "When it is important to reflect the author's intent in the 
output, processors should only accept markdown variants that they explicitly 
know about, and Markdown generators should avoid use of variants that are now 
known to be understood by the intended recipient."

In the penultimate sentence, the description "proportional to ..." seems a 
spurious description.  Maybe "which increases with ..."?

...

[[
]]

Typo: "particulalry"

#g
--

On 16/04/2015 06:39, Sean Leonard wrote:
> _______________________________________________
> media-types mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/media-types
>

---

From: "Eric Prud'hommeaux" <[email-masked]>
Date: Fri, 17 Apr 2015 22:17:23 -0400

> Many programming ecosystems have a text-oriented stream or processing
pipeline that abstracts the (usually octet based input) into a sequence of
non-negative integers. I am referring in vague terms to C/C++ stdlibs, PHP,
Perl, POSIX, Python, etc.
>
> Someone can write a Markdown processor using these programming libraries,
in such a way that the code operates on the abstract non-negative integers,
and sequences out another set of abstract non-negative integers, which get
converted into octets in accordance with the codepage or other settings in
the environment or programming library. (This is in fact how several
Markdown processors are written, such as Markdown.pl.)
without BOM, and with or without external metadata) and turns them into
abstract non-negative integers for the program to consume, there is no
reason to avoid (or promote) UTF-16 encoding over other forms. The program
itself just looks for things like "<" which maps to the non-negative
integer 0x3C (decimal 60). It is of course possible that EBCDIC is used,
but I am not aware of popular Markdown processors that work specifically on
EBCDIC values at the application level (obviously, such an application can
consume and produce EBCDIC content if the programming library does the
mapping from and to underneath), hence why the text says "will get along
just fine by operating on character codes that lie in printable US-ASCII".

Is it possible for a generic markdown editor, be it a script-sensitive GUI
tool or pico/notepad, to record words like "漢字" and "кириллица" in a way
that all markdown systems should recognize and render correctly (e.g.
transcoding to match HTTP preferences)?

long as it conforms to the character codes of the charset parameter. See
[RFC2046]. Markup characters in Original Markdown [MDSYNTAX] are limited to
printable US-ASCII; however other variants can define markup characters
outside this range (including control characters such as NUL and characters
encoded in multiple bytes).
> Looking at the text, it appears that the Original variant is not actually
registered, even though it is used in the example. I will draft a
registration in the text-markdown-07 document.

---

From: Sean Leonard <[email-masked]>
Date: Sat, 18 Apr 2015 11:21:40 -0700

On 4/17/2015 7:17 PM, Eric Prud'hommeaux wrote:
>
>
> On Apr 15, 2015 9:21 PM, "Sean Leonard" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> > Many programming ecosystems have a text-oriented stream or 
> processing pipeline that abstracts the (usually octet based input) 
> into a sequence of non-negative integers. I am referring in vague 
> terms to C/C++ stdlibs, PHP, Perl, POSIX, Python, etc.
> >
> > Therefore if the programming library recognizes UTF-16 input (with 
> or without BOM, and with or without external metadata) and turns them 
> into abstract non-negative integers for the program to consume, there 
> is no reason to avoid (or promote) UTF-16 encoding over other forms. 
> The program itself just looks for things like "<" which maps to the 
> work specifically on EBCDIC values at the application level 
> (obviously, such an application can consume and produce EBCDIC content 
> if the programming library does the mapping from and to underneath), 
> hence why the text says "will get along just fine by operating on 
> character codes that lie in printable US-ASCII".
>
> Is it possible for a generic markdown editor, be it a script-sensitive 
> GUI tool or pico/notepad, to record words like "漢字" and "кириллица" 
> in a way that all markdown systems should recognize and render 
> correctly (e.g. transcoding to match HTTP preferences)?
>

Yes.

First, you can use whatever text encoding you prefer, such as UTF-8. The 
generic markdown editor shouldn't disturb those characters, and a 
generic markdown processor should just pass them through. (The generic 
markdown editor, if not UTF-8 aware, would just show some other byte 
values such as åÙ but wouldn't change the octets.)

Second, you can use entity references such as &#12343;. Those will be 
passed as-is to the output.

Sean

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 19 Apr 2015 14:25:17 +0900

> values such as åÙ but wouldn't change the octets.)

Please be careful. Some encodings, in particular UTF-8, are carefully 
designed to avoid using any 7-bit byte values when encoding non-ASCII 
characters. But other encodings, e.g. ISO-2022-JP and Shift_JIS, do use 
7-bit bytes, and for them the "just pass through" will not work. If 
possible, please reflect this in your draft (at least to the extent as 
to not imply that "pass through" will work with all encodings.

Regards,   Martin.

---

From: Paul Libbrecht <[email-masked]>
Date: Sun, 19 Apr 2015 19:20:16 +0200

Dear Murray,

I see this draft includes a UTI declaration which is very good and
allows interoperability on the OSX-related platforms.
Could you include a windows flavour name for Windows? I suspect the
simple word "Markdown" (without quotes) would be sufficient, unless you
know of existing editors that use that.

Unfortunately, the only markdown editors I know of, Folding Text on
MacOSX and StackEdit in browsers, do not export a Markdown-specific
flavours.
So... no big experiment doable.

Paul

On 14/04/15 21:25, Murray S. Kucherawy wrote:
> Call in APPSAWG and I wanted to see if there's anything he should fix
> before the formal request comes through.
>

---

From: Sean Leonard <[email-masked]>
Date: Sun, 19 Apr 2015 11:00:43 -0700

Hi Paul,

What do you mean by "flavours"? In the latest draft, the term "flavor" 
has been discontinued, in favor of "variant". I am not aware of the term 
"flavour name" being used as a Windows software development term.

Thanks,

Sean

---

From: Paul Libbrecht <[email-masked]>
Date: Sun, 19 Apr 2015 22:41:00 +0200

Thanks Sean,

I dug a bit and indeed the word flavor seems to be far from the MS
world. It may well be something that is special to Java... Sorry for the
wrong naming.

DataFormat would be the closest.
A list of predefined values is here:
    https://msdn.microsoft.com/en-us/library/system.windows.dataformats.aspx
paul

On 19/04/15 20:00, Sean Leonard wrote:
> has been discontinued, in favor of "variant". I am not aware of the
> term "flavour name" being used as a Windows software development term.
>
> Thanks,
>
> Sean
>