Subject: Re: [rfc-i] missing line ends in generated TXT (from datatracker)

From: Julian Reschke <[email-masked]>
Date: Mon, 4 Nov 2019 19:18:51 +0100

On 25.10.2019 08:16, Julian Reschke wrote:
> On 21.10.2019 18:46, Julian Reschke wrote:
>> On 21.10.2019 18:14, Dave Rice wrote:
>>> Hi all,
>>>
>>>> On Oct 9, 2019, at 1:38 PM, Henrik Levkowetz <[email-masked]
>>>> <mailto:[email-masked]>> wrote:
>>>>
>>>> Hi David,
>>>>
>>>> On 2019-10-09 17:32, Dave Rice wrote:
>>>>> Thanks Henrik,
>>>>>
>>>>>> On Sep 20, 2019, at 5:26 PM, Henrik Levkowetz <[email-masked]
>>>>>> <mailto:[email-masked]>> wrote:
>>>>>>
>>>>>> Hi Michael, David,
>>>>>>
>>>>>> On 2019-09-20 23:04, Michael Richardson wrote:
>>>>>>>
>>>>>>> Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
>>>>>>>> Another annoyance here is the the xml processor in the submission
>>>>>>>> tool
>>>>>>>> and the xml processor at
>>>>>>>> https://xml2rfc.tools.ietf.org/experimental.html
>>>>>>>> <https://xml2rfc.tools.ietf.org/experimental.html> perform
>>>>>>>> differently,
>>>>>>>> so there’s effectively no test environment to use before
>>>>>>>> submitting.
>>>>>>>> Dave
>>>>>>>
>>>>>>> Dave, can you please try xml2rfc.ietf.org
>>>>>>> <http://xml2rfc.ietf.org>, which is a different installation,
>>>>>>> supported by a different entity?
>>>>>>
>>>>>> Michael:  Umm, no, that redirects to tools.ietf.org
>>>>>> <http://tools.ietf.org>, too, and the processor
>>>>>> behind that, and behind
>>>>>> https://xml2rfc.tools.ietf.org/experimental.html,
>>>>>> _and_ behind the submission tool are all the latest release of
>>>>>> xml2rfc.
>>>>>>
>>>>>> David: I'd be happy to look into any discrepancies you see, but I
>>>>>> can only
>>>>>> do so if you provide observations and input XML files for me to work
>>>>>> with.  It won't happen if I don't know about it :-)
>>>>>
>>>>
>>>>> When I use the tool at https://xml2rfc.tools.ietf.org or
>>>>> https://xml2rfc.tools.ietf.org/experimental.html, the plain text
>>>>> output appears ok, but when I use
>>>>> https://datatracker.ietf.org/submit/
>>>>> <https://datatracker.ietf.org/submit/>, the contents of the
>>>>> <sourcecode> elements (which are new in xml2rfc version 3) have their
>>>>> line breaks stripped out and thus cause an error when the line length
>>>>> limits are exceeded.
>>>>
>>>> That sounds like a bug.
>>>>
>>>>> For the moment, I’m proposing a find and replace to convert the newer
>>>>> <sourcecode> elements back to <artwork> elements as in
>>>>> https://github.com/FFmpeg/FFV1/pull/171/commits/a018f091534678f45cc157bfd154518ad6267c7e
>>>>>
>>>>>
>>>>> <https://github.com/FFmpeg/FFV1/pull/171/commits/a018f091534678f45cc157bfd154518ad6267c7e>,
>>>>>
>>>>>
>>>>> thus an xml2rfc version 2 processor or version 3 processor should
>>>>> proceed accurate results though not as semantically clear in the
>>>>> xml.
>>>>>
>>>>>> FWIW, if you have atools.ietf.org <http://tools.ietf.org/>login, you
>>>>>> can use the xml2rfc
>>>>>> issue
>>>>>> tracker athttps://trac.tools.ietf.org/tools/xml2rfc/trac/newticket,
>>>>>> for
>>>>>> xml2rfc issues, andhttps://trac.tools.ietf.org/tools/ietfdb/newticket
>>>>>> for datatracker issues.
>>>>>
>>>>> I don’t have a login, but I’ll see if I can make a tiny test file for
>>>>> this issue and file a concise ticket.
>>>>
>>>> Please don't put time into this; I'm perfectly happy to work from the
>>>> xml
>>>> file you found problematic on submission.  Just let me know the name
>>>> of the
>>>> draft in question, so I can pull it.
>>>
>>> I just uploaded this new version of the EBML specification to the
>>> datatracker.
>>>
>>>
>>> On lines 559-609 and 824-939 there are <sourcecode> nodes that contain
>>> multi-line xml expressions. During the upload the idnit test showed no
>>> issues or nits at all.  However in the resulting files at
>>> https://datatracker.ietf.org/doc/draft-ietf-cellar-ebml/12/ the outputs
>>> remove the line-breaks of the multi-line xml within the <sourcecode>
>>> nodes. So in the HTML version the <sourcecode> is shown as a single
>>> line, see
>>> https://www.ietf.org/id/draft-ietf-cellar-ebml-12.html#name-ebml-schema-example,
>>>
>>>
>>> and in the plain text as well, see Section 11.1.1 of
>>> https://www.ietf.org/id/draft-ietf-cellar-ebml-12.txt.
>>>
>>> In another document, I have been find/replacing the <sourcecode> node
>>> with the older <artwork> node as a workaround.
>>>
>>> Best Regards,
>>>
>>> Dave Rice
>>> ...
>>
>> I had the same problem and reported it as
>> <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/444>.
>>
>> Best regards, Julian
>
> Henrik,
>
> could you please let us know what the status is? Does it make sense to
> retry now?
> ...

Henrik,

any news on this? Any idea what could be causing this, and when it'll be
fixed?

Best regards, Julian
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Julian Reschke <[email-masked]>
Date: Thu, 7 Nov 2019 19:42:52 +0100

> any news on this? Any idea what could be causing this, and when it'll be

Ok, so this was reported first by Dave ~1 month ago.

Who's in charge and can give us an update?

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Thu, 7 Nov 2019 14:05:15 -0600

Julian -

I think that in the Project Manager role (particularly its priority 
management aspect), I'm "who's in charge" for the question you ask.

The update I can give you on this particular ticket is that it has been 
on the radar, but hasn't been higher priority than many of the immediate 
changes needed by the RPC as we first took v3 into production.

The rate of such changes is slowing (but only a little), and the nature 
of them are such that I think this particular issue will likely make the 
top of the stack within the next few weeks (remember there is an IETF 
meeting in that time).

Visibility into the full set of issues and prioritization is less than I 
would like - many issues were reported and addressed through direct 
communication between the RPC and Henrik, and I feel that was the right 
way to handle them given the nature of the cut-over. As the rate is 
becoming more sane, I expect we'll see increased use of the ticketing 
system for this set of tools.

RjS

---

From: Julian Reschke <[email-masked]>
Date: Thu, 7 Nov 2019 21:50:41 +0100

> system for this set of tools.

Robert,

thanks for the update.

It's indeed the lack of information that was bugging me mainly. So maybe
we need a ticket system for datatracker? Or is there one that I should
have used?

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Thu, 7 Nov 2019 14:57:02 -0600

On 11/7/19 2:50 PM, Julian Reschke wrote:
> we need a ticket system for datatracker? 

You meant the v3 format tools perhaps?

(The datatracker's ticket system entry point is at 
https://trac.tools.ietf.org/tools/ietfdb/newticket.)

> Or is there one that I should
> have used?
You had already entered this at 
<https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/444>, so I'm not 
sure what your question is?
>
> Best regards, Julian
>
> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
>
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
>
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Thu, 7 Nov 2019 22:40:22 +0100

On 07.11.2019 21:57, Robert Sparks wrote:
> ... >> thanks for the update.
>>

Ok.

> sure what your question is?

Well, the tickets that I raise there mostly get no feedback at all, so
at this point I'm not sure it's worth using that venue.

Best regards, Julian

---

From: Dave Rice <[email-masked]>
Date: Mon, 11 Nov 2019 16:23:34 -0500

> On Nov 7, 2019, at 4:40 PM, Julian Reschke <[email-masked]> wrote:
> 

FWIW, for the cellar working group, we pipe our rfc xml through sed to find all occurrence of <sourcecode> and replace them with <artwork>. It’s a loss in semantics but seems to workaround this issue for now. We’ll watch the progress of the ticket and remove our workaround when it’s resolved.
Dave Rice

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 12 Nov 2019 16:28:33 +0100

Hi David,

On 2019-11-11 22:23, Dave Rice wrote:
> FWIW, for the cellar working group, we pipe our rfc xml through sed
> to find all occurrence of <sourcecode> and replace them with
> <artwork>. It’s a loss in semantics but seems to workaround this
> issue for now. We’ll watch the progress of the ticket and remove our
> workaround when it’s resolved.

Another workaround is to generate the txt file yourself, and upload with
the XML file.  I've started looking into why standalone invocation of
xml2rfc handles <sourecode> correctly, while invocation as a library
function looses newlines, but I don't have an answer yet.

Best regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Nov 2019 17:14:32 +0100

> function looses newlines, but I don't have an answer yet.

A wild guess would be that the XML whitespace treatment might be
affected by the grammar (when the parser is in validating mode).

See:

>  artwork =
>      element artwork {
>        attribute xml:base { text }?,
>        attribute xml:lang { text }?,
>        attribute anchor { xsd:ID }?,
>        attribute pn { xsd:ID }?,
>        attribute xml:space { text }?,
>        [ a:defaultValue = "" ] attribute name { text }?,
>        [ a:defaultValue = "" ] attribute type { text }?,
>        attribute src { text }?,
>        [ a:defaultValue = "left" ]
>        attribute align { "left" | "center" | "right" }?,
>        [ a:defaultValue = "" ] attribute alt { text }?,
>        [ a:defaultValue = "" ] attribute width { text }?,
>        [ a:defaultValue = "" ] attribute height { text }?,
>        attribute originalSrc { text }?,
>        (text* | svg)
>      }

>    sourcecode =
>      element sourcecode {
>        attribute xml:base { text }?,
>        attribute xml:lang { text }?,
>        attribute anchor { xsd:ID }?,
>        attribute pn { xsd:ID }?,
>        [ a:defaultValue = "" ] attribute name { text }?,
>        [ a:defaultValue = "" ] attribute type { text }?,
>        [ a:defaultValue = "false" ]
>        attribute markers { "true" | "false" }?,
>        attribute src { text }?,
>        attribute originalSrc { text }?,
>        text
>      }

So those differ in the presence of xml:space, and the actual content
model (just text vs text or <svg>).

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 12 Nov 2019 19:59:19 +0100

Hi Dave,

> function looses newlines, but I don't have an answer yet.

I've found the problem and patched the datatracker.  When draft submission
starts again, this should be handled correctly.

Best regards,

	Henrik