Subject: Re: [Xml-sg-cmt] Ugly pdf hack

From: Kesara Rathnayake <[email-masked]>
Date: Wed, 2 Mar 2022 14:21:34 +1300

On 1/03/22 2:40 pm, John R Levine wrote:
> On Tue, 1 Mar 2022, Kesara Rathnayake wrote:
>> These page breaks happen because CSS rules stop `sourcecode` from 
>> breaking.
>> When `sourcecode` is large enough, render chooses to use the whole 
>> page for `sourcecode`.
>>
>> Another way to fix this form xml2rfc is:
>>
>> Option 1. Remove CSS rule that avoid inside breaks for `sourcecode`.
>>
>> Option 2. xml2rfc can add a class that allows `sourcecode` to break 
>> inside when `sourcecode` is larger than 80 lines or so.
>>
>> Option 3. Schema change. We can introduce a new attribute to 
>> `sourcecode` element to tell it's okay to break inside. So xml2rfc can 
>> use a different class for that.
> 
> If option 2 isn't too hard that seems like a reasonable thing to try.  
> Not sure 80 lines is the right number, but we can try some experiments.

I have attached, 4 PDFs for RFC9181 using different line numbers to add 
breakable class.
File name format: rfc9181.<line numbers>.pdf

Apart from 50 lines output, other PDFs caused a h2-h3 header separation 
resulting in a page break (see page 66 in other PDFs).

Let me know using if 50 lines to add a new breakable class is okay.

   --Kesara

> 
> Regards,
> John Levine, [email-masked], Taughannock Networks, Trumansburg NY
> Please consider the environment before reading this e-mail. https://jl.ly
> 

-- 
Kesara Rathnayake
Senior Software Development Engineer - IETF LLC
[email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 1 Mar 2022 21:37:00 -0500

On Wed, 2 Mar 2022, Kesara Rathnayake wrote:
>>> Option 2. xml2rfc can add a class that allows `sourcecode` to break inside 
>>> when `sourcecode` is larger than 80 lines or so.
>>> 
>>> Option 3. Schema change. We can introduce a new attribute to `sourcecode` 
>>> element to tell it's okay to break inside. So xml2rfc can use a different 
>>> class for that.
>> 
>> If option 2 isn't too hard that seems like a reasonable thing to try.  Not 
>> sure 80 lines is the right number, but we can try some experiments.
>

The 50 line one looks pretty good to me.

Question for Sandy and Alice: in this document the sourcecode blocks are 
mostly long blocks of numbers so it makes little difference where the 
breaks are.  Can you think of docs that had sourcecode or artwork blocks 
that are 50 lines and that it would have been a problem to break them over 
two pages?  Breaking 10 or 20 lines would be annoying, but 50 lines seems 
like more than one would look at in a single glance.

I can see that breaking an ascii-art diagram would be bad, but this is 
just for sourcecode, right?

Regards,

---

From: Alice Russo <[email-masked]>
Date: Fri, 4 Mar 2022 14:38:18 -0800

Hi Kesara,
Thanks for sending these examples.

> Let me know using if 50 lines to add a new breakable class is okay.

- Pro: the A, A.1, A.1.1 are not on separate pages anymore
- Con: the <sourcecode> is broken after just 3 lines (in the 50, 60, 70, and 80 examples).
Is it correct that it's not possible w/ WeasyPrint currently to "only break a piece of sourcecode (or artwork) after keeping [max lines] of it together on a page"?

If proceeding w/ a change to PDF page breaking, would appreciate reviewing the output of its effect on multiple files. For example, files from https://www.rfc-editor.org/in-notes/prerelease/
(where I'd have to look for specific useful examples).

Thanks,
Alice

> On Mar 1, 2022, at 5:21 PM, Kesara Rathnayake <[email-masked]> wrote:
> 
> 
> 
>>> These page breaks happen because CSS rules stop `sourcecode` from breaking.
>>> When `sourcecode` is large enough, render chooses to use the whole page for `sourcecode`.
>>> 
>>> Option 2. xml2rfc can add a class that allows `sourcecode` to break inside when `sourcecode` is larger than 80 lines or so.
>>> 
>>> Option 3. Schema change. We can introduce a new attribute to `sourcecode` element to tell it's okay to break inside. So xml2rfc can use a different class for that.
>> If option 2 isn't too hard that seems like a reasonable thing to try.  Not sure 80 lines is the right number, but we can try some experiments.
> 
> I have attached, 4 PDFs for RFC9181 using different line numbers to add breakable class.
> Apart from 50 lines output, other PDFs caused a h2-h3 header separation resulting in a page break (see page 66 in other PDFs).
> 
> [email-masked]<rfc9189.50.pdf><rfc9189.60.pdf><rfc9189.70.pdf><rfc9189.80.pdf><rfc9189.xml>-- 
> Xml-sg-cmt mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml-sg-cmt

---

From: Alice Russo <[email-masked]>
Date: Fri, 4 Mar 2022 15:00:13 -0800

Here's one; thanks, Sandy. Would be looking at p. 3 and p. 4 breaks.
input: https://www.rfc-editor.org/in-notes/prerelease/rfc9169.notprepped.xml
PDF: https://www.rfc-editor.org/rfc/rfc9169.pdf (from xml2rfc 3.12.0)

Thanks,
Alice

> On Mar 4, 2022, at 2:38 PM, Alice Russo <[email-masked]> wrote:
> 
> Hi Kesara,

---

From: Kesara Rathnayake <[email-masked]>
Date: Mon, 7 Mar 2022 13:51:31 +1300

> PDF: https://www.rfc-editor.org/rfc/rfc9169.pdf (from xml2rfc 3.12.0)

I've attached the PDF file with 50 line sourcecode break change.
Unwanted page breaks are gone now.

   --Kesara

---

From: Alice Russo <[email-masked]>
Date: Tue, 8 Mar 2022 09:31:02 -0800

Thanks for this file, Kesara. Here's where I think we are:

Issue 739 (https://github.com/ietf-tools/xml2rfc/issues/739) is re: RFC-to-be 9189 w/ bad page breaks in PDF.
Potential fix: Kesara added a class that allows sourcecode to break when it is larger than 50 lines. 
Q: Is the PDF sent 3/1 (rfc9189.50.pdf) acceptable for publication? 
Q: Are the changes acceptable when (potentially) rerendering PDF files in the future?

As far as 9189:
• Pro: the A, A.1, A.1.1 are not on separate pages anymore
• Con: the <sourcecode> is broken after just 3 lines (in the 50, 60, 70, and 80 examples).

I have some concern about introducing this change to how PDF page breaking is handled without reviewing the PDF output of previously published RFCs (which is not a minor task) bc presumably (if they are rerendered at a later date), the page breaks will be significantly different.

Thanks,
Alice

> On Mar 6, 2022, at 4:51 PM, Kesara Rathnayake <[email-masked]> wrote:
> 
> 
> 
> [email-masked]<rfc9169.notprepped.pdf>-- 
> Xml-sg-cmt mailing list
> [email-masked]

---

From: Alice Russo <[email-masked]>
Date: Wed, 9 Mar 2022 11:56:55 -0800

Hi Kesara,
My understanding from talking w/ John and Sandy yesterday is to review more PDF output before proceeding. That said, this effort is not needed if I'm off base about unintended consequences when potentially rerendering PDF in the future (lots of page breaks within sourcecode). Any thoughts on that?

Below are 2 types of input files:
1) docs w/ PDFs w/ page break issues (list courtesy of Jean)
2) docs w/ <sourcecode> over 50 lines (list courtesy of John)

Could you render the PDFs for some of these? (The lists overlap.) Or, I could do a local setup of dev instance (following steps from previous PDF review) and post files. For reviewing the output, John offered to look some over. Evaluation criteria is perhaps:
- Is this PDF better or worse that it was before? or
- Does putting more page breaks in <sourcecode> blocks matter? (John suggests no.)

Thanks,
Alice
--

1) bad page breaks in the PDF between headings and sourcecode and/or artwork
https://www.rfc-editor.org/in-notes/prerelease/rfc8829.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc8855.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc8866.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc9002.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc9022.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc9083.notprepped.xml
https://www.rfc-editor.org/in-notes/prerelease/rfc9067.notprepped.xml (a case where the current algo mostly works; it has quite a bit of sourcecode in the way of YANG trees and modules)

2) docs with multiple <sourcecode> elements that contain over 50 lines 
[I suggest using the not prepped file as input, but it shouldn't matter.]

rfc8652.xml lines  104, 104, 1460
rfc8675.xml lines  205, 72
rfc8676.xml lines  72, 168, 261, 599, 408, 75
rfc8695.xml lines  178, 1134, 122
rfc8696.xml lines  92, 84, 72
rfc8727.xml lines  51, 147, 947, 1429
rfc8743.xml lines  305, 56, 58, 67, 71, 73, 74, 61, 342, 54
rfc8748.xml lines  100, 204
rfc8776.xml lines  3381, 477
rfc8782.xml lines  69, 69, 105, 178, 532
rfc8783.xml lines  53, 55, 809, 52, 53
rfc8794.xml lines  52, 177
rfc8795.xml lines  124, 340, 1953, 2509, 313, 1784
rfc8805.xml lines  183, 113
rfc8807.xml lines  57, 116
rfc8819.xml lines  121, 96
rfc8829.xml lines  68, 53, 88, 82, 53, 52, 55, 55
rfc8846.xml lines  505, 359, 481
rfc8847.xml lines  51, 205, 363, 482
rfc8851.xml lines  55, 64
rfc8876.xml lines  84, 85
rfc8881.xml lines  73, 176, 121, 215, 100, 54, 50, 77
rfc8895.xml lines  103, 55
rfc8896.xml lines  94, 52, 62, 78
rfc8913.xml lines  108, 1317, 50, 50
rfc8916.xml lines  99, 801, 85, 122
rfc8944.xml lines  97, 587, 171, 197
rfc8960.xml lines  68, 464, 224
rfc8976.xml lines  130, 50
rfc8995.xml lines  169, 58, 61, 50, 52, 80
rfc9002.xml lines  70, 70
rfc9020.xml lines  87, 82, 244, 556, 50, 50
rfc9022.xml lines  454, 131, 171, 81, 106, 108, 297, 165, 125, 54, 55, 82, 83, 62, 256, 75, 432, 510
rfc9038.xml lines  58, 58, 53, 58
rfc9051.xml lines  61, 873
rfc9058.xml lines  73, 56, 122
rfc9061.xml lines  718, 144, 984, 141, 573, 128, 155
rfc9067.xml lines  70, 876, 112, 68
rfc9083.xml lines  117, 75, 156, 274, 129, 121, 74, 75, 76
rfc9094.xml lines  944, 1638
rfc9095.xml lines  51, 75
rfc9102.xml lines  59, 99, 65, 72, 71, 95, 58, 79
rfc9106.xml lines  61, 61, 51
rfc9115.xml lines  100, 219
rfc9127.xml lines  87, 87, 150, 98, 161, 663, 79, 154, 146, 191, 190, 98
rfc9132.xml lines  70, 70, 134, 187, 615
rfc9166.xml lines  72, 70, 955, 53
rfc9167.xml lines  52, 51, 231
rfc9169.xml lines  87, 79
rfc9179.xml lines  280, 53
rfc9180.xml lines  51, 50, 57, 60
rfc9181.xml lines  71, 112, 2254
rfc9182.xml lines  57, 56, 53, 76, 62, 56, 3073, 123, 125, 62
rfc9195.xml lines  89, 268
rfc9196.xml lines  173, 264, 90, 71

> On Mar 8, 2022, at 9:31 AM, Alice Russo <[email-masked]> wrote:
> 
>

---

From: Kesara Rathnayake <[email-masked]>
Date: Thu, 10 Mar 2022 13:43:24 +1300

> My understanding from talking w/ John and Sandy yesterday is to review more PDF output before proceeding. That said, this effort is not needed if I'm off base about unintended consequences when pot…

This change can affect how page breaks were done in already published RFCs.
Do new versions of xml2rfc have to guarantee that it can generate the 
exact same output for old RFCs?
I'm not even sure that is something we can guarantee, because xml2rfc 
rely on external libraries like Weasyprint.

> - Does putting more page breaks in <sourcecode> blocks matter? (John suggests no.)

This _temporary_ website has a list of rendered PDFs with the proposed 
change to sourcecode rendering: http://devbox.amsl.com:8000/
Note that the proposed change is done only for the sourcecode element.

   --Kesara

---

From: "John R Levine" <[email-masked]>
Date: 9 Mar 2022 20:42:18 -0500

On Thu, 10 Mar 2022, Kesara Rathnayake wrote:
> Do new versions of xml2rfc have to guarantee that it can generate the exact 
> same output for old RFCs?

Definitely not.  That's what we mean when we say the PDF is not canonical.

It would of course be desirable for new versions of xml2rfc to produce 
PDFs no worse than before.

Regards,

---

From: Robert Sparks <[email-masked]>
Date: Thu, 10 Mar 2022 10:16:32 -0600

On 3/9/22 7:42 PM, John R Levine wrote:
> Definitely not. 

Absolutely - we designed strong intention that regeneration wouldn't 
duplicate the old output, especially in trying to "look" exactly the same.

We're not going to break the general look and feel of anything casually, 
but if, for instance, the pdf reflows differently on regeneration it's 
ok (for the worried -that does mean page numbers might change even).

> That's what we mean when we say the PDF is not canonical.
>
Right - it's going to be subjective, and that's really the best metric 
to optimize over.

---

From: Sandy Ginoza <[email-masked]>
Date: Thu, 10 Mar 2022 13:56:23 -0800

Hi all,

The authors of RFC 9189 would like to publish the document without further delay.  I think it’s reasonable to remove this one from TI and publish under the assumption we can re-render at some time in the future if page-breaking changes are adopted.  Any objections? 

Thanks,
Sandy 

> On Mar 10, 2022, at 8:16 AM, Robert Sparks <[email-masked]> wrote:
> 
> 
>>> Do new versions of xml2rfc have to guarantee that it can generate the exact same output for old RFCs?
>> 
>> Definitely not. 
> 
> Absolutely - we designed strong intention that regeneration wouldn't duplicate the old output, especially in trying to "look" exactly the same.
> 
> We're not going to break the general look and feel of anything casually, but if, for instance, the pdf reflows differently on regeneration it's ok (for the worried -that does mean page numbers might change even).
> 
>> It would of course be desirable for new versions of xml2rfc to produce PDFs no worse than before.
> Right - it's going to be subjective, and that's really the best metric to optimize over.
>> 
>> Regards,

---

From: "John R Levine" <[email-masked]>
Date: 10 Mar 2022 20:39:35 -0500

> The authors of RFC 9189 would like to publish the document without further delay.  I think it’s reasonable to remove this one from TI and publish under the assumption we can re-render at some time…

If they can live with the white space, it's fine with me.

Regards,

---

From: Sandy Ginoza <[email-masked]>
Date: Mon, 21 Mar 2022 05:09:18 -0700

Hi Alice,

I was going to ask the group where we are with this, as 9215 had lots of bad page breaks in the appendix also.  I went ahead and published though, because it was similar to other docs we’ve already published (i.e., it wasn’t multiple pages of just a header).  Do you think I should still send mail as a reminder, as I’m sure we’ll come across more bad breaks shortly.

Thanks,
Sandy

---

From: Sandy Ginoza <[email-masked]>
Date: Mon, 21 Mar 2022 05:10:58 -0700

Hi all,

lol — so, just checking to see where we are with this.  Do we still want to change how the pages are being broken? 

Thanks,
Sandy 

> On Mar 21, 2022, at 5:09 AM, Sandy Ginoza <[email-masked]> wrote:
> 
> Hi Alice,
>