Subject: [rfc-i] Proposed survey - preview

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 11:40:06 +1300

The I-D authors survey is now ready for preview and I would appreciate a few of you to check it over before I send it:

	https://ietf.co1.qualtrics.com/jfe/preview/SV_8d21RFqe5dBijOJ?Q_CHL=preview&Q_SurveyVersionID=

It’s taken a lot longer to get to the preview stage than expected as we’re in the process of moving to a new survey system (to Qualtrics from SurveyMonkey).  I still have some work to do on look and feel so this preview is a bit stark.  Question numbers will not appear when it goes live, they’re there for reference for any comments.

thanks
Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 28 Oct 2020 10:23:52 +1100

Thanks Jay.  This is pretty good.  I filled out answers, which is the best way to find issues.

Q13 and Q14 have "Other" options, but I found that I wanted a free-form response field instead.

Q19 seems to open-ended to answer anything other than "maybe".  I think that people might have very different conceptions of what a modern toolchain is.  I know that my own views differ somewhat from Jay's, which is fine, but it leaves the answer here difficult.  Maybe a freeform field asking what an improved or ideal toolchain looks like is better.  Some people might think that this is a WYSIWY* editor, or web pages that allow for annotation and changes, or something else entirely.  

Maybe in the form:  Do you think that a more modern toolchain for draft authoring would be worth having?  [y/n] If so, what would that toolchain look like?

On Wed, Oct 28, 2020, at 09:40, Jay Daley wrote:
> The I-D authors survey is now ready for preview and I would appreciate 
> a few of you to check it over before I send it:
> 
> It’s taken a lot longer to get to the preview stage than expected as 
> we’re in the process of moving to a new survey system (to Qualtrics 
> from SurveyMonkey).  I still have some work to do on look and feel so 
> this preview is a bit stark.  Question numbers will not appear when it 
> goes live, they’re there for reference for any comments.
> 
> thanks
> Jay
> 
> -- 
> Jay Daley
> IETF Executive Director
> [email-masked]
>

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 14:58:58 +1300

Thanks Martin

> On 28/10/2020, at 12:23 PM, Martin Thomson <[email-masked]> wrote:
> 

Both of theses "Other" options have a freeform text box associated with them - did that not appear?

> Maybe in the form:  Do you think that a more modern toolchain for draft authoring would be worth having?  [y/n] If so, what would that toolchain look like?

You’re right this is not a good question.  I think it’s probably too early to ask it so I’ll remove it.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 15:06:11 +1300

Thanks Toerless

> On 28/10/2020, at 12:30 PM, Toerless Eckert <[email-masked]> wrote:
> 
> Nice work, Jay
> 
> Quite exhaustive/exhausting. Is there some incentive associated with
> this ? Blanket, t-shirts, tickets for free IESG lunch at next in-person IETF ?

If our response level ever drops then I will certainly consider incentivisation but so far people have been very generous with their time.

> 
> This is actually a better overview of all the tools related to authoring
> then i think i have seen on any tools web page. Unless i am wrong and there
> is such a web page, i would highly recommend to create one that just
> lists all the stuff from the query (maybe with URLs where to get the stuff).

I want to get the data from this survey before doing just that.

> 
> Was there / will there be a questionaire for consumers of I-D/RFCs ?
> Aka: As you would have already seen i think it would be good to
> query the community what they think about the new format.

Nothing planned as of yet.

> 
> For example stuff like:
>  How do you like that new Text RFCs are without page number
>  on pages and TOC ? 
> 
>  [a] Its wonderful. What is a page ?
>  [b] I only read the XML.
>  [b] I was told to suck it up.
> 
> Of course i think there are a lot of really useful questions to ask about
> consumption. Happy to help proposing some.
> 
> Ok, finally the feedback you want:
> 
> The intro text should explain or point to an explanation how
> the collected data is or is not available to whom and how it will
> be used.

Will do.

> 
> I think there should be a review at the end: Here is the list of
> answers you gave... and then click submit or be able to go back and
> correct/change answerse. The 100% page and you're done and can not
> go back comes as a surprise.

Will do.

> 
> removing question numbers will make discussing/referring to them
> more difficult. They should be kept.

My experience is that they reduce the completion rate as they make a survey seem longer than it is and we rarely get feedback on surveys outside of the survey so I will remove them and see if anyone complains.

> 
> the rendering on my chrome browser cuts off a lot of stuff because i
> am scaling up fonts to better read them. Aka: bad HTML syndrome for
> qualtrics.

Can you send me a screenshot directly please?  One of the nice things with Qualtrics is that I can override the default CSS.

> 
> The pages seem to be differently long. In general i'd say put fewer
> questions on a page. Maybe there is a theme/title that could be
> given to each page..

A title is a good idea.

Jay

> 
> Cheers
>   Toerless
> 
> P.S.: I guess/hope the tools team will sell you the names of all draft
> authors of the last 5 years so you can reach them all. If not, feel free
> to just spam all IETF mailing lists. This qualifies to do so ;-)
> 
> On Wed, Oct 28, 2020 at 11:40:06AM +1300, Jay Daley wrote:
>> It???s taken a lot longer to get to the preview stage than expected as we???re in the process of moving to a new survey system (to Qualtrics from SurveyMonkey).  I still have some work to do on look and feel so this preview is a bit stark.  Question numbers will not appear when it goes live, they???re there for reference for any comments.
>> 
>> thanks
>> Jay
>> 
>> -- 
>> Jay Daley
>> IETF Executive Director
>> [email-masked]
>> 
> ---
> [email-masked]

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 28 Oct 2020 02:10:06 +0000

Hiya,

I agree with Martin that this seems like a good survey.
Minor comments below, all are ignorable (so no need for
a specific response to me on this).

- I'm not sure if it's needed but would suggest you think
about a question about operating systems/platforms. I
generally find that not a problem (on Ubuntu) but it may
be for some people for some tools, not sure.

- The juxtaposition of "ease of use" and "XML v3 format"
seems a bit funny - that's ok though, it gave me a
chuckle:-)

- I'd maybe suggest a few "it's good enough even if it's
cumbersome" options.

- I've not used that survey platform but would be
interested in how privacy (un)friendly it may be. If
it's possible to say more about that when you send out
the mail (via a URL or in text) that would be good.

Cheers,
S.

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 28 Oct 2020 13:36:42 +1100

> them - did that not appear?

It did, but I found that I wanted to address comments toward the concrete options presented, and the box was a little small too.  And I didn't want to say "important" or "not important" for my feedback.  I was looking to add a sentence or two at most, but the slot was more suited to the name of a thing.

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 15:37:02 +1300

Thanks Stephen

> On 28/10/2020, at 3:10 PM, Stephen Farrell <[email-masked]> wrote:
> 
> 
> Hiya,
> 

I looked at this early on and I think it’s too low level.

Will do.

> the mail (via a URL or in text) that would be good.

As with most survey systems it has an anonymous mode where all we get is the data that is explicitly provided and we will run all our surveys in this mode.  

For full disclosure, I am hoping for a later survey to ask people if they want to sign up to ietf-announce and we will collect email addresses for that purpose at that point but only if we can disassociate them from the survey responses.

Jay

> <OpenPGP_0x5AB2FAF17B172BEA.asc>

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 15:47:32 +1300

> On 28/10/2020, at 3:36 PM, Martin Thomson <[email-masked]> wrote:
> 

Ah.  I’ve made that box bigger.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 28 Oct 2020 16:37:58 +1300

"How often have you used the following document format(s)...."

I find the conflation "associated output process(es) (editor/template/converter)" confusing. The editor isn't an *output* process, it's an input process. Editing an XML file is completely disjoint from converting it. 

You don't really want people to click on almost every "never" do you? "(Ignore any you don’t know about)" should probably be
"(Ignore any you never use or don’t know about)" and then you could drop the "never" buttons.

"Markdown using the kramdown-rfc2629 convertor" Well, I don't know for sure. I use the kramdown option at https://xml2rfc.tools.ietf.org/experimental.html but with v3 output. So which convertor is that? The web service is noticeably absent in the list but a lot of people use it without knowing what specific tools lie behind it.

"How important are the following characteristics of the v3 XML format to you?"

Ease of use? I don't see how that is even a question about a format.

Regards
   Brian Carpenter

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 27 Oct 2020 21:00:35 -0700

On Tue, Oct 27, 2020 at 8:38 PM Brian E Carpenter <
[email-masked]> wrote:

> I find the conflation "associated output process(es)
> (editor/template/converter)" confusing. The editor isn't an *output*
> process, it's an input process. Editing an XML file is completely disjoint
> from converting it.
>
> You don't really want people to click on almost every "never" do you?
> "(Ignore any you don’t know about)" should probably be
> "Markdown using the kramdown-rfc2629 convertor" Well, I don't know for
> sure. I use the kramdown option at
> https://xml2rfc.tools.ietf.org/experimental.html but with v3 output. So
> which convertor is that? The web service is noticeably absent in the list
> but a lot of people use it without knowing what specific tools lie behind
> it.
>

Without taking a position on the rest of the survey.... I am currently
doing auth48 for documents which were produced with xmlv2 and I expect to
soon have to do it for documents which were prepared in markdown. This is
relevant because I otherwise would not need to come in contact with xmlv3
at all. So I'm not sure I would call it "ease of use" but certainly there
is some characteristic here in the realm of "how easy is it to work with",
I think.

-Ekr

> > The I-D authors survey is now ready for preview and I would appreciate a
> few of you to check it over before I send it:
> >
> >
> we’re in the process of moving to a new survey system (to Qualtrics from
> SurveyMonkey).  I still have some work to do on look and feel so this
> preview is a bit stark.  Question numbers will not appear when it goes
> live, they’re there for reference for any comments.
> >
> > thanks
> > Jay
> >
>

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 20:16:06 +1300

Thanks Brian

> On 28/10/2020, at 4:37 PM, Brian E Carpenter <[email-masked]> wrote:
> 

If that’s the answer yes.

> "(Ignore any you never use or don’t know about)" and then you could drop the "never" buttons.

I’ve learnt from the IETF surveys that I’ve done so far that I can’t omit a non-answer because people simply choose another option even if its incorrect.

> 
> "Markdown using the kramdown-rfc2629 convertor" Well, I don't know for sure. I use the kramdown option at https://xml2rfc.tools.ietf.org/experimental.html but with v3 output. So which convertor is that?

It’s the kramdown converter.  If that wasn’t clear then I would expect you to use the "Markdown using a different output process" option.

> The web service is noticeably absent in the list but a lot of people use it without knowing what specific tools lie behind it.

There’s a later question about how people run these tools and that includes "On an IETF public web service".

> Ease of use? I don't see how that is even a question about a format.

Anecdotally I’ve heard from quite a few people that they use markdown because they find it easier to use than XML.  This is one way of testing that.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Oct 2020 08:50:56 +0100

> If our response level ever drops then I will certainly consider incentivisation but so far people have been very generous with their time.

It probably doesn’t help that the survey starts with a question a proper answer for which requires significant research by the respondent.
(How many I-Ds have I written in the last two years?  What is the datatracker interface to find out?)
Generally, you want to start with easy questions and only present the hard ones when the respondent already has invested enough time into the survey to reluctantly answer the hard ones, too.

I still don’t like calling kramdown-rfc a “converter”.  I don’t call my C compiler that, nor the HTML renderer that I call browser.  “Converter” conjures the image of a one-time conversion after which you go on by editing the converted result.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Oct 2020 09:05:07 +0100

> (How many I-Ds have I written in the last two years?  What is the datatracker interface to find out?)

Oh, and when have I “authored an I-D in the last two years”?
E.g., have I authored https://datatracker.ietf.org/doc/draft-ietf-cbor-7049bis/ this year?  In 2013?  (I wrote what became RFC 7049 in 2013 in markdown, then up-converted the RFC-editor XML to markdown again when we started work on the bis in 2017, then worked on the bis for three and a half years.)

Grüße, Carsten

---

From: Jay Daley <[email-masked]>
Date: Wed, 28 Oct 2020 22:20:02 +1300

> On 28/10/2020, at 9:05 PM, Carsten Bormann <[email-masked]> wrote:
> 

Thanks for reminding me that I intended to change that and add a reference to datatracker in response to earlier feedback.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Oct 2020 13:00:23 +0100

Am 27.10.2020 um 23:40 schrieb Jay Daley:
> ...

Looks good to me.

Best regards, Julian

---

From: Jay Daley <[email-masked]>
Date: Thu, 29 Oct 2020 07:05:50 +1300

Thanks Tony those are all good suggestions.

Jay

> On 29/10/2020, at 3:15 AM, HANSEN, TONY L <[email-masked]> wrote:
> 
> Trying not to repeat earlier comments.
> 
> I too filled it out and spotted these additional things:
> 
> ====
> Question: How do you ensure that your draft is correctly formatted/validated when you submit it? (check all that apply)
> 
> Shouldn't "I use the xml2rfc.tools.ietf.org web interface to check the format" be one of the standard replies? This is a totally different step from "submitting". It's also how I normally generate my html/text/pdf versions for private review.
> 
> 
> ====
> Question: How often do you use the following conversion tools? (you can ignore any you don't use)
> 
> 	The response "xml2rfc" could cover both the standalone tool AND using the web site.
> 
> 
> ====
> Question: Why do you think the uptake of the v3 XML as an authoring format is still so low? (check all that apply)
> 
> This presupposes something that the survey-taker already knows. I suggest adding a bit more introduction:
> 
> 	The majority of the inputs sent to the IETF internet-draft submission tool are still predominantly in the v2 XML and text formats. Why do you think submission of v3 XML is still so low?
> 
> Also, given that the various markdown tools GENERATE the xml that is then submitted, AND the tools working with markdown are NOT "IETF provided tools", don't you think an additional answer would be appropriate:
> 
> 	Not supported well enough by the non-IETF-provided tools/toolchain that I use.
> 	 
> ====
> Question: How satisfied are you with the following characteristics of authoring tools?
> 
> To paraphrase another quote: "The wonderful thing about authoring tools is that there are so many to choose from." And they vary VERY widely in quality, plus I'm definitely not a user of many of them. So I'm not sure how to answer this question.
> 
> Would it be better to make the question about "the authoring tools that you use"?
> 
> 
> Tony
> 
> On 10/27/20, 6:40 PM, "rfc-interest on behalf of Jay Daley" <[email-masked] on behalf of [email-masked]> wrote:
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 29 Oct 2020 10:15:43 +1300

On 28-Oct-20 20:16, Jay Daley wrote:
> Thanks Brian
> 
>> On 28/10/2020, at 4:37 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>

I think they won't. At least I won't, when there are so many irrelevant lines.

> I’ve learnt from the IETF surveys that I’ve done so far that I can’t omit a non-answer because people simply choose another option even if its incorrect.

How about pre-selecting "never" so that it becomes the default?
 
> There’s a later question about how people run these tools and that includes "On an IETF public web service".

Right, so here's the problem. People who use the web service for conversion simply don't know what tools they are using; they only know which radio buttons they select. And there are two web services for conversion (https://xml2rfc.tools.ietf.org/ and https://xml2rfc.tools.ietf.org/experimental.html). The whole "How often have you used the following document format(s)...." section makes little sense for users of the web service(s). I think you need to ask the web service question *first* and divert people who say Yes away from the whole suite of questions that are irrelevant to them.

> Anecdotally I’ve heard from quite a few people that they use markdown because they find it easier to use than XML.  This is one way of testing that.

Yes, but a format isn't a tool and the question seems to confound the two. I'd rather use XML Copy Editor than TECO for editing either XML or markdown; that's a tooling question. I have mixed feelings about kramdown vs XML, but I'd phrase the question as "preference" rather than "ease of use".

    Brian

> [email-masked] <mailto:[email-masked]>
>