Subject: Re: [irtf-discuss] RFCs accepted journal articles

From: Joe Touch <[email-masked]>
Date: Thu, 02 May 2013 16:35:43 -0700

IETF-stream and IRTF-streams are often considered peer-reviewed.

Other streams may or may not be considered peer-reviewed (I would not 
consider IAB documents as such), but independent submissions are 
generally not (they only purpose of the one review they get is to make 
sure they're on-topic and not bogus).

Joe

On 4/25/2013 7:08 AM, [email-masked] wrote:
>
>   at least a few university communities consider RFCs as peer reviewed papers.
>
> /bill
>
>
> On Thu, Apr 25, 2013 at 12:42:54PM +0000, Yaakov Stein wrote:
>> I just checked and my request is marked "under consideration" with no-one else voting for it.
>>
>> See https://feedback.researchgate.net/responses/please-allow-enter-ietf-rfcs-as-peer-reviewed-papers
>>
>> Perhaps if some other ResearchGate members vote for it the RG staff will take notice.
>>
>> Y(J)S
>>
>>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Yaakov Stein
>> Sent: 25 April, 2013 15:33
>> To: marcelo bagnulo braun; SM
>> Cc: [email-masked]; [email-masked]
>> Subject: Re: [irtf-discuss] RFCs accepted journal articles
>>
>>
>> I submitted a request to ResearchGate to accept RFCs as a valid publication type.
>>
>> However, I never received any feedback from them.
>>
>> Y(J)S
>>
>>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of marcelo bagnulo braun
>> Sent: 25 April, 2013 15:19
>> To: SM
>> Cc: [email-masked]; [email-masked]
>> Subject: Re: [irtf-discuss] RFCs accepted journal articles
>>
>> I already did quite a while ago. No reply so far.
>>
>> El 25/04/13 11:00, SM escribis:
>>> Hi Marcelo,
>>> At 01:32 25-04-2013, marcelo bagnulo braun wrote:
>>>> What would be really useful would be that the RFC series are included
>>>> in the Journal Citation Report. I have asked this to the RFC editor a
>>>> while ago, but i didnt received any reply.
>>>>
>>>> This would go a very long way (at least in this part of the world) to
>>>> make them valid as a an merit for the academics.
>>>
>>> I suggest asking the RSE about the above.
>>>
>>> Regards,
>>> -sm
>>>
>>

---

From: Jari Arkko <[email-masked]>
Date: Thu, 2 May 2013 20:11:35 -0400

> Other streams may or may not be considered peer-reviewed (I would not consider IAB documents as such), but independent submissions are generally not (they only purpose of the one review they get is to make sure they're on-topic and not bogus).

There are two review steps for the independent submissions, the ISR and the IESG review. I was under the impression that while the latter is just about non-conflict with the IETF process, the former would  have looked at content and worthwhileness to publish. But I've never been a part of the review board, so I don't really know…

Anyway, not trying to deflect your main point but just wondering about this one detail.

Also, IAB does publish documents with a comment period, but it is true that they are not obliged to take feedback into account, so in theory they are not peer reviewed while in practice they often are.

Jari

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 May 2013 17:31:16 -0700

On 5/2/2013 5:11 PM, Jari Arkko wrote:
> There are two review steps for the independent submissions, the ISR
> and the IESG review. I was under the impression that while the latter is
> just about non-conflict with the IETF process, the former would have
> looked at content and worthwhileness to publish. But I've never been a
> part of the review board, so I don't really know…

The information is here: http://www.rfc-editor.org/indsubs.html

ISR checks to make sure a doc is non-bogus and relevant to be an RFC.

IESG is a check on whether the doc should be part of the IETF process, 
or is somehow dangerous to the Internet in content.

But neither is a true content quality review as would be expected of a 
peer-review process.

> Also, IAB does publish documents with a comment period, but it is
> true that they are not obliged to take feedback into account, so in
> theory they are not peer reviewed while in practice they often are.

Any of us can solicit feedback and use that to make our documents better.

The IAB as a group decides what to publish and whether to incorporate 
comments, according to RFC 4845. That's self-publication, not peer review.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 May 2013 21:02:33 -0700

On May 2, 2013, at 8:25 PM, Brian E Carpenter <[email-masked]> wrote:

> On 03/05/2013 12:31, Joe Touch wrote:
>> 
>> 
>>>> generally not (they only purpose of the one review they get is to
>>>> make sure they're on-topic and not bogus).
>>> 
> Joe, I don't know where you get that from.

I gave the URL. 

> The ISE review is a peer review
> with the option of anonymity for the reviewer and feedback to the authors
> or to the RSE only.
> 
> This is covered in the CCR article, and anyone trying to justify the
> academic credentials of the RFC series is welcome to cite the CCR
> article; that's why we wrote it.
> http://www.sigcomm.org/sites/default/files/ccr/papers/2010/January/1672308-1672315.pdf
> 
>   Brian

RFC Editor website vs CCR article.  

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 03 May 2013 15:25:04 +1200

> But neither is a true content quality review as would be expected of a

Joe, I don't know where you get that from. The ISE review is a peer review
with the option of anonymity for the reviewer and feedback to the authors
or to the RSE only.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 03 May 2013 17:35:19 +1200

> RFC Editor website vs CCR article.  

I'm afraid I don't see the inconsistency.

    Brian

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 May 2013 22:52:38 -0700

> I'm afraid I don't see the inconsistency.

The web pages imply that everything in scope will be published, albeit if revised. Peer review would include peer rejection for quality too. 

Joe

---

From: "Tschofenig, Hannes (NSN - FI/Espoo)" <[email-masked]>
Date: Fri, 3 May 2013 06:53:05 +0000

> sure they're on-topic and not bogus).

FWIW: There is a lot of review happening for IAB documents from the IAB itself, from members in IAB programs (if the document belongs to a program) and also from the IETF community since the documents are typically last called.

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 May 2013 09:45:04 -0700

On 5/3/2013 9:40 AM, Phillip Hallam-Baker wrote:
> Peer review is just another name for community feedback.

http://en.wikipedia.org/wiki/Peer_review

A key component is the significant potential for rejection.

> Yes I know they dress it up as something different but seriously, I 
> don't think I am the only person who has done peer review for academic 
> journals here. the only reason I have stopped doing peer review for 
> journals is that my policy is that if someone else is going to profit 
> from my work product then they have to pay me for it.

And yet you participate in the IETF?

> We all know where the literature is going. The UK has already 
> established the principle that all work that is supported by public 
> funds has to be available to the public without cost. That model is 
> going to win. The UK is more than big enough to establish critical 
> mass even if the for-profit publishers manage to stave off that type 
> of policy being imposed in the US by lobbying (i.e. bribing) congress.

I completely agree that the UK is big enough to drive global issues. 
That's why the US drives on the left.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 May 2013 09:14:59 -0700

On 5/2/2013 11:53 PM, Tschofenig, Hannes (NSN - FI/Espoo) wrote:
> FWIW: There is a lot of review happening for IAB documents from the
> IAB itself, from members in IAB programs (if the document belongs to a
> program) and also from the IETF community since the documents are
> typically last called.

That's community feedback, not peer review.

Peer review includes a recommendation not to publish, and non-junk 
venues have rejection rates >50% (typically around 30% or less).

Joe

---

From: Aaron Yi DING <[email-masked]>
Date: Fri, 03 May 2013 20:37:29 +0300

> venues have rejection rates >50% (typically around 30% or less).

Reference here for instance

SIGCOMM
http://www.cs.ucsb.edu/~almeroth/conf/stats/#sigcomm 
<http://www.cs.ucsb.edu/%7Ealmeroth/conf/stats/#sigcomm>
MobiCom
http://www.cs.ucsb.edu/~almeroth/conf/stats/#mobicom 
<http://www.cs.ucsb.edu/%7Ealmeroth/conf/stats/#mobicom>
NSDI
http://www.cs.ucsb.edu/~almeroth/conf/stats/#nsdi 
<http://www.cs.ucsb.edu/%7Ealmeroth/conf/stats/#nsdi>

-- Aaron

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 May 2013 10:38:05 -0700

On 5/3/2013 9:58 AM, Phillip Hallam-Baker wrote:
> I am not a 'free software' zealot. I have never had a problem with 
> someone taking my designs and implementing them.
>
> What sticks in my craw is when people propose that they take my work, 
> make it their exclusive property and limit access to it. That does not 
> add any value to me.

So how do you feel when you provide advice or feedback to corporate IETF 
participants and they later reveal they own IPR on that protocol?

Joe

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 12:40:24 -0400

Yes I know they dress it up as something different but seriously, I don't
think I am the only person who has done peer review for academic journals
here. the only reason I have stopped doing peer review for journals is that
my policy is that if someone else is going to profit from my work product
then they have to pay me for it.

We all know where the literature is going. The UK has already established
the principle that all work that is supported by public funds has to be
available to the public without cost. That model is going to win. The UK is
more than big enough to establish critical mass even if the for-profit
publishers manage to stave off that type of policy being imposed in the US
by lobbying (i.e. bribing) congress.

There will be an interesting transition process but the reputation of a
journal is only as good as the people who write for it. A lot of for-profit
journals will disappear and a lot of not-for-profit institutions that fund
themselves through publishing overpriced journals to academic libraries are
going to be in doo-doo even deeper than the newspapers are right now. The
paper journals that die will be quickly replaced by online journals
supported by open source workflow tools, many of which have the same
editorial boards as the ones that are pushed under.

The journal publishers are intermediaries that do not add value. They are
facing a disruptive change in their business model and they have absolutely
no chance of survival. None.

[And remember that I am not an open source zealot by any means]

So while I think it is good that we support the academics here, I don't
think we should worry too much about the extent to which it is going to
help them. Any tenure committee that is deciding a position in computer
networking that is not capable of evaluating the worth of a technical paper
on its own merits has no business being a tenure committee. They should
resign and take emeritus immediately.

When I looked at various frauds being perpetrated in the reputation systems
for online auctions and the like, my starting point was my experience of
the well known and long practiced frauds in 'peer-review'. Almost no peer
review is anonymous in a meaningful sense. Everyone who is established in
their field has a pretty good idea of what everyone else works on and what
their pet ideas are. The link spamming schemes designed to foil search
engine rankings were anticipated by decades by the citation cartels that
infest most of the literature.

I don't think we should worry unduly about the old system. Do whatever is
necessary to at least have the information being tracked. The review we
give is at least as good as 'academic' peer review. The people who are
trying to insist on a distinction are the legacy journal publishers who are
about to lose their entire market.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 12:58:34 -0400

Well first off I am actually paid to participate in IETF, so it isn't an
issue.

I did participate in IETF while I was retired but that was because:

* IETF did not charge me to participate
* IETF makes my immediate work product (specifications) available at no
charge
* I couldn't spend all my time making daleks even if I wanted to

I am not a 'free software' zealot. I have never had a problem with someone
taking my designs and implementing them.

What sticks in my craw is when people propose that they take my work, make
it their exclusive property and limit access to it. That does not add any
value to me.

On Fri, May 3, 2013 at 12:45 PM, Joe Touch <[email-masked]> wrote:

> http://en.wikipedia.org/wiki/**Peer_review<http://en.wikipedia.org/wiki/Peer_review>
>

-- 
Website: http://hallambaker.com/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 14:23:38 -0400

On Fri, May 3, 2013 at 1:38 PM, Joe Touch <[email-masked]> wrote:

>> make it their exclusive property and limit access to it. That does not add
>> any value to me.
>>
>

That is the reason that I think WGs should specify their IPR policy for
contributions when they are formed. I find it completely unacceptable.

I do have pending patent applications that might eventually appear in IETF
as proposed standards but if I submit them I will certainly be telling
people that I have the claims or else how are they going to know to pay me?
If someone has a patent though, they should probably write the specs at
their own expense and submit as individual submissions so the documents can
be recorded and tracked. IETF process is not really suited to a case where
one party has essential controlling patent claims and would be useless if
there were conflicting claims.

Unfortunately, that is the problem that the IETF IPR policy attempts to
address which is not the same as the principal IPR problems we face. And
the cost of policy compliance is still very high.

Intentional abuse of IETF process by actual IETF participants is a very
rare event if it has happened at all. Much more common problems are:

* Participant files 'defensive' patents, goes bust or otherwise loses
control of the IPR which is bought by a troll.

* Participant has IPR that is relevant but not critical to a proposal but
their legal department is unwilling to consider reasonable commercial terms
for making it available.

* A patent applicant reads traffic on IETF mailing lists and either files a
new patent with claims covering the ideas or adds claims to a continuation
in part of an existing patent application.

* An IPR holder files a bad faith patent suit knowing that it is going to
be much less expensive for the targets to settle than to defend the suit.

* Some participants in a WG have obtained a license for certain IPR and
they attempt to forestall competition that have not obtained licenses by
making it essential to the specification.

I have encountered all the above numerous times.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 04 May 2013 08:09:01 +1200

One final comment at the end...

> The web pages imply that everything in scope will be published, albeit if revised. Peer review would include peer rejection for quality too. 

In that case the web pages miss out an important fact: the ISE sometimes says "No".

    Brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 May 2013 13:29:17 -0700

On 5/3/2013 1:14 PM, Nico Williams wrote:
> And the IESG can request that the RFC-Editor not publish.

Only for dangerous or end-run around IETF process. Not for mere quality 
issues.

Joe

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 17:01:27 -0400

Tim Berners-Lee submitted a paper on the WWW to Hypertext '92, it was
rejected.
In 1993 they asked him to give the keynote.

I don't think getting papers published in the lit is a valid or useful
criteria for career progression. Any tenure committee that is relying on
peer review to decide is stupid.

I am happy for the IETF to do whatever is necessary to for academics to
gain checklist compliance. Debugging a stupid process is not our problem.

On Fri, May 3, 2013 at 4:29 PM, Joe Touch <[email-masked]> wrote:

> ______________________________**_________________
> rfc-interest mailing list
> [email-masked]
> https://www.rfc-editor.org/**mailman/listinfo/rfc-interest<https://www.rfc-editor.org/mailman/listinfo/rfc-interest>
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 03 May 2013 14:34:05 -0700

On 5/3/2013 1:50 PM, Nico Williams wrote:
> On Fri, May 3, 2013 at 3:29 PM, Joe Touch <[email-masked]> wrote:
> The IETF process itself can lead to non-publishing.  How many more
> RFCs are there than expired I-Ds?

Expired can mean abandoned for various reasons (lack of resources, 
etc.), not just rejected from publication.

The key metric is how many IDs were rejected after being submitted for 
publication.

Joe

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 May 2013 15:14:19 -0500

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 May 2013 15:50:47 -0500

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 May 2013 16:42:54 -0500

On Fri, May 3, 2013 at 4:34 PM, Joe Touch <[email-masked]> wrote:
> Expired can mean abandoned for various reasons (lack of resources, etc.),
> not just rejected from publication.
>

That's a misleading metric.  The number of such rejections is probably
in the single digits.  Abandonment of I-Ds is often a result of lack
of interest because the proposed RFC was crap, or there's no interest
in the technology in question, but there's no way to measure that --
at best you could do some sampling and make your own [often
subjective] determination of why an I-D was abandoned, something that
requires a lot of effort.

An I-D that reaches WGLC or IETF LC is one that has reached consensus
to get pretty down far on the road to publication, so any problems
with it will be fixed rather than result in abandonment.

RFC publication rejection is informal, and highly social.  But it does
happen, and plenty.

Nico
--

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sun, 05 May 2013 20:41:23 +0200

Just one clarification.

I am not sure if this the point that you were trying to make but in 
journal there are also editorial papers that are usually not considered 
peer reviewed.

I mean, my point is that even if there are some RFCs that are not peer 
reviewed, this should not be an obstacle to consider the RFCs that are 
peer reviewed as peer reviewed papers imho

El 03/05/13 01:35, Joe Touch escribió:
>>> See 
>>> https://feedback.researchgate.net/responses/please-allow-enter-ietf-rfcs-as-peer-reviewed-papers
>>>
>>> From: [email-masked] 
>>> [mailto:[email-masked]] On Behalf Of Yaakov Stein
>>> From: [email-masked] 
>>> [mailto:[email-masked]] On Behalf Of marcelo bagnulo 
>>> braun

---

From: Michael Richardson <[email-masked]>
Date: Mon, 06 May 2013 08:53:08 -0400

>>>>> "Nico" == Nico Williams <[email-masked]> writes:
    >> Only for dangerous or end-run around IETF process. Not for mere
    >> quality issues.

    Nico> The IETF process itself can lead to non-publishing.  How many
    Nico> more RFCs are there than expired I-Ds?

I'd still like to non-publish the documents in the CFR (anti-RFC) or
some such series.  Probably in HTML only, with a CSS of:
  color: #fff;
  background-color: #fff;

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [