Subject: Re: [Nfvrg] Requesting review of and comments on draft-mlk-nfvrg-nfv-​reliability-using-co​

From: Dave Dolson <[email-masked]>
Date: Tue, 1 Dec 2015 17:05:51 +0000

Some comments:

1. In the draft there is a premise that software errors on COTS hardware are different from hardware/software errors on dedicated telecom equipment. Obviously there could be silent hardware and software errors on any type of device. If you are going to have this premise, I think there needs to be some evidence, citation or explanation. In my career I have seen some pretty nasty hardware defects. Some were fatal but some were silent. One could claim that functional hardware flaws are *less* likely in COTS hardware, based on wide adoption and wide testing base. And both types of systems are largely controlled by software.

2. You state COTS machines have relaxed design criteria. Obviously this is a huge generalization, not supported by any citations. Certainly there are also custom hardware solutions with relaxed design criteria. I think you need to be specific by referring to specific designs. E.g., RAM with error correction? Dual power supplies? RAID disks? And in that case, you should discuss hardware with or without such features, not “COTS vs. traditional”.

3. Some of the language in the draft is derogatory or dismissive of COTS equipment. I think you should avoid such language to be respectful of different designs. (use of “cowboy manner”, “COTS hardware and all its undesirable features”, “the COTS hardware and its related software are notorious” …) Don’t treat the COTS systems as poorly designed; rather see the design goals as different.

4. On the subject of five-9s, clearly the NFV approach is to create five-9s at the *system* level vs. the level of the individual box. Can you consistently adopt that definition? For example, it is confusing near the end of section 5 when you say “the requirements for traditional five 9s could be relaxed while keeping the same user experience in downtime.” Above that you say “with only two 9s server availability, the five 9s server part of  system availability can be achieved,” which I think is a more useful statement.

5. I think it would be helpful if you explained the function of the system being simulated. It seems like it is a database replicated on several servers?

My suspicion is that juxtaposing COTS vs. traditional telecom in this draft is distracting from the value of your research. The silent error idea could be applied to any type of hardware. 

Sorry, but I’m not comfortable commenting on your theoretical analysis or simulation. Yes, I know that's the main point of the work... But as seen above, I find the assumptions about COTS vs. traditional telecom hardware to undermine any conclusions.

-Dave

From: Nfvrg [mailto:[email-masked]] On Behalf Of B. Khasnabish
Sent: Saturday, October 17, 2015 8:24 PM
To: [email-masked]
Subject: [Nfvrg] Requesting review of and comments on draft-mlk-nfvrg-nfv-​reliability-using-co​

Dear All,
We have recently published a draft 
(datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/) on
"NFV Reliability using COTS Hardware" discussing the results of a recent study on the feasibility of using Commercial Off-The-Shelf (COTS) hardware for virtualized network functions in telecom equipment. 
This draft is based on the presentation 
(https://www.ietf.org/proceedings/92/slides/slides-92-nfvrg-9.pdf)
during IETF/IRTF-92 mtg. in Dallas, TX, USA on 
the 26th of March 2015. This draft attempts to address the work item 
no. 4 in the charter (http://datatracker.ietf.org/rg/nfvrg/charter/)
-------------------------------------------------------------------------------
4. Service Verification with Regards to Security and Resiliency
Reliability and security issues and relevant solutions related to the
nature of VNFs are the objectives of this work item.
----------------------------------------------------------------------------------
We are requesting your review of this draft and comments on it 
so that we can update the draft ASAP.
Many thanks in advance for your kind attention.
Best.
Bhumip
=====================================================

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Tue, 1 Dec 2015 17:36:18 +0000

Hi,

<unchair_mode>

 Just to express my support to all of Dave´s comments, and especially to the suspicion you express as conclusion. I see a great value in this draft from the point of view of strategies to achieve system-wide five-nines (or even beyond), and don’t see the need to confront COTS vs special-purpose telecom hardware.

</unchair_mode>

Be goode,

On 1 Dec 2015, at 18:05 , Dave Dolson <[email-masked]<mailto:[email-masked]>> wrote:

Some comments:

-Dave

From: Nfvrg [mailto:[email-masked]] On Behalf Of B. Khasnabish
Sent: Saturday, October 17, 2015 8:24 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [Nfvrg] Requesting review of and comments on draft-mlk-nfvrg-nfv-​reliability-using-co​

Dear All,
(datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/<http://datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/>) on
_______________________________________________
Nfvrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/nfvrg

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D
http://people.tid.es/diego.lopez/

e-mail: [email-masked]
Tel:    +34 913 129 041
Mobile: +34 682 051 091
----------------------------------

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: "B. Khasnabish" <[email-masked]>
Date: Wed, 2 Dec 2015 11:03:17 -0500

​Dave, Diego,

Many Thanks to both of you.

We plan to address these comments/suggestions in a future version of this
draft by

(a) adding appropriate citations and reasons behind the assumptions ,

​(b) adopting a consistent definition,

(c) exploring further the categories of COTS which are more suitable for
reliable services

(d) discussing any add-on features of COTS that  may make certain COTS
hardware more suitable for reliable services than others

(e) clearly mentioning the functions being simulated, and

(f) exploring how the concept of silent error can be applied to both
hardware and software.

Once again, many thanks, and we looking forward to other suggestions and
comments.

Best.

Bhumip
​

Best.
​

On Tue, Dec 1, 2015 at 12:36 PM, DIEGO LOPEZ GARCIA <
[email-masked]> wrote:

> Hi,
>
> <unchair_mode>
>
>  Just to express my support to all of Dave´s comments, and especially to
> the suspicion you express as conclusion. I see a great value in this draft
> from the point of view of strategies to achieve system-wide five-nines (or
> even beyond), and don’t see the need to confront COTS vs special-purpose
> telecom hardware.
>
> </unchair_mode>
>
> Be goode,
>
> On 1 Dec 2015, at 18:05 , Dave Dolson <[email-masked]> wrote:
>
> Some comments:
>
> 1. In the draft there is a premise that software errors on COTS hardware
> are different from hardware/software errors on dedicated telecom equipment.
> Obviously there could be silent hardware and software errors on any type of
> device. If you are going to have this premise, I think there needs to be
> some evidence, citation or explanation. In my career I have seen some
> pretty nasty hardware defects. Some were fatal but some were silent. One
> could claim that functional hardware flaws are *less* likely in COTS
> hardware, based on wide adoption and wide testing base. And both types of
> systems are largely controlled by software.
>
> 2. You state COTS machines have relaxed design criteria. Obviously this is
> a huge generalization, not supported by any citations. Certainly there are
> also custom hardware solutions with relaxed design criteria. I think you
> need to be specific by referring to specific designs. E.g., RAM with error
> correction? Dual power supplies? RAID disks? And in that case, you should
> discuss hardware with or without such features, not “COTS vs. traditional”.
>
> 3. Some of the language in the draft is derogatory or dismissive of COTS
> equipment. I think you should avoid such language to be respectful of
> different designs. (use of “cowboy manner”, “COTS hardware and all its
> undesirable features”, “the COTS hardware and its related software are
> notorious” …) Don’t treat the COTS systems as poorly designed; rather see
> the design goals as different.
>
> 4. On the subject of five-9s, clearly the NFV approach is to create
> five-9s at the *system* level vs. the level of the individual box. Can you
> consistently adopt that definition? For example, it is confusing near the
> end of section 5 when you say “the requirements for traditional five 9s
> could be relaxed while keeping the same user experience in downtime.” Above
> that you say “with only two 9s server availability, the five 9s server part
> of  system availability can be achieved,” which I think is a more useful
> statement.
>
> 5. I think it would be helpful if you explained the function of the system
> being simulated. It seems like it is a database replicated on several
> servers?
>
> My suspicion is that juxtaposing COTS vs. traditional telecom in this
> draft is distracting from the value of your research. The silent error idea
> could be applied to any type of hardware.
>
>
> Sorry, but I’m not comfortable commenting on your theoretical analysis or
> simulation. Yes, I know that's the main point of the work... But as seen
> above, I find the assumptions about COTS vs. traditional telecom hardware
> to undermine any conclusions.
>
>
> -Dave
>
>
>
> From: Nfvrg [mailto:[email-masked] <[email-masked]>] On
> Behalf Of B. Khasnabish
> Subject: [Nfvrg] Requesting review of and comments on
> draft-mlk-nfvrg-nfv-​reliability-using-co​
>
> Dear All,
> "NFV Reliability using COTS Hardware" discussing the results of a recent
> study on the feasibility of using Commercial Off-The-Shelf (COTS) hardware
> for virtualized network functions in telecom equipment.
> [email-masked]
> ------------------------------
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
> puede contener información privilegiada o confidencial y es para uso
> exclusivo de la persona o entidad de destino. Si no es usted. el
> destinatario indicado, queda notificado de que la lectura, utilización,
> divulgación y/o copia sin autorización puede estar prohibida en virtud de
> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
> que nos lo comunique inmediatamente por esta misma vía y proceda a su
> destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual or
> entity named above. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this transmission in error, do not read it. Please immediately reply to the
> sender that you have received this communication in error and then delete
> it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
> pode conter informação privilegiada ou confidencial e é para uso exclusivo
> da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
> indicado, fica notificado de que a leitura, utilização, divulgação e/ou
> cópia sem autorização pode estar proibida em virtude da legislação vigente.
> Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
> imediatamente por esta mesma via e proceda a sua destruição
>