Subject: [bmwg] concerning software reset measurements

From: "Ilya Varlashkin" <[email-masked]>
Date: Mon, 16 Aug 2010 10:46:14 +0200

Good morning group,

I've been pondering about 'software reset' section of
draft-ietf-bmwg-reset, and spoke to few people on the subject. The
question is what exactly are we trying to measure here - resetting the
DUT without making any changes to the software, or restart due to
software upgrade, or restart due to software crash? 

First case seems to be mostly academic excercise and does not
necessarily corelates to duration of outage in real-life scenarios. Why?
Because normally operators would not reload a box without changing
software (except as temporary workaround for memory leaks or similar).
On the other hand restart due to crash has few variables that influence
reset (outage) time. For example, watchdog-timeout type of reset implies
that DUT is unresponsive for some while before reset actually happens;
this means reset time will be larger compare to reset due to 'reload'
command. Additionally, software crash usually involves writing some
debug information which also takes time. Configuration settings may
influence amount of details in that info. With debug info disabled reset
will probably be quickest of all but will be not very representative of
real life. Crash summary enabled by default on given platform will
require more time. Finally, creating core dump may take ages. 

I believe the draft should specify condition of the reset more precesely
so that results are comparable. From operator's perspective the case
with exception-initiated reset and typical debug dump is the most
interesting, but I see value in having measurements for other scenarios
as well. So my proposal is to ammend section 4.2 with description of
three following mandatory test parameters:

1) crash with debug info disabled
2) crash with typical info (what vendor's support team will ask for)
enabled
3) crash with full (most complete for given DUT platform) debug info

Additionally, I'd like to poll the group and ask Al (as the chair) if it
would be valuable and feasible to expand scope of the software reset to
include reset due to software upgrade? Ammending draft-ietf-bmwg-reset
is probably not good idea as it will delay progress of the draft, so
there could be separate draft that addresses measurement of reset due to
software upgrade. Would that be still within scope of BMWG?

Cheers,
iLya

---

From: Al Morton <[email-masked]>
Date: Mon, 16 Aug 2010 08:28:15 -0400

Hi Ilya,

Thanks for making time to attend bmwg again in Maastricht,
and for raising these issues at the meeting and again on the list.

At 04:46 AM 8/16/2010, Ilya Varlashkin wrote:
>...I believe the draft should specify condition of the reset more precesely

I think the challenge is to specify the "crash" in the black-box world
of bmwg: a repeatable crash that has relevance across the OS's of
different vendors' DUTs.

>software upgrade. Would that be still within scope of BMWG?

I think it would be within the scope, IF the black-box & multi-OS problems
can be solved.  So, I think you've wisely anticipated that the current
draft should proceed while looking into the crash/upgrade aspects.

Al
bmwg chair

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Mon, 16 Aug 2010 15:44:24 +0200

> -----Original Message-----
> >scenarios as well. So my proposal is to ammend section 4.2 with 
> >description of three following mandatory test parameters:
> >
> I think the challenge is to specify the "crash" in the 
> black-box world of bmwg: a repeatable crash that has 
> relevance across the OS's of different vendors' DUTs.
> 

would following be suitable?

"Crash - an automatic reset of the DUT or one or more of its components
caused by a condition where system is unable to perform its functions
and must reinitialize itself to a known state, or a simulation of such
reset by an operator. Unlike regular reset (initiated by operator) crash
often records some information about the system at the time of
occurance, this information helps post-mortem analysis of the problem
that caused crash. For the purpose of reset measurements the experiments
MUST be performed with such configuration as found on regular production
system of the same kind, i.e. level of recorded information should be
the same as what technical support would expect to see from a crashed
production system."

This sounds quite abstract to be applicable to most of platforms, yet it
leaves less room for doctoring conditions in order to obtain best
values.

> >to include reset due to software upgrade? Ammending 
> >draft-ietf-bmwg-reset is probably not good idea as it will delay 
> >progress of the draft, so there could be separate draft that 
> addresses 
> >measurement of reset due to software upgrade. Would that be 
> still within scope of BMWG?
> 
> I think it would be within the scope, IF the black-box & 
> multi-OS problems can be solved.  So, I think you've wisely 
> anticipated that the current draft should proceed while 
> looking into the crash/upgrade aspects.
> 

here is one possible pragmatic definition: "reset due to software
upgrade - a reset of the DUT performed in order to activate new software
version of the whole system or one or more of its component while
leaving configuration unchanged. In other words, software version on the
DUT or its components before the reset is different from the software
after the reset while configuration is not". 

/iLya

---

From: Al Morton <[email-masked]>
Date: Mon, 16 Aug 2010 11:48:33 -0400

There's also an insightful comment from Steve Maxwell in
the minutes (which I'm working on at the moment):

Essentially, SW upgrade includes the time to load the upgrade
(if it can't be done in parallel with normal operation)
and the time to restore operation afterward.

Al
bmwg chair

At 09:44 AM 8/16/2010, Ilya Varlashkin wrote:
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 16 Aug 2010 11:05:14 -0500

Al while I understand this  point, different platform / vendors will
deploy different technologies  to  upgrade software or even patches
(tftp vs ftp / serial line transfer etc ..  , in service  software
upgrades may or may not be supported some patches may required a
software / system  reload while others may not ... , so don't you think
that this will get us into platform / vendor  specific implementation
terrain? 

Fernando 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Al Morton
Sent: Monday, August 16, 2010 11:49 AM
To: Ilya Varlashkin; [email-masked]
Subject: Re: [bmwg] concerning software reset measurements

Al
bmwg chair

---

From: [email-masked]
Date: Mon, 16 Aug 2010 16:09:00 +0000

Defining the return to operational state to make the benchmark will be challenging.  Does the startup config need to be fully loaded? Do routing protocol sessions need to be established?  Do route tables need to be updated? Do traffic filter rules need to be applied?  If we are measuring our benchmarks on the data plane then I say "yes" to all of these.

Scott

Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: "Al Morton" <[email-masked]>
Sender: <[email-masked]>
Date: Mon, 16 Aug 2010 11:48:33 
To: Ilya Varlashkin<[email-masked]>; <[email-masked]>
Subject: Re: [bmwg] concerning software reset measurements

Al
bmwg chair

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Mon, 16 Aug 2010 18:19:21 +0200

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Monday, August 16, 2010 6:09 PM
> To: Al Morton; [email-masked]; Ilya Varlashkin; [email-masked]
> Defining the return to operational state to make the 
> benchmark will be challenging.  Does the startup config need 
> to be fully loaded? Do routing protocol sessions need to be 
> established?  Do route tables need to be updated? Do traffic 
> filter rules need to be applied?  If we are measuring our 
> benchmarks on the data plane then I say "yes" to all of these.
> 
None different from reset as currently defined by the draft - we should
measure time between last packet before outage and first packet after
outage. The only change is what caused reset.

To answer to Fernando's concern that we tread into implementation waters
- I don't think so. If we consider 'software upgrade' as a function with
working system as input and working system as output, we don't care what
exactly happens inside the function, we're only interested how long does
it take to produce output. So it looks precisely as a black box.

/iLya

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 16 Aug 2010 11:23:28 -0500

Well, don't forget that we have to provide a methodology to follow  for
each test case, having to describe in addition to the rest how to
perform the upgrade / patch  installation will place  us definitely into
implementation waters 

Rgds 

Fernando 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ilya Varlashkin
Sent: Monday, August 16, 2010 12:19 PM
To: [email-masked]
Subject: Re: [bmwg] concerning software reset measurements

/iLya

---

From: Andrew Yourtchenko <[email-masked]>
Date: Mon, 16 Aug 2010 17:57:16 +0200 (CEST)

Hi Ilya,

thanks for the comments and indeed the offline discussions. They were 
quite useful. My personal thinking about this is below.

> necessarily corelates to duration of outage in real-life scenarios. Why?

It seems merely one of clarification of rfc2544 that was "Cause a 
reset in the DUT."

> require more time. Finally, creating core dump may take ages.

Agree 100%. This number of factors might be a reason why the behaviour is 
specified as it is - to have the least common denominator that can be a 
base for something else.

> 3) crash with full (most complete for given DUT platform) debug info

I know in our offline chats I put this one into the "reset" basket - but 
decomposing it now, I think the "crash" may be treated as a
superset of a "reset" much as the "upgrade" would be - in one case we're 
reshuffling the configuration/state data to be suitable for the new 
version, in the other case we're storing the old runtime state (or 
are in a process of detecting the irrecoverable nature of the problem) 
before proceeding with the software reset.

> software upgrade. Would that be still within scope of BMWG?

As per our offline discussions, agree and I'll be happy to help. Same goes 
about the "crash" one if you think it is useful to spend more time on. I 
think it might be a fruitful exercise - like you mention, there can be 
multiple types of typical software malfunctions that result in a reset 
after saving the runtime state.

thanks,
andrew

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Mon, 16 Aug 2010 18:33:55 +0200

> Well, don't forget that we have to provide a methodology to 
> follow  for each test case, having to describe in addition to 
> the rest how to perform the upgrade / patch  installation 
> will place  us definitely into implementation waters 
> 
we don't care what specific steps are necessary to perform the upgrade.
All we concerned about is that we bring system from one operational
state to another operational state. Consider this on an example: vendor
A says I must change software from version X1 to version X2, as a
network operator I'd like to know how long system will be unavailable
for forwarding traffic, vendor A says it will be K1 minutes; same time
vendor B says to go from version Y1 to Y2 would take my system out of
service for K2 minutes, I'd like to know if K1 and K2 are comparable of
if one of the vendors took typical upgrade conditions while another took
best case scenario which is unlikely to happen in real life.

Kind regards,
iLya

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Mon, 16 Aug 2010 11:40:46 -0500

Inline ,,, 

-----Original Message-----
From: Ilya Varlashkin [mailto:[email-masked]] 
Sent: Monday, August 16, 2010 12:34 PM
To: Fernando Calabria (fcalabri); [email-masked]
Subject: RE: [bmwg] concerning software reset measurements

>> actually we do, if you look at the draft each test case has a
methodology attached to it , since we want to be sure that the operator
/ tester follows it whule benchmarking different platforms /
implementations .... 

>> Like I mentioned before I do see your point here, but vendor A may
support ISSU and vendor Y may not so  (this is the point I am trying to
make) 

Rgds 

Fernando 

Kind regards,
iLya

---

From: Ilya Varlashkin <[email-masked]>
Date: Mon, 16 Aug 2010 19:39:30 +0200

> support ISSU and vendor Y may not so  (this is the point I am trying to

and from network operator perspective I don't care how each platform is upgraded - ISSU or not - all I'm interested in is how long a system will be out of service. So each vendor is free to offer any way of upgrading a system and from the measurements PoV they will be all equal to the black boxes.

Kind regards,
iLya