Subject: Re: [bmwg] Software Update Doc

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Mon, 4 Mar 2013 14:14:41 -0500

> On 2/25/13 2:16 PM, Sarah Banks wrote:
>David Newman wrote:
> ...
> > Having said this, I think that if or when downgrades are covered, the
> > methodology would be exactly the same. Would you agree?
> 
> Yes.
> 
> dn

I tend to agree, possibly without the download step in some DUTs
(though this seems to be included in section 6 on rollback).

I'm happy to see this draft in text form, after much discussion
and support for the topic in general (when on the agenda in the past).

Here are some other comments, below.

Al
(as a participant)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Editorial:  this paragraph indent is a little distracting,
I assume this is a MSWord source?  Let's talk doc formatting...

1. Introduction

ISSU is a technique implemented by forwarding devices to upgrade or
   downgrade from one software version to another as applicable. The

-=-=-=-=-

Editorial:  it's a good idea to avoid wording that might lead to acceptance
criteria in bmwg specs. I'll try to flag them all, but here's an example:

3.1. Software Download
In this first phase, ...
... Such separation
allows an administrator to download the new code inside or outside
of a maintenance window; it is expected that downloading new code
and saving it to disk on the router will not impact operations.

s/expected/anticipated/

-=-=-=-=-

later in 3.1, Verification is purely functional testing, did it happen as planned
or not?

... Verification should include both positive verification (ensuring
that an ISSU action should be permitted) as well as negative tests
(creation of scenarios where the verification mechanisms would
report exceptions).

It would be good to separate procedural verifications as pre-requisites
for the benchmarking tests that follow. All good benchmarks assume/require
proper functions are performed. I see that similar verifications are
included in section 3.2, Software Staging, because at that stage they are an
inextricable part of the process.

-=-=-=-=-

3.3 Upgrade Run

2nd paragraph
This is the critical phase of the ISSU, where the control plane MUST
not be impacted and any interruptions to the forwarding plane should
be minimal to none.

This is a case where the ideal results have been expressed as a requirement
for the DUT to meet, rather than an anticipated result.

I suggest NEW:
There will be continuous monitoring of control-plane and forwarding-plane
functions, anticipating that any interruptions SHOULD be characterized.

-=-=-=-=-=-
Editorial:
4.1 Test Topology
The hardware configuration of the DUT (Device Under test) MUST be
identical to the one expected to be or currently deployed in
production.

suggest inserting the phrase below at end of first sentence:
... in order for the benchmark to have relevance in production.

-=-=-=-=-=-
4.2 Load Model
2nd para
For mixed protocol environments, frames SHOULD be distributed
between all the different protocols. The distribution SHOULD
approximate the network conditions of deployment. In all cases, the
details of the mixed protocol distribution MUST be included in the
reporting.

?? what protocols are being mixed?  IPv4 and IPv6? please clarify

-=-=-=-=-=-

later, at the end of 4.2:

All in all, the load model should attempt to simulate the production
network expectations to the greatest extent possible in order to
maximize the applicability of the results generated.

s/expectations/conditions/

-=-=-=-=-

5.2 Software Staging

General comment:  It is simpler in a procedure like this to ask the tester
to record the current time at different steps, then calculate the durations
later.

Start time:                          T0
Secondary RP stabilized to new code: T1
Start of Upgrade Run:                T2
Completion of Upgrade:               T3

Then when reporting (section 7), it looks like:

                                    Duration
Software Download/Secondary Update: T1 - T0
Upgrade Run:                        T3 - T2
...

worth a try?

-=-=-=-=-

5.3 Upgrade Run

4th para
... Examine the traffic
generators for any indication of traffic loss over this interval. If
the Test Set reported any traffic loss interval, note the duration
of the outage as "TP".

The above requires more detail, so that outage durations are 
reported the same way.  I think the recently approved MPLS
protection methods can provide some help here:
http://tools.ietf.org/html/draft-ietf-bmwg-protection-meth-14#page-12

-=-=-=-=-

5.4 Post ISSU
... last bullet

* Document the hitless or outage dark windows detected based upon
the (TP) counter value (provided by the Test Set)

update to reflect the metric chosen: although "outage dark windows"
sounds pretty cool, I don't know how to measure them.

---

From: Sarah Banks <[email-masked]>
Date: Tue, 5 Mar 2013 01:14:17 +0000

Hi Al,
	Thanks for your feedback. 
Let me answer inline.

> (though this seems to be included in section 6 on rollback).

So let me point out - the document scopes ISSU, not ISSD; and a change to the next rev (after we post on March 11) to discuss WHAT makes an ISSU an ISSU; for example, it'll deal with questions like, "Can I upgrade just the Line Cards and not the Route Engines and still have it be an ISSU?" Rather than get into which vendors do what for downgrades, as I think there's a good amount of misunderstanding and confusion there, I thought we were clear on the scope. I am not opposed to ISSD; but I do have reservations.

> and support for the topic in general (when on the agenda in the past).

+1. it's been a long road getting here. Thanks for your support!

> I assume this is a MSWord source?  Let's talk doc formatting…

Yes sir, it is indeed an MSWord source, with two active editors. :) Shall we take the formatting discussion offline?

> and saving it to disk on the router will not impact operations.

OK

> inextricable part of the process.

ok

> functions, anticipating that any interruptions SHOULD be characterized.

Hrmm. The difference between the two statements is that the suggested revised wording would not exclude interruption, and would require that it be captured; IMHO, an ISSU isn't an ISSU if there's a drop or break in control plane. Is there an amenable way to compromise here?

ok

It would seem that you're looking for us to denote specifically what protocols, whereas the section calls for what you have today; if you have an IPv4 only network today, then it'd be an IPv4 protocol being used. I think the spirit of the statement is very different; we want you to use what's representative of your network today, not protocol a, b and c that make vendor a, b and c look good on paper. Looking good on paper is great, but if it's not representative of your network, as an operator, what's the point?

ok

> http://tools.ietf.org/html/draft-ietf-bmwg-protection-meth-14#page-12

OK, we'll take a look.

> sounds pretty cool, I don't know how to measure them.

I agree. We'll work this out after next rev.

Al, thanks for the great feedback. Overall, I think the agreement was to post THIS (the current version of the doc, the one you've been commenting on) on March 11th, and then revise thereafter. Is this still desirable? Let me reword; It's unlikely I'll be able to meet with my coauthors and work in changes, given our current geographic and timezone challenges. :) Can we still proceed as aforementioned?

Thanks
Sarah

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Tue, 5 Mar 2013 08:53:23 -0500

Thanks for your reply, Sarah.

> functions, anticipating that any interruptions SHOULD be characterized.

 Hrmm. The difference between the two statements is that the suggested revised wording would not
 exclude interruption, and would require that it be captured; IMHO, an ISSU isn't an ISSU if 
 there's a drop or break in control plane. Is there an amenable way to compromise here?

ACM - sure, I think the procedure needs some objective way to judge
that interruptions are "minimal to none", but we can't try to place 
a performance requirement on the DUT in a benchmarking RFC, and that's
how I read the "MUST" in the 2nd para...

> It would seem that you're looking for us to denote specifically what
> protocols, whereas the section calls for what you have today; if you have
> an IPv4 only network today, then it'd be an IPv4 protocol being used. I
> think the spirit of the statement is very different; we want you to use
> what's representative of your network today, not protocol a, b and c that
> make vendor a, b and c look good on paper. Looking good on paper is great,
> but if it's not representative of your network, as an operator, what's the
> point?

ACM - I'm simply pointing out that the phrase 
"Mixed Protocol Environments" has a lot of latitude. 
Just give some examples, like you did in your reply...

> ...Overall, I think the agreement was to
> post THIS (the current version of the doc, the one you've been commenting
> on) on March 11th, and then revise thereafter. Is this still desirable?
> Let me reword; It's unlikely I'll be able to meet with my coauthors and
> work in changes, given our current geographic and timezone challenges. :)
> Can we still proceed as aforementioned?

ACM - yes, post this version on Monday.

thanks,
Al

> -----Original Message-----
> From: Sarah Banks [mailto:[email-masked]]
> Sent: Monday, March 04, 2013 8:14 PM
> To: MORTON JR., ALFRED (AL)
> Cc: [email-masked]
> Subject: Re: [bmwg] Software Update Doc
> 
> Hi Al,
> So let me point out - the document scopes ISSU, not ISSD; and a change to
> the next rev (after we post on March 11) to discuss WHAT makes an ISSU an
> ISSU; for example, it'll deal with questions like, "Can I upgrade just the
> Line Cards and not the Route Engines and still have it be an ISSU?" Rather
> than get into which vendors do what for downgrades, as I think there's a
> good amount of misunderstanding and confusion there, I thought we were
> clear on the scope. I am not opposed to ISSD; but I do have reservations.
> 
> Yes sir, it is indeed an MSWord source, with two active editors. :) Shall
> we take the formatting discussion offline?
> 
> >
> > 1. Introduction
> >
> Hrmm. The difference between the two statements is that the suggested
> revised wording would not exclude interruption, and would require that it
> be captured; IMHO, an ISSU isn't an ISSU if there's a drop or break in
> control plane. Is there an amenable way to compromise here?
> 
> > -=-=-=-=-=-
> > Editorial:
> > 4.1 Test Topology
> It would seem that you're looking for us to denote specifically what
> protocols, whereas the section calls for what you have today; if you have
> an IPv4 only network today, then it'd be an IPv4 protocol being used. I
> think the spirit of the statement is very different; we want you to use
> what's representative of your network today, not protocol a, b and c that
> make vendor a, b and c look good on paper. Looking good on paper is great,
> but if it's not representative of your network, as an operator, what's the
> point?
> 
> > -=-=-=-=-=-
> >
> Al, thanks for the great feedback. Overall, I think the agreement was to
> post THIS (the current version of the doc, the one you've been commenting
> on) on March 11th, and then revise thereafter. Is this still desirable?
> Let me reword; It's unlikely I'll be able to meet with my coauthors and
> work in changes, given our current geographic and timezone challenges. :)
> Can we still proceed as aforementioned?
> 
> Thanks
> Sarah
>