Subject: [bmwg] comments on draft-banks-bmwg-issu-meth-01

From: "Erum Frahim (efrahim)" <[email-masked]>
Date: Sat, 27 Jul 2013 21:24:41 +0000

Hi Team,

Here is some of the feedback and suggestions:

Missing the 5.1 in table of Content.

 5. ISSU Test Methodology............................................10

       5.1   Pre-ISSU recommended verifications ………………………………………………

1. Introduction

<efrahim> Can we define the patching and maintenance upgrade little bit more.

ISSU operation can be categorized into multiple scenario

- Whole Atomic version change: In this scenario, the whole system including all the process are upgraded as well in certain cases the firmware and bios can be upgraded.

- Patching: This usually refer to the one or two process upgrade. Patching usually helpful to fix a bug on the current version.

- Maintenance: If there are multiple fixes are coming for many modules.

<efrahim> Some Datacenter layer 3 switches are referred their Routing Processor as Supervisors (SUP). It would be nice to add both reference.

Different hardware configurations may be expected to be benchmarked,

      but a typical configuration for a forwarding device that supports

      ISSU consists of at least one pair of Routing Processors (RP's) or Supervisors

(SUP)

<efrahim>

Since most modern forwarding devices, where ISSU would be applicable,

      do consist of redundant RP's or Sups and hardware-separated control plane

      and data plane functionality, this document will focus on

      methodologies which would be directly applicable to those platforms.

<efrahim> Do we really want to restrict the ISSU to only redundant supervisors or RP. A lot of the new DC switches can easily do patching and ISSU without the redundant supervisor or RP. It would be good to include both ISSU testing methodology. The procedure if there is single SUP.

 3.3 Upgrade Run

<efrahim> May be change to include and clear the process better.  Here is the suggestion

 In this phase, the secondary RP load the new software first. Once the software is loaded on the secondary RP and comes to HA State, the secondary RP/SUP takes over, forcing the RP/SUP which was

      previously designated as primary, to adopt the standby role. At this

      point, the new primary RP drives the required updates to other

      specific components and forces warm-updates or re-initializations

      with the new software, as applicable. In addition, the now-standby

      RP will be updated with the desired software.

Depend of vendor implementation, after upgrading and downgrading the SUP/RP, the line cards (LC) and other modules gets upgraded either serially or in parallel.

5.3 Upgrade Run

<efrahim> There is no reference for any T3 Time where on the section 7 Final Report, there is a mention of T3 time. It would be nice to have some reference of T Time.

<efrahim> There was no mention of LC upgrades after the Supervisor. Upgrade processes do include both Supervisor and Lcs. May be log that as T4.

At this point, pay particular attention to any indications of

      control plane disruption, traffic impact or other anomalous

      behavior. Once the DUT has converged upon the new code and returned

      to normal operation note the completion time and log the duration of

      this step as T2.

Once both RP/SUPs are up on new code and synchronized, upgrade the LC and other modules. Log this duration of this time as T4.

7 Final Report

Add the T4 time for LC and other modules upgrade.

ISSU for all the LCs and other components T4

        Total ISSU Maintenance Window            T5 (sum of T1+T2+T3+T4)

Regards

Erum

---

From: Sarah Banks <[email-masked]>
Date: Sat, 27 Jul 2013 21:43:06 +0000

Hi Erum,
Thanks for the feedback! Please let me comment inline.

1. Introduction

We don't expand here, because we are not focusing on the patching or maintenance upgrades; that is, we don't distinguish between a major, or minor, or maintenance release; we take the approach that ISSU loads new code. We wanted to focus on how to measure that loading; not WHAT  we were loading, as the notion of what major, minor, maintenance and patching, and downgrades for that matter, seems to be polarizing. Having said this, though, there's nothing in the methodology that stops it from being applied in that manner.

We'll take this under consideration.

<efrahim>

Others on the list, what do you think? We picked a spot to start; that was with redundant RPs, because it's what the authors are familiar with and the customers we speak with are familiar with. If there's large consensus to bring this into the fold, we'll certainly consider it. I'd like to hear what others think - but it's a great point.

 3.3 Upgrade Run

We (authors) will discuss. I personally dislike part of this, as it assumes a behaviour that might not matter at the end; that the primary RP runs the ISSU or that the secondary does isn't what concerns me (personally), speaking for myself. But I understand your point, and we'll discuss.

5.3 Upgrade Run

OK, we'll discuss.

7 Final Report

Regards

Erum

Got it. Thanks!

Sarah

_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Sun, 28 Jul 2013 07:30:23 +0000

Erum, thanks a lot for taking the time to provide feedback on the doc !

My comments –/ toughs inline … (in addition to Sarah's reply)

From: "Erum Frahim (efrahim)" <[email-masked]<mailto:[email-masked]>>
Date: Saturday, July 27, 2013 5:24 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [bmwg] comments on draft-banks-bmwg-issu-meth-01

Hi Team,

>> ops – good catch  !

1. Introduction

>> We took ISSU as a 'holistic approach' the methodology described applies for a major software upgrade as well as a software – maintenance path , but point taken, we will add some additional text on this …

(SUP)

>> We can add a comments making a SUP module = as an RP

<efrahim>

>> We do not really want to restrict it, this is not a mandatory statement, at the time of writing this doc and even today an ISSU process with a single RP / SUP is 'disruptive' in nature (some of us in the industry)  do not consider it an 'IN SERVICE'  event, while in some  specific platforms this can be achieved (for a very limited type of deployments) this just applies for a very limited set of devices and deployments .>> We will discuss this internally and within the WG and let you know where we go with this ..

 3.3 Upgrade Run

>> I personally have some concerns here,  a basic one is is that the operator / tester does not necessarily needs to know that for example the STBY RP / SUP is upgraded first,   maybe even software is 'stage' on the specific LCS, this is why we are just describing a 'typical' process, but we do not want to make this a restrictive requirement as different vendors may deploy – come up with new ways to achieve this >>

5.3 Upgrade Run

>> We define T3 as the time for boTH SUP/RP s to me bcd up to NSR/HA synced state .. WE can add a reference to this time at the end of 5.3 …

      this step as T2.

>> By the end of T2 time all forwarding plane should be resumed, and reported by TP_frames , this considers the LC upgrade time (we may need to add a reference / comment here)  what some on the industry call  as 'dark window' I do not see the Lcs being upgrade after or taking more than T4, you lost me on this one .. :)

7 Final Report

Regards

Erum

>> Thanks a lot for all you comments!

---

From: "Lucien Avramov (lavramov)" <[email-masked]>
Date: Tue, 30 Jul 2013 05:07:29 -0700

Authors and BMWG'ers,

I find this draft really useful and support it. Thank you for putting 
this together.

Here are my comments:

section1: least one pair of Routing Processors (RP's), actually many 
switches in the DC are single RP. let's not restrict this to a minimum 
of a pair, and characterize a DUT providing ISSU feature as eligible for 
the test, regardless if there are at least 2 RPs.

Section 1) it should be stated when doing the test if the DUT is 
undergoing a patch or a full SW release upgrade, because this is 
different by essence in the implications for the DUT itself. a full SW 
release upgrade addresses a lot more capabilities than just patching a 
process.

Some DUT support ISSU when used in L2 mode only others when L3 is used, 
this should be specified along with patching or full upgrade when a user 
performs a test

Section 3.1) certain vendors verify the image and checksum as well as 
eligibility for iSSU prior to starting the process, if such exist, it 
should be used to confirm the success of step 3.1)

3.3) as indicated in section 1) some switches have a single sup but are 
hardware switching and have therefore a separated control plane from 
data plane. that is sufficient for those DUT to undergo 3.3) without a 
switchover of the RP as indicated here.

I would add a section 5.5 for ISSU testing when there is a topology 
change *during* the ISSU process for L2 and L3 traffic when supported to 
understand how the DUT would behave if such occurs during the upgrade 
process

Cheers,
Lucien

On 7/28/13 7:30 PM, Fernando Calabria (fcalabri) wrote:
> From: "Erum Frahim (efrahim)" <[email-masked] <mailto:[email-masked]>>
> To: "[email-masked] <mailto:[email-masked]>" <[email-masked]
> <mailto:[email-masked]>>
> - Whole Atomic version change: In this scenario, the whole system
> including all the process are upgraded as well in certain cases the
> firmware and bios can be upgraded.
>
> - Patching: This usually refer to the one or two process upgrade.
> Patching usually helpful to fix a bug on the current version.
>
>  >> We took ISSU as a 'holistic approach' the methodology described
> applies for a major software upgrade as well as a software – maintenance
> path , but point taken, we will add some additional text on this …
>
>
>
> <efrahim> Some Datacenter layer 3 switches are referred their Routing
> Processor as Supervisors (SUP). It would be nice to add both reference.
>
>
> <efrahim> Do we really want to restrict the ISSU to only redundant
> supervisors or RP. A lot of the new DC switches can easily do patching
> and ISSU without the redundant supervisor or RP. It would be good to
> include both ISSU testing methodology. The procedure if there is single
> SUP.
>
>
>  >> We do not really want to restrict it, this is not a mandatory
> statement, at the time of writing this doc and even today an ISSU
> process with a single RP / SUP is 'disruptive' in nature (some of us in
> the industry)  do not consider it an 'IN SERVICE'  event, while in some
>   specific platforms this can be achieved (for a very limited type of
> deployments) this just applies for a very limited set of devices and
> deployments .>> We will discuss this internally and within the WG and
> let you know where we go with this ..
>
>
>   3.3 Upgrade Run
>
>
>   In this phase, the secondary RP load the new software first. Once the
> software is loaded on the secondary RP and comes to HA State, the
> secondary RP/SUP takes over, forcing the RP/SUP which was
>
> Depend of vendor implementation, after upgrading and downgrading the
> SUP/RP, the line cards (LC) and other modules gets upgraded either
> serially or in parallel.
>
>
>  >> I personally have some concerns here,  a basic one is is that the
> operator / tester does not necessarily needs to know that for example
> the STBY RP / SUP is upgraded first,   maybe even software is 'stage' on
> the specific LCS, this is why we are just describing a 'typical'
> process, but we do not want to make this a restrictive requirement as
> different vendors may deploy – come up with new ways to achieve this >>
>
>
> 5.3 Upgrade Run
>
>
> <efrahim> There is no reference for any T3 Time where on the section 7
> Final Report, there is a mention of T3 time. It would be nice to have
> some reference of T Time.
>
>
>  >> We define T3 as the time for boTH SUP/RP s to me bcd up to NSR/HA
> synced state .. WE can add a reference to this time at the end of 5.3 …
>
>
> <efrahim> There was no mention of LC upgrades after the Supervisor.
> Upgrade processes do include both Supervisor and Lcs. May be log that as T4.
>
>
>  >> By the end of T2 time all forwarding plane should be resumed, and
> reported by TP_frames , this considers the LC upgrade time (we may need
> to add a reference / comment here)  what some on the industry call  as
> 'dark window' I do not see the Lcs being upgrade after or taking more
> than T4, you lost me on this one .. :)
>
>
> Once both RP/SUPs are up on new code and synchronized, upgrade the LC
> and other modules. Log this duration of this time as T4.
>
>
> 7 Final Report
>
>
> [email-masked]

---

From: "Fernando Calabria (fcalabri)" <[email-masked]>
Date: Tue, 30 Jul 2013 12:17:14 +0000

Thanks a lot for you comments / review

Please allows us to address some of them @ the WG meeting (in about ~ 1
hour) (detail e-mail will follow before the end of the week )

Rgds 

Fernando 

On 7/30/13 2:07 PM, "Lucien Avramov (lavramov)" <[email-masked]> wrote:

>> My comments / toughs inline Š (in addition to Sarah's reply)
>>
>>
>>         5.1   Pre-ISSU recommended verifications ŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠ
>>
>>
>>  >> ops  good catch  !
>>
>>
>>
>> 1. Introduction
>>
>> applies for a major software upgrade as well as a software  maintenance
>> path , but point taken, we will add some additional text on this Š
>>
>>
>>
>> different vendors may deploy  come up with new ways to achieve this >>
>>
>>
>> 5.3 Upgrade Run
>>
>>
>> synced state .. WE can add a reference to this time at the end of 5.3 Š
>>
>>

---

From: Sarah Banks <[email-masked]>
Date: Tue, 30 Jul 2013 12:21:50 +0000

Hi Lucien,
	Thanks for the feedback! All of the authors will discuss; however, some of what you pointed out Erum previously did (GMTA? :)). Your note about the topology change is close to my heart; let's see what comes out of the working group when we discuss today. 

Kind regards,
Sarah

On Jul 30, 2013, at 5:07 AM, Lucien Avramov (lavramov) <[email-masked]> wrote:

> Authors and BMWG'ers,
> 
> I find this draft really useful and support it. Thank you for putting this together.
> 
> Here are my comments:
> 
> section1: least one pair of Routing Processors (RP's), actually many switches in the DC are single RP. let's not restrict this to a minimum of a pair, and characterize a DUT providing ISSU feature as eligible for the test, regardless if there are at least 2 RPs.
> 
> Section 1) it should be stated when doing the test if the DUT is undergoing a patch or a full SW release upgrade, because this is different by essence in the implications for the DUT itself. a full SW release upgrade addresses a lot more capabilities than just patching a process.
> 
> Some DUT support ISSU when used in L2 mode only others when L3 is used, this should be specified along with patching or full upgrade when a user performs a test
> 
> Section 3.1) certain vendors verify the image and checksum as well as eligibility for iSSU prior to starting the process, if such exist, it should be used to confirm the success of step 3.1)
> 
> 3.3) as indicated in section 1) some switches have a single sup but are hardware switching and have therefore a separated control plane from data plane. that is sufficient for those DUT to undergo 3.3) without a switchover of the RP as indicated here.
> 
> I would add a section 5.5 for ISSU testing when there is a topology change *during* the ISSU process for L2 and L3 traffic when supported to understand how the DUT would behave if such occurs during the upgrade process
> 
> 
> Cheers,
> Lucien
> 
> 
>