Subject: [Isis-wg] Notes on draft-ietf-isis-restart-03.txt

From: Jeff Parker <[email-masked]>
Date: Thu, 1 May 2003 17:16:50 -0400

Some comments on draft-ietf-isis-restart-03.txt.

The biggest request is for a FSM describing the state transitions 
for the starting router and it's helpers.  The comments below are
for clarity: I don't think I understand the protocol well enough
to comment.  

Indented text is quote from draft or proposed alternative.
Outdented text is commentary.  

page 1
                  "This is necessary in order to invoke the protocol 
   mechanisms to ensure correct synchronization of the LSP database."

But you go on to demonstrate that it -isn't- necessary.

page 2
  "This draft additionally describes a mechanism to optimize database
   synchronization and minimize transient routing disruption when a
   router starts."

I'd suggest something to flag the difference between this and the
restarting case, as this is the first time this is encountered. 
Perhaps replace the last line of the three above with

  "router starts without wishing to reuse prior adjacencies."

  "Firstly, when a routing process restarts and an adjacency to a 
   neighboring router is reinitialized the neighboring routing process 
   does three things:"

The point of view changes from that of the restarting router to
the peer.  Something like the following keeps in focused in place
 
  "When a router's peer restarts and the peer adjacency 
   is reinitialized, the router does three things:"

This issue appears in other places.  I think it would be helpful 
to introduce terminology to describe three or four entities: 
	The restarting router
		Could be split into starting router and restarting router
	An aware and helpful peer
		I don't recall needing an aware and unwilling peer
	A peer who is not aware of the protocol

The names could be fanciful (Phoenix, Newborn, Samaratin, and Innocent)
or more prosaic: Restarter, Rebooter, Helper, and Bystander.

Page 3
    "1. It reinitializes the adjacency and causes its own LSP(s) to be 
        regenerated, thus triggering SPF runs throughout the area (or 
        in the case of Level 2, throughout the domain). 
 
     2. It sets SRMflags on its own LSP database on the adjacency 
        concerned. 

     3. In the case of a Point-to-Point link it transmits a (set of) 
        CSNP(s) over the adjacency. 

   In the case of a restarting router process, the first of these is 
   highly undesirable, but the second is essential in order to ensure 
   synchronization of the LSP database."

And the third?  If only for balance, you should say something about
the third.  

  "It is assumed that the three-way handshake [5] is being used on 
   Point-to-Point circuits."

Does this also belong in the overview, as an assumption?

  "An instance of T1 is maintained per interface, and indicates the 
   time after which an unacknowledged (re)start attempt will be 
   repeated."

It is not clear to me what an "unacknowledged restart attempt" is.
I assume this is sending out the hello packets with the special
TLV, but it could be read to include (re)setting the timers.

  "A single instance of T3 is maintained for the entire system. It 
   indicates the time after which the router will declare that it has 
   failed to achieve database synchronization (by setting the overload 
   bit in its own LSP). This is initialized to 65535 seconds, but is 
   set to the minimum of the remaining times of received IIHs 
   containing a restart TLV with RA set."

We can expect one of at least two policies from the Samaritans: a 
fixed time, and the remaining time left on the adjacency.  Any
suggestions on policies, or time required for a reboot?  Is this
a configuration parameter?  A MIB parameter?

page 4
    "Type   211 
     Length 3 
     Value (3 octets) 
        Flags (1 octet) 
               Bit 1 - Restart Request (RR) 
               Bit 2 - Restart Acknowledgment (RA) 
               Bit 3 - Suppress adjacency advertisement(SA) 
               Bits 4-8 - Reserved 
       Remaining Time (2 octets) 
               Remaining holding time (in seconds) 
               (note: only required when RA bit is set)"

Do we need to think about versioning?  We -can- always append more
fields.

  "b) immediately (i.e. without waiting for any currently running timer 
     interval to expire, but with a small random delay of a few 10s of 
     milliseconds on LANs to avoid "storms"), transmit over the 
     corresponding interface an IIH including the restart TLV with the 
     RR bit clear and the RA bit set, having updated the "Point-to-
     Point Adjacency State" option to reflect any new values received 
     from the (re)starting router."

Any new values?  I presume this only means the link ID.  System ID should
not change (if it does, the restart won't work) we should not restate our
Link ID, and this does not apply to the 3way state field.  Marking what
you are looking for explicit will clear up questions.

Page 5
  "Both restarting and starting routers will make use of the RR bit in 
   the restart TLV, though at different stages of the (re)start 
   procedure."

I found this a bit easier to read

  "Restarting and starting routers will make use of the RR bit in 
   the restart TLV, though each will use them at different stages 
   of the (re)start procedure."

page 6
  "On receipt of an IIH by the restarting router, a local adjacency is..." 

Consider 

  "When a restarting router receives an IIH, a local adjacency is..." 

  "T3 is set to the minimum of its current value and the value of the"

Consider 

  "The global timer T3 is set ..."

page 7
  "When BOTH a complete set of CSNP(s) (for each active level, in the 
   case of a pt-pt circuit) and an acknowledgement have been received 
   over the interface, the timer T1 is cancelled."

And then what happens?  We don't send IIH with RR bit set?  We start 
the SPF?  Perhaps a FSM for restart and Samaritan would help.

page 8
 
4.3.2      Adjacency acquisition during start 

   The starting router wants to ensure that in the event a neighboring 

Suggest you start the section with something to suggest the transition.

  "Next we turn to the case of a router starting, without assuming 
   any prior adjacencies."

- jeff parker
_______________________________________________
Isis-wg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/isis-wg

---

From: "DUBOIS Nicolas FTRD/DAC/ISS" <[email-masked]>
Date: Fri, 2 May 2003 16:43:02 +0200

At FT R&D we also strongly support a FSM description, it would greatly support our testing and interop efforts, as FSM for both restarting and helping routers are complex. We tried to figure out on diagrams what was and should be happening in different cases, but our efforts were not very satisfactory due to the multiple timers system. Trying to figure out what should be the best timer settings in the case of our network was also very difficult from a theoretical and modelling point of view.

Anyway in the lab it seems to be working in most cases.

Part of that,

>I don't think I understand the protocol well enough
>to comment.

Same for me :-)

Our current thought (not very advanced) on graceful restart are:

It could definetely help in case of maintenance operations if it were possible to activate and deactivate it in a simple manner. Official engineering rules recommend using the overload bit, using Graceful Restart could be an inprovement.

It is deployed on some routers that activate it by default ... but we do not know if it has proven useful or not. Equivalent question is : "do you prefer a simple or a complex fire detection system "  in a situation where unexpected events that are candidate for Graceful Restart should never happen in our networks and  knowing that we do not check wether a graceful restart did occur or not in case of such events.

When we'll have deployed a powerful enough metrology system in order to confirm whether an ISIS Graceful Restart was or wasn't successful in real network conditions, we'll tell you that we absolutely need to have it working as stated in the draft. 

Even though it is out of the scope of this working group ... one other thing we would support and find useful is a simple detection mechanism for lost hellos (or expected hellos that do not come in the expected timeframe) as this would help having finer granularity in the approximation of link reliability at the IS-IS level. Having this granularity would help us tell you wether we need graceful restart or not.

Best regards,

Nicolas Dubois
core packet network lab
France Telecom R&D

-----Message d'origine-----
De : Jeff Parker [mailto:[email-masked]]
Envoyé : jeudi 1 mai 2003 23:17
À : ISIS-WG (E-mail)
Objet : [Isis-wg] Notes on draft-ietf-isis-restart-03.txt

page 1

page 2

 

Page 3

page 4
    "Type   211 
     Length 3 
     Value (3 octets) 
        Flags (1 octet) 

Page 5

page 6

Consider 

Consider 

page 7

page 8

- jeff parker

---

From: Jeff Parker <[email-masked]>
Date: Fri, 2 May 2003 12:46:18 -0400

> An earlier version of the draft had an FSM. It was removed based on 
> feedback. (Hard to please everyone I guess.) We will consider 
> adding it back in.

Thanks.  

> How about:
>    "This invokes the protocol mechanisms which ensure correct 

Works for me 

> The terms "restart/restarting","start/starting", and 
> "(re)start/(re)starting" are very carefully defined so as to 
> try to avoid 
> confusion. (I like these better than "Phoenix", etc.) But the 
> definitions 
> occur AFTER the Abstract, which makes this first usage 
> ambiguous. I am 
> thinking we need to make the ABSTRACT much shorter and then 
> use much of the 
> current abstract as an introduction AFTER the terms have been defined.

There are two issues: the text in this case, which caught
me unprepared for the difference you were discussing
between start and restart, and the general case.  This
case can be handled by flagging the difference.

For the general case, I find starting and restarting very
close, and thus confusing.  I agree that the terms are
carefully defined.  

> How about:
>   "When an adjacency is reinitialised as a result of a 
> neighbor restating, the router does 3 things:".

I think I know whose point of view we have here, but I'm
still not certain.  Adding "router's" helps me.  

	"When a router's adjacency is reinitialised as a result of a 
 neighbor restating, the router does 3 things:".

> I was hoping the terms defined in the "Conventions" section 
> would be both clear and sufficient.

These are all suggestions: take those which help you.   

> How about rewording the next paragraph to read:
> 
> "The third action above minimizes the number of LSPs which must be 
> exchanged and, if made reliable, provides a means of 
> determining when the 
> LSP databases of the neighboring routers have been 
> synchronized. This is 
> desirable whether the router is being restarted or not. This document 
> describes modifications to achieve this."

Now I'm not sure what the last sentence refers to.  
 
> Ummm...errrr....it IS in the overview. Do you want it in the 
> Abstract?? Or 

Fine as is.  

> If we reversed the order of the sub-sections:
>     4.1 Timers
>     4.2 Restart TLV
> 
> so that the description of the use of the SA bit came before 
> the timers section would that clarify things for you?

I'm honestly confused about what the timers do: some of this is
the different roles in the starting and restarting cases, and 
some is not fully understanding the interactions.  I can try 
rereading in that order to see if it makes things clearer.  

> The only "option" that the Helper router has here is 
> described in 4.2.1:
> 
>    a) The state of the adjacency is not changed. It is an
>       implementation choice whether or not the holding time of the
>       adjacency is refreshed. Not refreshing the holding time 
> preserves
>       the intention of the original holding time. Refreshing it may
>       allow a longer grace period for the completion of the (re)start
>       process. Whichever option is chosen, the "remaining time"
>       transmitted according to (b) below MUST reflect the actual time
>       after which the adjacency will now expire.

Thanks

> There has been some discussion that the Helper router should 
> be required to 
> accept the hold time in an IIH with RR set so as to allow the 
> restarting 
> router to get the time that it thinks it needs to complete 
> restart. The 
> danger is that a router which continually restarts without 
> ever completing 
> the restart procedure could cause the Helper router to maintain the 
> adjacency to the continually restarting router past the point of 
> reasonableness. There are some simple strategies to avoid this.

I guess I would vote for simplicity right not.  

> >Do we need to think about versioning?  
> 
> [No]

> >Any new values?  I presume this only means the link ID.  
> 
> In text preceding item b) we say:
> 
> " ...if there exists on this interface an adjacency in state 
> "Up" with the 
> same System ID, and in the case
>     of a LAN circuit, with the same source LAN address"

but the text above is about p2p circuits, no?
 
> "Otherwise (i.e. if there was no adjacency in the "UP" state 
> to the system 
> ID in question), process the IIH as normal by reinitializing
> the adjacency, and setting the RA bit in the returned IIH."

My comment was simply that this was a discussion about how to
deal with a p2p TLV from a restarting router.  The original
text mentions "any new values", while I think that there is 
only one new value, the restarter's Link Id, that we would
actually take.

> In 4.1 we say:
> "A single instance of T3 is maintained for the entire system."
> 
> Not clear why we would need the term "global".

When I started writing technical docs, it was mathematics, 
and one rule I learned was "never start a sentence with a
variable name."  It seems to make things read better.  
Drop "global", and keep "The timer".  I would have used
a more descriptive term for T3 if I had it: global was
just a verbal twitch.  

> Is the sub section title not sufficient??
 
The first couple of times I read this, I thought the sections were
duplicated by mistake.  The prose suggested just moves the reader
into your mindset.  Again, this gets back to the similarity between
"start" and "restart"

- jeff parker

---

From: Les Ginsberg <[email-masked]>
Date: Fri, 02 May 2003 09:11:49 -0700

Jeff -

Thanks for your thoughtful review. Please see my responses inline.

     Les

An earlier version of the draft had an FSM. It was removed based on 
feedback. (Hard to please everyone I guess.) We will consider adding it 
back in.

How about:

The terms "restart/restarting","start/starting", and 
"(re)start/(re)starting" are very carefully defined so as to try to avoid 
confusion. (I like these better than "Phoenix", etc.) But the definitions 
occur AFTER the Abstract, which makes this first usage ambiguous. I am 
thinking we need to make the ABSTRACT much shorter and then use much of the 
current abstract as an introduction AFTER the terms have been defined.

How about:

  "When an adjacency is reinitialised as a result of a neighbor restating, 
the router does 3 things:".

>                 Could be split into starting router and restarting router

I was hoping the terms defined in the "Conventions" section would be both 
clear and sufficient.

>         A peer who is not aware of the protocol

Not sure there is enough justification for using names for these cases, but 
I'll review the text and see. "Helper" certainly is good. Not sure about 
the "unaware".

"The third action above minimizes the number of LSPs which must be 
exchanged and, if made reliable, provides a means of determining when the 
LSP databases of the neighboring routers have been synchronized. This is 
desirable whether the router is being restarted or not. This document 
describes modifications to achieve this."

>Does this also belong in the overview, as an assumption?

Ummm...errrr....it IS in the overview. Do you want it in the Abstract?? Or 
...???

so that the description of the use of the SA bit came before the timers 
section would that clarify things for you?

>a configuration parameter?  A MIB parameter?

The only "option" that the Helper router has here is described in 4.2.1:

      adjacency is refreshed. Not refreshing the holding time preserves

There has been some discussion that the Helper router should be required to 
accept the hold time in an IIH with RR set so as to allow the restarting 
router to get the time that it thinks it needs to complete restart. The 
danger is that a router which continually restarts without ever completing 
the restart procedure could cause the Helper router to maintain the 
adjacency to the continually restarting router past the point of 
reasonableness. There are some simple strategies to avoid this.

>Do we need to think about versioning?  We -can- always append more

Inserting versions into TLVs has been discussed in the past in other 
contexts and rejected. It also seems possible to add things to the TLV in a 
backwards compatible way. The three way TLV managed to do that - though it 
did cause some confusion for the uninformed.

" ...if there exists on this interface an adjacency in state "Up" with the 
same System ID, and in the case
    of a LAN circuit, with the same source LAN address"

so the matching System ID requirement has been specified.
We go on to say:

"Otherwise (i.e. if there was no adjacency in the "UP" state to the system 
ID in question), process the IIH as normal by reinitializing
the adjacency, and setting the RA bit in the returned IIH."

>    of the (re)start procedure."

OK.

>   "When a restarting router receives an IIH, a local adjacency is..."

OK

In 4.1 we say:

>the SPF?  Perhaps a FSM for restart and Samaritan would help.

No further action is taken as a result of T1 cancellation. It is T2/T3 
cancellation/expiration which is required for further action.

---

From: Les Ginsberg <[email-masked]>
Date: Fri, 02 May 2003 10:22:42 -0700

>close, and thus confusing.  I agree that the terms are

I understand that the terms "start" and "restart" look a lot alike. However 
they do accurately capture what the router is doing. And we have tried to 
use them very carefully and consistently throughout the document. This 
seems like the best way to make the text as clear as possible.

Although terms like "Phoenix" and "Newbie" are more colorful, my informal 
and unscientific survey says they are not very popular.  (Translation: 
Neither Mike nor I like them)

>  neighbor restating, the router does 3 things:".

I think we are close enough to declare agreement.

>Now I'm not sure what the last sentence refers to.

Second attempt:

"The third action above minimizes the number of LSPs which must be 
exchanged and, if made reliable, provides a means of determining when the 
LSP databases of the neighboring routers have been synchronized. This is 
desirable whether the router is being restarted or not.

This document describes modifications to prevent undesirable adjacency 
reinitialization and to reliably determine when the LSP databases of 
neighboring routers have been synchronized."

>but the text above is about p2p circuits, no?

No. It talks about both. Perhaps we need to say

  "in the case of Point-to-point adjacencies having updated the "Point-to-
      Point Adjacency State" option to reflect any new values received"

Seems a bit redundant, but perhaps clearer.

>only one new value, the restarter's Link Id, that we would

Well, you are correct, but with the current text we are "future-proofed" 
should the 3-way TLV ever be extended to include additional information.

>a more descriptive term for T3 if I had it: global was

Point taken. "The timer T3" is better.

---

From: Les Ginsberg <[email-masked]>
Date: Fri, 02 May 2003 10:30:14 -0700

At 04:43 PM 5/2/2003 +0200, DUBOIS Nicolas FTRD/DAC/ISS wrote:
>Even though it is out of the scope of this working group ... one other 
>thing we would support and find useful is a simple detection mechanism for 
>lost hellos (or expected hellos that do not come in the expected 
>timeframe) as this would help having finer granularity in the 
>approximation of link reliability at the IS-IS level. Having this 
>granularity would help us tell you wether we need graceful restart or not.
>
>Best regards,
>
>Nicolas Dubois
>core packet network lab
>France Telecom R&D

Nicolas -

Could you explain what you think the relationship between link reliability 
and graceful restart is? I am having trouble making the connection.

Thanx.

    Les

---

From: "DUBOIS Nicolas FTRD/DAC/ISS" <[email-masked]>
Date: Sat, 3 May 2003 09:13:16 +0200

Lee,

Maybe it is a misunderstanding from our side :

To summarize my understanding was that Graceful restart would help in
case there is a problem at the ISIS level (control plane in our
terminology) , but not at the below level (forwarding plane in our
terminology that is interface and sonet level).

So in our current network (with almost no graceful restart implemented)
we tried to gather statistics that would match with following conditions
(that should theoretically not exist):
"ISIS problem + Sonet OK + Interface OK"

Results are:
This occurs quite often:
	In case of minor congestion on a link, it happens that once upon
a while an adjacency fails, whereas the link stays up from the other
indicators point of view ( no Sonet problems / no interface crash
problem) (in fact it behaves as if it were dropping aleatory hellos, and
at some point 3 consecutive hellos are dropped).
	In case of severe congestion on a link, it always happens that
the adjacency flaps a lot, whereas sonet and interface stay up (at least
in the log we have, maybe we are missing some logs)

Our understanding was that the two above phenomenon would interfere with
graceful restart, and that instead of having a simple usual congestion
panic (on the 2000 ISIS links we have, it happens once a week on
average), we would have a graceful congestion panic which is not
particularly desirable.We do not think graceful restart is the adequate
answer to this problem and we would prefer to address this problem in
another way.

Can you tell me whether graceful restart would trigger in the above
described flapping situations ?

I must admit I didn't read the latest version of the draft and that the
latest tests I have done on the subject are at least 6 months old, so I
might be confused.  

Best regards, 

Nicolas

-----Message d'origine-----
De : Les Ginsberg [mailto:[email-masked]]
Envoye : vendredi 2 mai 2003 19:30
A : DUBOIS Nicolas FTRD/DAC/ISS
Cc : Jeff Parker; ISIS-WG (E-mail)
Objet : RE: [Isis-wg] Notes on draft-ietf-isis-restart-03.txt

Nicolas -

Could you explain what you think the relationship between link
reliability 
and graceful restart is? I am having trouble making the connection.

Thanx.

    Les

---

From: Les Ginsberg <[email-masked]>
Date: Sat, 03 May 2003 14:08:25 -0700

Nicolas -

Graceful restart is not applicable to your situation. Graceful restart is 
targeted at two situations:

1)Where the control plane restarts and the forwarding plane is maintained. 
In this case (referred to as "restarting" in the draft) the protocol 
enhancements described in the draft try to eliminate unnecessary adjacency 
flapping and the associated temporary network instability.
An example case where this might be used is when a software upgrade is 
performed on the control plane software in a router in which it is possible 
to maintain the forwarding plane state across the software upgrade.

2)Where the control plane restarts and the forwarding plane is not 
maintained. In this case (referred to as "starting" in the draft) the 
protocol enhancements described in the draft try to eliminate temporary 
blackholes caused by the presence in the network of stale LSPs issued by 
the router which has just started.
An example case where this might be used is an ordinary reboot of a router.

In both cases the protocol extensions allow the router which has 
restarted/started to reliably determine when its LSP database is synced 
with its neighbors.

In the case you discuss below, the control plane operational state never 
changes - it is always up - but for other reasons adjacencies are flapping. 
The restart draft is not applicable to this case.

    Les