Subject: major changes: Protocol overview section

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 13 Sep 2005 14:44:24 -0400

This section went through some major changes to reflect discussion in
last IETF meeting. It still requires some work.
For this reason i am attaching both the text and the diff from last
revision.
This is requred reading for tommorow concal. Please review.

cheers,
jamal
@@ -622,10 +622,10 @@
    tearing down such associations.  An FE uses the PL layer to throw
    various subscribed-to events to the CE PL layer as well as respond to
    various status requests issued from the CE PL.  The CE configures
-   both the FE and associated LFBs attributes using the PL layer.  In
-   addition the CE may send various requests to the FE to activate or
-   deactivate it, reconfigure its HA parametrization, subscribe to
-   specific events etc.  More details in Section 6.
+   both the FE and associated LFB'ss operational parameters using the PL
+   layer.  In addition the CE may send various requests to the FE to
+   activate or deactivate it, reconfigure its HA parametrization,
+   subscribe to specific events etc.  More details in Section 6.
 
 3.1.2.  The TML layer
 
@@ -648,11 +648,10 @@
    ForCES protocol.  The best way to think of them are as
    configurations/parameterizations for the PL and TML before they
    become active (or even at runtime based on implementation).  In the
-   simplest case, the FE or CE read a static configuration file which
-   they use as the FEM/CEM interface.  RFC 3746 has a lot more detailed
-   descriptions on how the FEM and CEM could be used.  We discuss the
-   pre-association phase where the CEM and FEM play briefly in section
-   Section 3.2.1.
+   simplest case, the FE or CE read a static configuration file.  RFC
+   3746 has a lot more detailed descriptions on how the FEM and CEM
+   could be used.  We discuss the pre-association phase where the CEM
+   and FEM play briefly in section Section 3.2.1.
 
    An example of typical things FEM/CEM would configure would be TML
    specific parameterizations such as:
@@ -664,6 +663,7 @@
 
    c.  Security parameterization such as keys etc.
 
+   d.  Connection association parameters

@@ -673,8 +673,6 @@
 Internet-Draft                   ForCES                      August 2005

-   d.  Connection association parameters
-
    Example "send up to 3 association messages each 1 second apart" Vs "
    send up to 4 association messages with increasing exponential
    timeout".
@@ -684,7 +682,7 @@
    ForCES, in relation to NEs, involves two phases: the Pre-Association
    phase where configuration/initialization/bootup of the TML and PL
    layer happens, and the association phase where the ForCES protocol
-   operates.
+   operates to parametrize FEs.
 
 3.2.1.  Pre-association
 
@@ -697,7 +695,8 @@
    considered static pre-association.  Dynamic configuration may also
    happen using the Fc, Ff and Fl reference points.  Vendors may use
    their own proprietary service discovery protocol to pass the
-   parameters.
+   parameters.  Essentially only guidelines are provided here and the
+   details are left to the implementation.
 
    The following are scenarios reproduced from the Framework Document to
    show a pre-association example.
@@ -724,6 +723,7 @@

+
 Doria (Ed.), et al.     Expires February 2, 2006               [Page 13]
 
 Internet-Draft                   ForCES                      August 2005
@@ -750,18 +750,18 @@
 
    Before the transition to the association phase, the FEM will have
    established contact with the appropriate CEM component.
-   Initialization of the ForCES interface will be completed, and
-   authentication as well as capability discovery may be complete as
-   well.  Both the FE and CE would have the necessary information for
-   connecting to each other for configuration, accounting,
-   identification and authentication purposes.  Both sides also would
-   have all the necessary protocol parameters such as timers, etc.  The
-   Fl reference point may continue to operate during the association
-   phase and may be used to force a disassociation of an FE or CE.
-   Because the pre-association phase is out of scope, these details are
-   not discussed any further in this specification.  The reader is
-   referred to the framework document [RFC3746] for more detailed
-   discussion.
+   Initialization of the ForCES interface will have completed, and
+   authentication as well as capability discovery may be complete.  Both
+   the FE and CE would have the necessary information for connecting to
+   each other for configuration, accounting, identification and
+   authentication purposes.  To summarize, at the completion of this
+   stage both sides have all the necessary protocol parameters such as
+   timers, etc.  The Fl reference point may continue to operate during
+   the association phase and may be used to force a disassociation of an
+   FE or CE.  Because the pre-association phase is out of scope, these
+   details are not discussed any further in this specification.  The
+   reader is referred to the framework document [RFC3746] for a slightly
+   more detailed discussion.
 
 3.2.2.  Post-association
 
@@ -769,11 +769,11 @@
    using the ForCES protocol (PL over TML) as defined in this document.
    There are three sub-phases:
 
-   o  Association setup state
+   o  Association Setup state
 
    o  Established State
 
-   o  Association teardown state.
+   o  Association Lost state.

@@ -785,7 +785,7 @@
 Internet-Draft                   ForCES                      August 2005

-3.2.2.1.  Association setup state
+3.2.2.1.  Association Setup state
 
    The FE attempts to join the NE.  The FE may be rejected or accepted.
    Once granted access into the NE, capabilities exchange happens with
@@ -803,16 +803,44 @@
 
    In this state the FE is continuously updated or queried.  The FE may
    also send asynchronous event notifications to the CE or synchronous
-   heartbeat notifications.  This continues until a termination is
-   initiated by either the CE or the FE.
+   heartbeat notifications if programmed to do so.  This continues until
+   a termination occurs because of loss of connectivity or is initiated
+   by either the CE or the FE.
 
    Refer to section on protocol scenarios Section 7 for more details.
 
+3.2.2.2.1.  Association Lost state
+
+   In this state, both or either the CE or FE declare the other side is
+   no longer associated.  The disconnection could be physically
+   initiated by either party for administrative purposes but may also be
+   driven by operation reasons such as loss of connectivity.
+
+   When an association is lost between a CE and FE, the FE continues to
+   operate as instructed by the CE via the CE failover policy (for
+   further discussion refer to in XXXX).
+
+   Editorial note: We need text on XXXX - issue 36 Editorial note: We
+   need A state machine to show transitions
+
+   For this revision of the protocol (as defined in this document), the
+   FE, upon re-association, MUST declare all the state it has as invalid
+   and retrieve new state.  This approach is motivated by a desire for
+   simplicity (as opposed to efficiency).
+
 3.3.  Protocol Mechanisms
 
    Various semantics are exposed to the protocol users via the PL header
    including: Transaction capabilities, atomicity of transactions, two
    phase commits, batching/parallelization, High Availability and
+
+
+
+Doria (Ed.), et al.     Expires February 2, 2006               [Page 15]
+
+Internet-Draft                   ForCES                      August 2005
+
+
    failover as well as command windows.
 
 3.3.1.  Transactions, Atomicity, Execution and Responses
@@ -885,60 +913,66 @@
    vendor where the FE-CE pair is tightly coupled.  In such a case, the
    transactional operations maybe simplified further by extra
    computation at the CE.  We do not discuss this view further other
-   than to mention it in not dissallowed.  For the purpose of
+   than to mention it is not dissallowed.  For the purpose of
    interopability, we define a classical transactional protocol known as
    two phase commit which meets the ACID properties to be used for
    transactions.
 
-
-
-Doria (Ed.), et al.     Expires February 2, 2006               [Page 16]
-
-Internet-Draft                   ForCES                      August 2005
-
-
 3.3.1.1.3.  Transaction protocol
 
-   A 2PC starts with a START | ATOMIC flag on its first message of a
-   transaction.  A transaction may span multiple messages.  It is up to
-   the CE to keep track of the different seq #s making up a transaction.
-   This may then be followed by more messages which are part of the same
-   atomic transaction.
+   A 2PC starts with the header flags containg a SOT(start of
+   transaction) and AT (Atomic Transaction) on its first message
+   configuration message.  A transaction may span multiple messages.  It
+   is up to the CE to keep track of the different outstanding messages
+   making up a transaction.  This may then be followed by more messages
+   which are part of the same atomic transaction.
 
    Any failure notified by the FE causes the CE to execute an ABORT to
    all FEs involved in the transaction, rolling back all previously
    executed operations in the transaction.
 
+   [Editorial Note: We need to discuss how to issue the ABORT - flags vs
+   type]
+
    The transaction commitment phase is signalled by an empty DONE msg
    type.
 
+   Editorial note: Ok to rename DONE to be EOT (end of transaction)?
+
+   [Editorial Note: We need to discuss how to issue the DONE - flags vs
+   type]
+
+   The FE MUST respond to the CE's DONE message.  If no response is
+   received from the CE within a specified timeout, the transaction MUST
+   be aborted by the CE.
+
+
+
+Doria (Ed.), et al.     Expires February 2, 2006               [Page 17]
+
+Internet-Draft                   ForCES                      August 2005
+
+
 3.3.1.1.4.  Recovery
 
    Any of the participating FEs, or the CE, or the associations between
-   them, may fail after the DONE message has left the CE and before it
-   has received all the responses, (possibly the DONE never reached the
-   FEs).  At this point it is known that none of the operations failed
-   but it is presumed that the data has not yet been made durable by the
-   FEs.  The means of detecting such failures may include loss of
-   heartbeat (within the scope of ForCES) or mechanisms outside the
-   scope of ForCES.  When the associations are re-established, the CE
-   will discover a transaction in an intermediate state.  Some FEs will
-   have made the data durable and closed the transaction; others may
-   have failed while doing so, and may, or may not, still have that
-   data.  At this point the transaction enters the recovery phase.
+   them, may fail after the DONE response message has left the FE and
+   before it has received all the responses, (possibly the DONE response
+   never reached the CE).
 
-   The CE re-issues an empty DONE message to all FEs involved in the
-   transaction.  Those that completed the transaction confirm this to
-   the CE.  Those that did not, commit the data and confirm this to the
-   CE.  An FE that has lost all records of the transaction MUST reply
-   with status UNKNOWN and the actions subsequently taken by the CE are
-   implementation dependent.
+   In this protocol revision, for sake of simplicity as indicated in
+   Section 3.2.2.2.1, an FE losing an association would be required to
+   get new state from the newly associated CE from scratch upon a
+   reassociation.  The decision on what an FE should do after a lost
+   association is dicated by the CE Failover policy (refer to XXXX).
+
+   Editorial note: section XXXX needed.
 
 3.3.1.1.5.  continue-execute-on-failure
 
    In which several independent operations are targeted at one or more
    LFB selectors.  Execution continues at the FE when one or more
-   operations fail.  This mode is signalled by a missing ATOMIC flag.
+   operations fail.  This mode is signalled by a missing AT flag.
 
 3.3.1.1.6.  execute-until-falure
 
@@ -960,28 +987,37 @@
    Multipart flags apply.  I.e all messages in a transaction except for
    the last have a MULTIPART flag on.
 
-   There has to be consistency across the multi parts of the messages.
-   In other words the first message starting with mode #1 above, implies
-   the rest do.  Any inconsitency implies a cancelled transaction in
-   which all messages are dropped and the sender NACKED.
+   Multi-part messages MUST be consistent in their use of execution
+   modes.  If the first message starts with one mode (such as continue-
+   execute-on-failure mode), then it implies the rest that are part of
+   the same multi-part messages do.  Any inconsitency implies an error
+   and a cancelled transaction in which all messages are dropped and the
+   sender NACKED.
 
-3.3.2.  FE, CE, and FE protocol LFBs
+3.3.2.  FE, and FE protocol LFBs
 
-   All PL messages operate on LFB structures as this provides more
+   All PL messages operate on LFB constructs as this provides more
    flexibility for future enhancements.  This means that maintenance and
    configurability of FEs, NE, as well as the ForCES protocol itself
    must be expressed in terms of this LFB architecture.  For this reason
    special LFBs are created to accomodate this need.
    In addition, this shows how the ForCES protocol itself can be
    controlled by the very same type of structures (LFBs) it uses to
    control functions such as IP forwarding, filtering, etc.
 
-   To achieve this, the following LFBs are used:
+   To achieve this, the following speacilized LFBs are introduced:
 
    o  FE Protocol LFB
 
-   o  FE LFB
+   o  FE Object LFB
 
    These LFBs are detailed in Section 6.2.  A short description is
    provided here:
@@ -1001,16 +1037,12 @@
       between the CE and the FE.  Such attributes can be FEState (refer
       to model draft), vendor, etc.  The FE LFB contains in particular a
       table that maps a virtual LFB Instance ID to one or more Instance
       IDs of LFBs in the FE.
 
+   o  [Editorial note: the term "FE attributes" is more technical than
+      implied above.  We now use it to define a subsection of the LFB
+      elements, discuss].
+
 3.3.3.  Scaling by Concurrency
 
    It is desirable that the PL layer not become the bottleneck when
@@ -1037,8 +1077,8 @@
 
    o  multiple PL PDUs can be aggregated under one TML message
 
-   o  multiple LFB classes and instances can be addressed within one PL
-      PDU
+   o  multiple LFB classes and instances (as indicated in the LFB
+      selector) can be addressed within one PL PDU
 
    o  Multiple operations can be addressed to a single LFB class and
       instance

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 13 Sep 2005 21:42:35 -0400

The change at 682 (intro to section 3.2) result in  "the ForCES protocol 
operates to parameterize FEs."  This claims that the association phase 
parameterizes the FE.  Do you mean "the ForCES protocol operates to 
manipulate the parameters of the FEs?"

Yours,
Joel

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 14 Sep 2005 11:12:18 +0800

Jamal,

That's a very good change. 

Some comments are as below.

3.3.2.  FE, and FE protocol LFBs

   All PL messages operate on LFB constructs as this provides more
   flexibility for future enhancements.  This means that maintenance and
[Weiming] I'm not sure if  it is proper to say 'All PL message' here, because for some messages like HB, association teardown messages, although we may say they are also addressed to LFBs,  at least, from the message format point of view, they are not.  

   To achieve this, the following speacilized LFBs are introduced:

   o  FE Protocol LFB

   o  FE Object LFB

   These LFBs are detailed in Section 6.2.  
[Weiming] I think it may look better for  the overall draft if we stop this section here, or put the following  text in Section 6.2 if necessary. 

   A short description is  provided here:

   o  The FE Protocol LFB is a logical entity in each FE that is used to
      control the ForCES protocol.  The CE operates on this LFB to
      subscribe or unsubscribe to Heartbeat messages, define the
      Heartbeat interval, or to discover which ForCES protocol version
      is supported and which TMLs the FE supports.  The FE Protocol LFB
      also contains the various ForCES ID to be used: unicast IDs a
      table of the PL multicast IDs the FE must be listening to.

   o  The FE LFB (referred to as "FE attributes" in the model draft)
      should not be confused with the FE Protocol Object.  The FE LFB is
      a logical entity in each FE and contains attributes relative to
      the FE itself, and not to the operation of the ForCES protocol
      between the CE and the FE.  Such attributes can be FEState (refer
      to model draft), vendor, etc.  The FE LFB contains in particular a
      table that maps a virtual LFB Instance ID to one or more Instance
      IDs of LFBs in the FE.
[Weiming] the 'FE LFB' name is not a consistent name.

   o  [Editorial note: the term "FE attributes" is more technical than
      implied above.  We now use it to define a subsection of the LFB
      elements, discuss].
[Weiming] Again, I highly suggest to put above paragraph in Section 6.2 to ease the consistency management.

Thanks,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 14 Sep 2005 11:19:49 +0800

3.3.1.1.4.  Recovery

   Any of the participating FEs, or the CE, or the associations between
   them, may fail after the DONE response message has left the FE and
   before it has received all the responses, (possibly the DONE response
   never reached the CE).

   In this protocol revision, for sake of simplicity as indicated in
   Section 3.2.2.2.1, an FE losing an association would be required to
   get new state from the newly associated CE from scratch upon a
   reassociation.  The decision on what an FE should do after a lost
   association is dicated by the CE Failover policy (refer to XXXX).

   Editorial note: section XXXX needed.
[Weiming] Please just refer to Section 6.2.1. I don't think we need to add a more specifica section for this.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 14 Sep 2005 06:49:27 -0400

> manipulate the parameters of the FEs?"

yes - thanks. Fixed.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 14 Sep 2005 07:38:10 -0400

On Wed, 2005-14-09 at 11:12 +0800, Wang,Weiming wrote:
> [Weiming] I'm not sure if  it is proper to say 'All PL message' here, 
> because for some messages like HB, association teardown messages, although 
> we may say they are also addressed to LFBs,  at least, from the message 
> format point of view, they are not.  
> 

The HB is the closest it comes to some message that has no LFB, but in
my opinion it does. Even though we decided to have a shortcut for the HB
by not having a LF selector, there is infact an LFB - the FEPO to be
precise - that is used to configure HB parameters.
I am thinking hard and i cant think of any other message which doesnt
have an LFB associated with it.
But if you could mention one, I will go back and make changes.

> [Weiming] I think it may look better for  the overall draft if we stop this 
> section here, or put the following  text in Section 6.2 if necessary. 
> 

Did you mean not to give the short description below for the two LFBs
when you say to "to stop this section here"?

> [Weiming] the 'FE LFB' name is not a consistent name.

Good catch - infact i think some of the description was valid a while
back, no longer true since we have formalized these two LFBs. 
To address (if i understood you) your earlier point, I have changed the
text to:

-------
3.3.2.  FE Object and FE protocol LFBs

   o  FE Protocol LFB which is used to control the ForCES protocol.

   o  FE Object LFB which is used to controls attributes relative to the
      FE itself.  Such attributes include FEState (refer to model
      draft), vendor, etc.

----

> ease the consistency management.

I have removed that text since it was intended to highlight that text to
point to a draft-global issue. This issue is now captured in issue #58.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 14 Sep 2005 07:58:31 -0400

> add a more specifica section for this.

Thanks.

the new text now reads:

---
3.3.1.1.4.  Recovery

   association is dicated by the CE Failover policy (refer to Section 8
   and Section 6.2).
----------

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Thu, 15 Sep 2005 00:11:24 +0800

----- Original Message ----- 
> But if you could mention one, I will go back and make changes.
I agree your description above, I'm just a little worry that because we currently do not use LFB selectors for all messages, therefore, it may make readers a little confused when reading the followed message definitions. May I suggest to add some words to mention that "All PL messages operate on LFB constructs..., (even though some PL messages may omit the LFB selection in the message PDU because the LFB is unique for the message."  

> when you say to "to stop this section here"?
Sure.
>    These LFBs are detailed in Section 6.2.
That looks nice to me. Thanks.
> point to a draft-global issue. This issue is now captured in issue #58.

thanks,
Weiming