Subject: Re: [issue76] Comments from Alex

From: Weiming Wang <[email-masked]>
Date: Sun, 5 Feb 2006 22:53:24 +0800

Avri,

Because some of the replies to the issue were missing on the issue tracker, to
ease your work, I summarize the related text update as below:

Place #1:

<Original text>

4.2.  ForCES Protocol Phases

   ForCES, in relation to NEs, involves two phases: the Pre-Association
   phase where configuration/initialization/bootup of the TML and PL
   layer happens, and the association phase where the ForCES protocol
   operates to manipulate the parameters of the FEs.

         start                  FEObject Admin up
         -------+    +--->---->---->---->------->----+
                |    |                               Y
                Y    |                               |
                |    |                               Y
              +------+--+                        +--------+
              |         |                        |        |
              | DOWN    |                        | UP     |
              | State   |                        | State  |
              |         |                        |        |
              +---------+                        +--------+
                   ^                                  Y
                   |                                  |
                   +-<---<------<-----<------<----<---+
                            FEObject Admin Down/
                            Association lost

   Figure 4: The FE State Machine

   The FE can only be in one of two states as indicated above.  When the
   FE is in the DOWN state, it is not forwarding packets.  When the FE
   is in the UP state it may be forwarding packets depending on the
   configuration of its specific LFBs.

   On start up the FE is in the DOWN state unless explicitly configured
   by the CE to transition to the UP state.  This MUST be done before
   configuring any other LFBs that affect packet forwarding.

   The FE transitions from the DOWN to the UP state when explicitly
   configured to do so by the CE or when it loses its association with
   the CE.  For the FE to properly complete the transition to the DOWN
   state it MUST stop packet forwarding and that this may affect
    multiple LFBs.  How this is achieved is outside the scope of this
   specification.
</original text>

<changed to>
4.2.  ForCES Protocol Phases
   ForCES, in relation to NEs, involves two phases: the Pre-Association
   phase where configuration/initialization/bootup of the TML and PL
   layer happens, and the Post-association phase where the ForCES protocol
   operates to manipulate the parameters of the FEs.

        FE start            CE configures
         -------+    +--->---->---->---->------->----+
                |    |                               Y
                Y    |                               |
                |    |                               Y
              +------+--+                        +--------+
              | FE      |                        | FE     |
              | DOWN    |                        | UP     |
              | State   |                        | State  |
              |         |                        |        |
              +---------+                        +--------+
                   ^                                  Y
                   |                                  |
                   +-<---<------<-----<------<----<---+
                    CE configures or FE loses association

  Figure 4: The FE State Machine

The FE can only be in one of two states as indicated above.  When
the FE is in the DOWN state, it is not forwarding packets.  When the FE is in
the UP state it may be forwarding packets depending on the configuration of its
specific LFBs.

CE configures FE states transitions by means of a so-called FEObject LFB,
which is defined in [FE-MODEL] and also explained in Section 4.3.3 of this
document.
In FEObject LFB, FE state is defined as an attribute of the LFB, and CE
configuration of the FE state
equals CE configuration of this attribute. Note that even in the FE DOWN state,
the FEObject LFB itself is active.

On start up the FE is in the DOWN state unless explicitly configured by the
CE to transition to the UP state.  This must be done before configuring any
other LFBs that affect packet forwarding.

The FE transitions from the UP to the DOWN state when explicitly configured to
do so by the CE or when
FE loses its association with CE. Note that in the association lost case, FE can
also possibly not go to DOWN state. Whether or not FE goes to DOWN state in this
case depends on the policy CE sets for the FE to do when such failover happens.
See  the 'CE failover policy' part in Section 7.2.1 for more details on this.

For the FE to properly complete the transition to the DOWN state it must
stop packet forwarding and that this may affect multiple LFBs.  How this is
achieved is outside the scope of this specification.
</changed to>

Place #2:  Section 4.3 (the whole section) update as below:

4.3.  Protocol Mechanisms

   Various semantics are exposed to the protocol users via the PL header
   including: Transaction capabilities, atomicity of transactions, two
   phase commits, batching/parallelization, High Availability and
   failover as well as command windows. EM (Execute Mode) flag, AT (Atomic
Transaction) flag, and TP (Transaction Phase) flag defined in Common Header
Section (Section 6.1) are highly related to these mechanisms.

4.3.1.  Execution, Transactions, Atomicity, and Responses

   In the master-slave relationship the CE instructs one or more FEs on
   how to execute operations and how to report back the results.

   This section details the different modes of execution that a CE can
   order the FE(s) to perform as defined in Section 4.3.1.1.  It also
   describes the different modes a CE can ask the FE(s) to use for
   formatting the responses after processing the operations as
   requested, which relates to transactional two phase commitment operations.

4.3.1.1.  Execution

   There are 3 execution modes that could be requested for a batch of
   operations spanning one or more LFB selectors in one protocol
   message. The EM flag defined in Common Header Section (Section 6.1) selects
the execution mode for a protocol message, as below:

   a.  execute-all-or-none

   b.  execute-until-failure

   c.  continue-execute-on-failure

4.3.1.1.1.  execute-all-or-none

   When set to this mode, indipendent operations in a message targeted at one or
more LFB selectors will all be executed if no failure happens for the
operations, or none of the operations will be executed if there is any failure
for the operations.

4.3.1.1.2.  continue-execute-on-failure

   If several independent operations are targeted at one or more LFB
   selectors, execution continues at the FE even if one or more
   operations fail.

4.3.1.1.3.  execute-until-failure

   In this mode all operations are executed on the FE sequentially until
   the first failure.  The rest of the operations are not executed but
   operations already completed are not undone, i.e. there is no roll
   back in this mode.

4.3.1.2. Transaction and Atomicity

4.3.1.2.1.  Transaction Definition

   A transaction is defined as a collection of one or more ForCES
   operations within one or more PL messages that MUST meet the ACIDity
   properties[ACID], defined as:

   o  *Atomicity*.  In a transaction involving two or more discrete
      pieces of information, either all of the pieces are committed or
      none are.

   o  *Consistency*.  A transaction either creates a new and valid state
      of data, or, if any failure occurs, returns all data to its state
      before the transaction was started.

   o  *Isolation*.  A transaction in process and not yet committed must
      remain isolated from any other transaction.

   o  *Durability*.  Committed data is saved by the system such that,
      even in the event of a failure and system restart, the data is
      available in its correct state.

   There are cases where the CE knows exact memory and implementation
   details of the FE such as in the case of an FE-CE pair from the same
   vendor where the FE-CE pair is tightly coupled.  In such a case, the
   transactional operations may be simplified further by extra
   computation at the CE.  This view is not discussed further other than
   to mention that it is not disallowed.

   Defined as above, a transaction is always atomic and may be:

   a.  Within an FE alone
       Example: updating multiple tables which are dependent on each
       other.  If updating one fails, then any others already updated
       must be undone.

   b.  Distributed across the NE
       Example: updating table(s) that are inter-dependent across
       several FEs (such as L3 forwarding related tables).

 4.3.1.2.2. Transaction protocol

   By use of the execute mode as defined in Section 4.3.1.1, the protocol has
provided a mechanism for transactional operations within one stand-alone
message. The 'execute-all-or-none' mode can
meet the ACID requirements.

   For transactional operations of multipul messages within one FE or across
FEs, a classical transactional protocol known as Two Phase Commit (2PC) [2PCREF]
is supported by the protocol to achieve the transactional operations.

   AT flag and TP flag in Common Header (Section 6.1) are provided for 2PC based
transactional operations spaaning multipul messages.

   The AT flag, when set, indicates this message belongs to an Atomic
Transaction. All messages for a transaction operation must have the AT flag set.
If not set, it means the message is just stand-alone message and does not
participate in any transaction operation that spans mulitpul messages.

   The TP flag indicates the Transaction Phase this message belongs to. There
are possibly 4 phases for an transactional operation as: `SOT (Start of
Transaction)` , `MOT (Middle of Transaction)`, `EOT (End of Transaction)`, and
`ABT (Abort)`.

   A transaction operation is started with a message the TP flag is set to Start
of Transaction (SOT). Multipart messages, after the first one, are indicated by
the Middle of Transaction flag (MOT).  The last message is indicated by by EOT.

   Any failure notified by the FE causes the CE to execute an Abort
   Transaction (ABT) to all FEs involved in the transaction, rolling
   back all previously executed operations in the transaction.

   The transaction commitment phase is signaled from the CE to the FE by
   an End of Transaction (EOT) configuration message.
   The FE MUST respond to the CE's EOT message.  If no response is
   received from the FE within a specified timeout, the transaction MUST
   be aborted by the CE.

Note that a transactional operation is generically atomic, therefore it requires
that the execute modes of all messages in a transaction operation should always
be kept the same and be set to 'execute-all-or-none'. If the EM flag is set to
other execute modes, it will result in a transaction failure.

   As noted above, a transaction may span multiple messages.  It is up to the CE
to keep track of the different outstanding messages making up a transaction.  As
an example, the correlator field could be used to mark transactions and a
sequence field to label the different messages within the same atomic
transaction, but this is out of scope and up to implementations.

4.3.1.2.3.  Recovery

   Any of the participating FEs, or the CE, or the associations between
   them, may fail after the EOT response message has left the FE but
   before it has received all the responses, e.g if the EOT response
   never reaches the CE.

   In this protocol revision, for sake of simplicity as indicated in
   Section 4.2.2.3, an FE losing an association would be required to get
   new state from the newly associated CE from scratch upon a re-
   association.  The decision on what an FE should do after a lost
   association is dictated by the CE Failover policy (refer to Section 9
   and Section 7.2).

4.3.2.  Scalability

   It is desirable that the PL layer not become the bottleneck when
   larger bandwidth pipes become available.  To pick a hypothetical
   example in today's terms, if a 100Gbps pipe is available and there is
   sufficient work then the PL layer should be able to take advantage of
   this and use all of the 100Gbps pipe.  Two mechanisms have actually been
provided to achieve this.  The first one is batching and the second one is a
command window.

   Batching is the ability to send multiple commands (such as Config) in
   one Protocol Data Unit (PDU).  The size of the batch will be affected
   by, amongst other things, the path MTU.  The commands may be part of
   the same transaction or part of unrelated transactions that are
   independent of each other.

   Command windowing allows for pipelining of independent transactions
   which do not affect each other.  Each independent transaction could
   consist of one or more batches.

4.3.2.1.  Batching

   There are several batching levels at different protocol hierarchies.

   o  multiple PL PDUs can be aggregated under one TML message

   o  multiple LFB classes and instances (as indicated in the LFB
      selector) can be addressed within one PL PDU

   o  Multiple operations can be addressed to a single LFB class and
      instance

4.3.2.2.  Command Pipelining

   The protocol allows any number of messages to be issued by the CE
   before the corresponding acknowledgments (if requested) have been
   returned by the FE.  Hence pipelining is inherently supported by the
   protocol.  Matching responses with requests messages can be done
   using the correlator field in the message header.

4.3.3.  Heartbeat Mechanism

   Heartbeats (HB) between FEs and CEs are traffic sensitive.  A HB is
   sent only if no PL traffic is sent between the CE and FE within a
   configured interval.  This has the effect of reducing the amount of
   HB traffic in the case of busy PL periods.

   A HB can be sourced by either the CE or FE.  When sourced by the CE,
   a response can be requested (similar to the ICMP ping protocol).  The
   FE can only generate HBs in the case of being configured to do so by
   the CE.  Refer to Section 7.2.1 and Section 7.9 for details.

4.3.4.  FE Object and FE protocol LFBs

   All PL messages operate on LFB constructs as this provides more
   flexibility for future enhancements.  This means that maintenance and
   configurability of FEs, NE, as well as the ForCES protocol itself
   must be expressed in terms of this LFB architecture.  For this reason
   special LFBs are created to accommodate this need.

   In addition, this shows how the ForCES protocol itself can be
   controlled by the very same type of structures (LFBs) it uses to
   control functions such as IP forwarding, filtering, etc.

   To achieve this, the following specialized LFBs are introduced:

   o  FE Protocol LFB which is used to control the ForCES protocol.

   o  FE Object LFB which is used to controls attributes relative to the
      FE itself.  Such attributes include FEState (refer to model
      draft), vendor, etc.

   These LFBs are detailed in Section 7.2.

Place #3:  Page 28 changed to:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   |       |   |   | |   |                                   |
      |ACK|  Pri  |Rsr|EM |A|TP |     Reserved                      |
      |   |       |vd.|   |T|   |                                   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       - ACK: ACK indicator(2 bit)
           The ACK indicator flag is only used by the CE when sending a
           Config Message(Section 7.5.1) or a HB message (Section 7.9)
           to indicate to the message receiver whether or not a
           response is required by the sender.  Note that for all other
           messages this flag MUST be ignored.

(*** Please note the change in the text above)

Place #4: Page 30 changed to:

       - EM: Execution mode (2 bits)
           There are 3 execution modes, refer to Section 4.3.1.1 for
           details.

                Reserved.................... (0b00)

               `execute-all-or-none` ....... (0b01)

               `execute-until-failure` ..... (0b10)

               `continue-execute-on-failure` (0b11)

       - AT: Atomic Transaction (1 bit)
This flag indicates if the message is stand-alone message or one of multipul
messages that belongs to 2PC transaction operations. See Section 4.3.1.2 for
details.

               'Stand-alone message'........ (0b0)

               `2PC transaction message' ... (0b1)

       - TP: Transaction phase (2 bits)
           A message from the CE to the FE within a transaction could be
           indicative of the different phases the transaction is in.
           Refer to Section 4.3.1.2 for details.

               `SOT (Start of Transaction)` --- (0b00)

               `MOT (Middle of Transaction)` -- (0b01)

               `EOT (End of Transaction)` ----- (0b10)

               `ABT (Abort)` ------------------ (0b11)

*** The End ***

Thank you.
Weiming