Subject: Multiple processes in the CE talking to the same FE LFBs

From: Robert Haas <[email-masked]>
Date: Thu, 9 Jun 2005 15:05:38 +0200

Re-reading the draft I realize that we show in Fig. 2 how multiple 
processes (OSPF, RIP, etc) interface with the ForCES layer, but we never 
explain how the demuxing occurs. Some exchanges in this old thread 
suggested to use PL sequence number, PL message types, and/or LFB 
Instance IDs  as a means to correlate and demux a ForCES response to the 
appropriate process that sent the request. One could even store the 
entire request messages and event registrations in order to match 
responses appropriately. Such solutions are mere by-products (not to say 
hacks ;-)

Another solution consists in adding a correlator in the CE-originated PL 
messages that must be returned in the FE responses and event notifications.

I am interested to hear how others have addressed this issue.

Regards,
-Robert

Robert Haas wrote:

> Hormuzd,
>
> Khosravi, Hormuzd M wrote:
>
>> Robert,
>>
>> My comments on this inline....
>> -----Original Message-----
>> From: Forwarding and Control Element Separation
>> [mailto:[email-masked]] On Behalf Of Robert Haas
>>
>> Coming back to implementation issues that fire up in the protocol:
>>
>> In the case of a single active CE, there can be multiple processes 
>> "hiding" behind the same CE ID, i.e., a process updating firewalling 
>> rules and another process updating the routing table, in parallel. We 
>> must be able to:
>>
>> 1) correlate at the CE the responses from the FE with the commands 
>> that were originated by each process.
>> 2)  forward at the CE FE-event notifications to the appropriate 
>> processes that registered for it.
>>
>> So if the CE ID is the same for all these processes, something is 
>> missing in the protocol  to support 1) and 2) ... Unless the ForCES 
>> PL decides to use different TCP/UDP ports for each process (I 
>> wouldn't recommend this), but this will anyway not solve 2).
>>
>> [HK] This is more of an implementation issue than a protocol issue in my
>> opinion. You could have the ForCES protocol daemon demuxing between
>> different processes using a shim layer with standard IPC. The shim layer
>> would take care of translating requests from different processes to
>> different ForCES protocol message types, sequence nos, etc. Let me know
>> if this sounds reasonable or I am missing something here?
>>  
>>
> How far did you go with your implementation of demuxing ? In our case, 
> we found that relying on sequence numbers or message types is not 
> useful when you want to direct an event notification coming from the 
> FE to the appropriate process in the CE. But maybe you found a way to 
> solve that ?
>
> I hope that with our implementations we will find not only 
> implementation issues, but protocol issues too. I doubt we can get the 
> protocol right on the first shot.
>
>>
>> BTW, the multi-CE problem is something one should still think of 
>> while defining the model. Although the requirements draft only talks 
>> of a single active CE for each FE, it is a scalability issue if I 
>> can't run the firewalling and routing processes on two different 
>> boxes (i.e. two CEs). So let's at least consider the impact of the 
>> multi-manager problem
>>
>> on the model so we don't close all the doors for future support.
>>
>> [HK] I agree with Joel on this. I believe this is something which can be
>> addresses in a later rev of the protocol draft. We should focus on
>> getting the basic protocol messages/encoding correct at this point. That
>> itself is not a straight forward task cause of the many tradeoffs. We
>> can come back to the multi-CE issue in the next version of the protocol.
>>
>>  
>>
> I would treat this the same way we treated the keys: make provision 
> for multi-CE support  in the model as early as possible as it may 
> otherwise become more difficult to include it later.
>
> For instance, whenever CEs register for events: the model should not 
> only indicate with a boolean if a CE has registered to be notified 
> when the event occurs, but also include a list of the CEs that have 
> registered for the same event. Limited complexity for the model, 
> almost no additional complexity for the protocol.
>
> Regards,
> -Robert
>
>> Regards
>> Hormuzd
>>
>> Joel M. Halpern wrote:
>>
>>  
>>
>>> I thought we had agreed that for now we would focus on a single active
>>>   
>>
>>
>>  
>>
>>> CE per FE>  An FE that wants to be managed by several CEs is (I 
>>> thought) to pretend it is several FEs, probably with distinct
>>>   
>>
>> resources.
>>  
>>
>>> The multi-manager problem is very very messy.
>>>
>>>   
>>
>>
>>
>>  
>>
>

-- 
Dr. Robert R. Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-44-724-8698  fax +41-44-724-8578  http://www.zurich.ibm.com/~rha

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 9 Jun 2005 09:22:31 -0400

I don;t think there should be higher layer demux in the sense below.
I believe that the ForCES messges will need to be processed by the ForCES 
interface, turned into meaningful information for the CE internal 
activities, and passed on to the relevant component(s).

In general, I think that the selection of which component is interested in 
ForCES information is going to be based on semantics, not on demux 
identifiers in the message.

Note that specifically as drawn in the diagram it would be exceedingly 
unlikely for OSPF, RIP, and BGP to interact directly with LFBs.  They will 
interact with some form of RIB, which in turn (at least logically) drives 
some sort of FIB process.
To carry the example further, if one also has MPLS one can easily imagine 
FE notifications which are of interest to both the FIB and LIB 
activities.  and the FE would have no reasonable way of even knowing 
that.  Explicit demux seems the wrong answer.

On the other hand, Request - Response correlators are a very good idea, for 
many reasons.

Yours,
Joel M. Halpern

At 09:05 AM 6/9/2005, Robert Haas wrote:
>suggested to use PL sequence number, PL message types, and/or LFB Instance 
>IDs  as a means to correlate and demux a ForCES response to the 
>appropriate process that sent the request. One could even store the entire 
>request messages and event registrations in order to match responses 
>appropriately. Such solutions are mere by-products (not to say hacks ;-)
>
>>>1) correlate at the CE the responses from the FE with the commands that 
>>>were originated by each process.
>>>missing in the protocol  to support 1) and 2) ... Unless the ForCES PL 
>>>decides to use different TCP/UDP ports for each process (I wouldn't 
>>>recommend this), but this will anyway not solve 2).
>>>
>>How far did you go with your implementation of demuxing ? In our case, we 
>>found that relying on sequence numbers or message types is not useful 
>>when you want to direct an event notification coming from the FE to the 
>>appropriate process in the CE. But maybe you found a way to solve that ?
>>
>>I hope that with our implementations we will find not only implementation 
>>issues, but protocol issues too. I doubt we can get the protocol right on 
>>the first shot.
>>
>>>
>>>BTW, the multi-CE problem is something one should still think of while 
>>>defining the model. Although the requirements draft only talks of a 
>>>single active CE for each FE, it is a scalability issue if I can't run 
>>>the firewalling and routing processes on two different boxes (i.e. two 
>>>CEs). So let's at least consider the impact of the multi-manager problem
>>>
>>I would treat this the same way we treated the keys: make provision for 
>>multi-CE support  in the model as early as possible as it may otherwise 
>>become more difficult to include it later.
>>
>>For instance, whenever CEs register for events: the model should not only 
>>indicate with a boolean if a CE has registered to be notified when the 
>>event occurs, but also include a list of the CEs that have registered for 
>>the same event. Limited complexity for the model, almost no additional 
>>complexity for the protocol.
>>
>>Regards,
>>-Robert
>>
>>>Regards
>>>Hormuzd
>>>
>>>Joel M. Halpern wrote:
>>>
>>>
>>>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 10 Jun 2005 07:00:46 -0400

> On the other hand, Request - Response correlators are a very good idea, for 

Where would you stick the correlator in the message in such a case?
Is it a TLV in the top hierachy or would there be another 32 bit in the
main header?

We have not so far needed the correlator so far for the simple reason
that we have a _single process_ - so all messages get the opportunity to
be demux from a single point. We do use the sequence number as a
correlator when we have some app using the NPF APIs over forces in
particular.
Regardless: should be noted that this(single process/task) is
implementation detail i.e that there are other ways to implement. 
It is more efficient for us to implement a single task - but i do wonder
about implementations that have multi processes. Our coding has not yet
got to HA, but i have been worried at times while doodling on paper that
we may need something akin to a second 32 bit number.
Now if one was to use TIPC as a TML, TIPC would take care of this
demuxing amongst multi processes since it is inheritenly good at IPCs
(thats where the IPC in TIPC comes from ;->). But again that would be
implementation specific.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 10 Jun 2005 09:15:37 -0400

I would put the correlator in the main header.
Not only is it useful if the CE is multiple processes, but it is useful if 
the FE can respond to pipelined requests out of order.  (Suppose that some 
operations require long hardware manipulation, and some just update a 
table.  One could easily have responses come back out of order.  The 
correlator makes that a non-issue.  The absence of the correlator in HTTP 
makes some things "interesting".)

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 10 Jun 2005 11:48:21 -0400

> I would put the correlator in the main header.

This would be fine with me. I have no immediate use for it;
Lets hear what other folks have to say.

cheers,
jamal