Subject: Response to Ron Cohen's comments on control words

From: "Yaakov Stein" <[email-masked]>
Date: Mon, 25 Nov 2002 14:01:56 +0200

I would like to respond here to Ron Cohen's remarks
regarding controls words.

Ron said that in his opinion there is no reason
for layer-one and layer-two services to use the same control word.

There are in fact several good reasons.

First, we have a single PWE WG and not two, 
and we are trying to emphasize commonality between
all the services we carry.
Why should a single WG suggest two different CWs?
(Remember the recent RSVP-TE vs CR-LDP squabble.)

Second, we are going to try to handle these issues
(e.g. sequencing) in the general drafts and outside the service drafts.
This would seem to require a single unified CW.

Third, why should implementers have to support two different formats,
when the same information is carried?

Ron further stated that layer 1 services could NOT use the Martini CW.  
TDMoIP is a layer 1 transport service and DOES use the Martini CW.

The only case where the CEP header makes any sense is when
there is a structure pointer, which is not all layer 1 services.
However, CESoPSN and uCESoPSN (which do not have structure pointers)
have adopted this CW as well, in order to unify implementations. 
So this format is influencing non-SONET drafts as well.

In addition to these reasons, there are specific
characteristics to the CEP header which make it less than ideal.

First, it uses fields which are not whole bytes
and are not easily aligned.

Second, it has the extension bit E, which implies
more processing is required.

Third, it does not reserve the first nibble as in the Martini CW.
This would limit its application.

As I show in my draft, all of these problems 
can be easily resolved, 
if the parties involved are open to consider
unification of the CWs.

Ron did not like the proposal I depicted in the slides
due to it's higher overhead.  This is indeed a problem,
but in my draft I have alternative proposals that do NOT
require extended CW formats.

Furthermore, Ron himself corrected me my stating that they 
DO use an extended CW format.
In any case, people using SONET frames to transport
single T1/E1 trunks as in the SONET-VT draft
seem not to be overly worried about high overhead.

Y(J)S

---

From: "tom k. johnson" <[email-masked]>
Date: Mon, 25 Nov 2002 12:36:05 -0500

Yaakov - 

I understand your intent, but I think there are some 
practical reasons for the L1 and L2 control words to 
be different.  Note that we recently split the L1 
requirements from the rest of the PWE3 requirements.  
I think that in general L1 services are quite 
different from L2 services, and the differences 
between the L1 and L2 control words are logical 
and warranted.  

A few responses to your specific points below...

> Why should a single WG suggest two different CWs?

Because L1 services and L2 services 
have different requirements.  

> This would seem to require a single unified CW.

Not necessarily.  For example, we'd like the 
sequence number in the L1 control word to follow the 
LSBs of the sequence number in the RTP header 
(if it's present).  That contradicts how the 
L2 sequence numbers are currently used.  Unless
we are considering the possibility of using the 
RTP sequencing rules for L2 services (?)

> when the same information is carried?

I would submit that the information carried is not
the same.  For example, none of the L1 service 
drafts utilize a length field (which is present
in the L2 control word).  Conversely, all of the 
structured L1 service drafts currently moving forward
in the WG utilize the concept of a structure pointer, 
although (for some reason) your draft opted not 
to use the one in the L1 Control Word :-)

> and are not easily aligned.

The lengths for the fields in the CEP header were 
arrived at through a trade-off of requirements.
Whole-byte alignment of the fields was not viewed
to be as important as the other issues we were
trading off (such as seq # length, packet length,
etc.)

> more processing is required.

No additional processign is necessary for 
basic operation.  The 32-bit CEP header has all the 
information necessary to carry a L1 service 
over a PSN.  The E-bit is strictly for 
enhancements beyond basic transport of the L1 
service, such as payload compression or improved
performance monitoring.  

> if the parties involved are open to consider

Yaakov - We've made a lot of progress 
in moving all of the L1 service drafts 
towards a single L1 control word.  And, 
all of the L1 services fit pretty neatly 
into the format we've defined.  

So, rather than re-arranging the L1 and L2 
control words to fit each others requirements,
I would think the time and energy could 
be better spent on other issues.

My 2 cents.

Thanks!

-Tom

---

From: "Daniel Feldman" <[email-masked]>
Date: Mon, 25 Nov 2002 22:35:45 +0200

Hello Andrew, Ron, Yaakov and all,
	I have to humbly disagree with Ron and Andrew.
	I don't know what are the hidden motives for backing two
different formats in PWE3, but one thing is sure: this will make
implementations more complicated. If the goal is charging more money
from box-makers because of extra silicon, then OK, lets sell some more
sand. Else, I think the goal of a standardization body is to simplify
things. BTW, this is not a philosophical issue, it is a very important
issue: uniformity->simplicity->efficiency->cost effectiveness.
	Regards,
	Daniel Feldman.

-----Original Message-----
From: Andrew G. Malis [mailto:[email-masked]]
Sent: Monday, November 25, 2002 8:07 PM
To: Ron Cohen; 'Yaakov Stein'; [email-masked]
Subject: RE: [PWE3] Response to Ron Cohen's comments on control words

Ron hit the nail on the head when he said:

>Because there are different requirements.
>
>Yes, we thought that it would be a good thing to have unified layer 1
>CW, don't you think?  In layer 1 emulation logic, there are
>commonalities in the way CW is handled. ...
>  A real need, not a philosophical one.

I know the list is the final arbiter, but for the info of those of you
not 
in Atlanta, I saw general consensus in the room to keep the control
words 
intact.

Cheers,
Andy

_______________________________________________
pwe3 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pwe3

---

From: "Gray, Eric" <[email-masked]>
Date: Mon, 25 Nov 2002 16:24:20 -0500

Daniel,

	There is a certain amount of over-head that would result
from attempting to define a "one-size-fits-all" control word.  
Are you completely convinced that this would make sense?

Eric W. Gray
Systems Architect
Celox Networks, Inc.
[email-masked]
508 305 7214

-----Original Message-----
From: Daniel Feldman [mailto:[email-masked]] 
Sent: Monday, November 25, 2002 3:36 PM
To: Andrew G. Malis; Ron Cohen; Yaakov Stein; [email-masked]
Subject: RE: [PWE3] Response to Ron Cohen's comments on control words

        Hello Andrew, Ron, Yaakov and all, 
        I have to humbly disagree with Ron and Andrew. 
        I don't know what are the hidden motives for backing two different
formats in PWE3, but one thing is sure: this will make implementations more
complicated. If the goal is charging more money from box-makers because of
extra silicon, then OK, lets sell some more sand. Else, I think the goal of
a standardization body is to simplify things. BTW, this is not a
philosophical issue, it is a very important issue:
uniformity->simplicity->efficiency->cost effectiveness.
        Regards, 
        Daniel Feldman. 
-----Original Message----- 
From: Andrew G. Malis [mailto:[email-masked]] 
Sent: Monday, November 25, 2002 8:07 PM 
To: Ron Cohen; 'Yaakov Stein'; [email-masked] 
Subject: RE: [PWE3] Response to Ron Cohen's comments on control words 

Ron hit the nail on the head when he said: 
>commonalities in the way CW is handled. .... 
I know the list is the final arbiter, but for the info of those of you not 
in Atlanta, I saw general consensus in the room to keep the control words 
intact. 
Cheers, 
Andy 
_______________________________________________ 
pwe3 mailing list 
[email-masked] 
https://www1.ietf.org/mailman/listinfo/pwe3

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 25 Nov 2002 17:09:40 -0500

Daniel,

Sure, if it makes sense.  In this case, there are good reasons, which Ron 
went into at length, why there are different control word formats for L1 
and L2.  Yaakov's draft is artificial shoe-horning for its own sake, 
ignoring the differing functional requirements of the two layers.

Cheers,
Andy

--------

At 11/25/2002 10:35 PM +0200, Daniel Feldman wrote:

> implementations more complicated. If the goal is charging more money from 
> box-makers because of extra silicon, then OK, lets sell some more sand. 
> Else, I think the goal of a standardization body is to simplify things. 
> BTW, this is not a philosophical issue, it is a very important issue: 
>From: Andrew G. Malis 
>[<mailto:[email-masked]>mailto:[email-masked]]
><https://www1.ietf.org/mailman/listinfo/pwe3>https://www1.ietf.org/mailman/listinfo/pwe3 
>

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 08:45:14 +0200

Tom, 

Just a few comments interleaved.
 
> have different requirements.  

It is NOT an issue of L1 vs L2.
It is ONLY the SONET draft that has the extra structure pointer.
TDMoIP is a TDM draft, in fact it was the first of all the drafts.
It does NOT require a structure pointer, and the Martini CW is just fine.

Actually CESoPSN and uCESoPSN do NOT require the structure pointer either.
They could also use the Martini CW with no problem.

This is NOT an issue of layer 1 or layer 2.
It is simply an issue that one particular draft needs an extra field.

> RTP sequencing rules for L2 services (?)

Once again, I do not see any connection between
the sequencing convention and layer 1 or layer 2.
Drafts which want to make the sequence number optional,
whether layer 1 or layer 2, have used the "avoid zero"
convention.

I don't care which is chosen, but I do not see any reason
to use both conventions in one WG.

> in the L2 control word).  

Once again, I think you should go back and read the TDMoIP draft.
We DO use a length field.

> Conversely, all of the 

Sorry, TDMoIP is for structured TDM, but there is no structure pointer.
The new CESoPSN draft is for structured TDM, and there is no structure pointer.
The structure pointer is only for the SONET draft (and its direct derivative,
SONET-VT) and has no connection whatsoever with layer 1 vs layer 2.

> trading off (such as seq # length, packet length, etc.)

I can understand that.

> be better spent on other issues.

I do not recall any discussions on the CW in the design team.
In fact, when Sasha and I were working on the uCESoPSN
draft I said that I didn't want to go into the CW at this time,
since I consider this an issue for the WG at large.

Y(J)S

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 08:48:54 +0200

> I know the list is the final arbiter, but for the info of 
> those of you not in Atlanta, I saw general consensus in the room to keep the 
> control words intact.
> 
> Cheers,
> Andy

There were two comments for unifying the CWs
(Townsley and Koleyni)

and two for keeping two different CWs (Ron Cohen and Sasha).

Y(J)S

---

From: David Zelig <[email-masked]>
Date: Tue, 26 Nov 2002 09:42:35 +0200

Yaakov,

> From: Yaakov Stein [mailto:[email-masked]]
> Sent: Tuesday, November 26, 2002 8:45 AM
> To: tom k. johnson; [email-masked]

In your draft there are two options:
Option 1:
4 bytes unified control word: Here you sacrifice the S/N length to get it
squeezed. Especially in circuit emulation, if the S/N is too small you will
have uncertainty to where to place the packet in the buffer. The flags in
the suggested CEP header are not in the same place as in the suggested CW
for packets: where the consistency?
Option 2:
4 bytes common control word+extra 4 bytes to carry the pointer adjustments
in SONET: but here in martini you do not need the extra header, so you need
two different treatment anyway (unless you suggest to carry empty 4 bytes on
each L2 circuit. -:).

So from my observation:
1 is not technically correct solution
2 is actually different CW between L2 services and SONET based services ==>
different treatment anyway --> why pay the overhead?

>From implementation point of view, even if on the same ASIC, once the
packets is recognized as L1, anyway the treatment is totally different, so
any communality in CW is for external beauty only.

If you do not like the L1/L2 partition, use <L2, CESoPSN, TDMoIP><SONET>
partition, but here I still think the communality in implementation should
dictate L1/L2 partition.

David

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 11:59:50 +0200

> Are you completely convinced that this would make sense?

The extra overhead is in one particular case
(the one I chose to show in the slides).

If you wish to avoid this, I gave alternatives
in my draft, but they involve fewer bits in the sequence number.

The question is - 
  what are the criteria for selection of the CW ?

Is byte alignment important ?

What size sequence number is really needed ?

Is fragmentation indication needed ?

Is payload type indication a good idea ?

The purpose of my talk was to start a discussion of these issues.

If it really turns out that the SONET/SONET-VT draft
and all the other drafts are not compatible,
then so be it. (Although we will have to figure out
how to interwork - for example, for TDM with ATM.)

But I haven't seen a discussion of the issues,
just defence of one or the other based on pre-existing implementations.

Y)J(S

---

From: Sasha Vainshtein <[email-masked]>
Date: Tue, 26 Nov 2002 12:32:39 +0200

Yaakov, Tom, Ron, Andy and all,
Please see two comments/clarifications inline.
They refer more to the way this discussion is conducted than
to purely technical issues.

I also have some purely technical comments regarding one of the
CW-related issues, but I prefer to post these separately.

With best regards,
                                   Sasha Vainshtein
email:     [email-masked] <mailto:[email-masked]> 
tel:       +972-3-7659993 (office)
           +972-8-9254948 (res.)
           +972-58-674833 (cell.)

> It does NOT require a structure pointer, and the Martini CW 
To the best of my understanding, TDMoIP, when dealing with 
structured emulation,  uses its own structure pointers (inherited from 
AAL1, but no less structure poinetrs for that). It simply(?) keeps 
them in spread all the way in the payload instead of localizing 
them in the CW. In other words, the functionality is there, 
but the method is different. We can discuss at length which 
method is preferable, but to deny that the related functionality 
is not needed or not present  is a blatant contradiction to the facts!
... snipped ...
> since I consider this an issue for the WG at large.
I think that quoting one's own statement made in a private discussion,
especially without bothering to quote any response  (and thus creating 
a false impression of consensus) is absolutely inappropriate and
definitely cannot be taken as a valid argument in a public discussion 
on the WG mailing list.  And, with full respect,  neither Yaakov by 
himself nor even Yaakov and me when working together on the 
uCESoPSN draft do not represent the TDM DT.

---

From: [email-masked]
Date: Tue, 26 Nov 2002 10:41:46 -0000

Yakov,

One observation below.  regards, Neil

> Sent: 26 November 2002 10:00
> To: Gray, Eric; Daniel Feldman
> Cc: pwe3
NH=> I have also asked about this, ie why a particular size was chosen....I
never got any good answers.  One issue to consider is this....

If we have a failure that leads to a significant corruption of pkts (eg
large error burst or 'break'), then we really don't want the sequence number
to wrap *before* we declare that an unavailability threshold has been
crossed.....noting that before that time any QoS metrics being measured
should be allocated to the up-state, but not after, ie QoS collection should
be suspended and an unavailabilty duration measurements started instead.

This is perhaps not a concern for constant rate sources and fixed pkt sizes
since one can infer where a free-running sequence counter would be even when
its not present.  However, this would not be the case for a variable rate
client source, since now one could not tell the difference between a severe
pkt loss case (high rate clinet) and minor pkt loss case (low rate client)
if they both wrapped the SN to the same value.
[There is also the issue here of consistency amongst the CWs used for
various clients, ie do we want the same for all?]

As a general rule therefore, it would seem to make sense to me to ensure
that SN counters should not wrap in a period commensurate with that required
to define available/unavailable performance state changes for the highest
pkt rate expected.  In TDM and ATM client cases this is defined against a
10s evaluation period.

---

From: Sasha Vainshtein <[email-masked]>
Date: Tue, 26 Nov 2002 12:53:10 +0200

Yaakov and all,
Please see some comments inline.

With best regards,
                                   Sasha Vainshtein

> Sent: Tuesday, November 26, 2002 12:00 PM
> Cc: [email-masked]
... snipped ...
>   what are the criteria for selection of the CW ?
Probably we should start with that.
> Is byte alignment important ?
It depends. The answers may differ for fields you use as counters
and fields you use in logical operations only (and I happen to know that
all arithmetic can be reduced to shifts and a single logical operation...) 
> What size sequence number is really needed ?
It depends. For emulation of L1 services, some input has been provided by
Neil on this list.
> Is fragmentation indication needed ?
It depends. IMHO it is required only for emulation of packet-oriented
services.
> Is payload type indication a good idea ?
I do not see any reason for it. The payload type is signaled during
the PW setup. 
BTW, you have missed length indicator. IMHO this is needed only
for packet-oriented services over MPLS networks.
And, of course, you have missed the structure pointers...
> The purpose of my talk was to start a discussion of these issues.
It is possible that I have got it all wrong, but this was NOT my 
impression while reading the draft or attending the Atlanta 
presentation. But it is never too late to know...
> how to interwork - for example, for TDM with ATM.)
I am not sure I understand what kind of interworking you
have in mind. 
> just defence of one or the other based on pre-existing 
Can it be because other people have also misunderstood
the purpose of your talk, not just me?

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 15:03:57 +0200

Sasha (and everyone else)

> is not needed or not present  is a blatant contradiction to the facts!

I didn't say that TDMoIP in AAL1 mode doesn't use any pointers,
just that it doesn't need a structure pointer field in the CW.
(and even if it did the pointers are only single bytes,
and could easily be fit into a Martini-like CW).

However, as far as I know CESoPSN neither has nor needs
any structure pointer at all. So I am suprised that you are
making the point that such pointers are needed for all L1 services.

> especially without bothering to quote any response  (and thus 
> creating a false impression of consensus) is absolutely inappropriate and

If anyone thought I had implied that you agreed to this statement,
then I apologize.  I was just stating my own reason for going along
with a particular CW in a draft to which my name is attached.

In order to be completely clear, Sasha, nor any other person
on the TDM design team,  at no time endorsed
this opinion, which was my own.

Y(J)S

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 15:26:53 +0200

Neil, 

> NH=> I have also asked about this, ie why a particular size 
> was chosen....I never got any good answers.  One issue to consider is this....
> 

I agree that this is one way of defining what a sequence number
should do. However, as I discuss in my draft, I do not really think that
precise sequence numbers are really that important.

Consider how the sequence number is used.
It is either to detect packet loss, or to help re-order packets.

For the latter case, I don't really think anyone is going
to hold 64000 packets and then seamlessly re-order
an errant packet.  Reordering makes sense when the
misguided packet is delayed by a small number of packets.
Certainly one byte is sufficient for a sequence number 
whose main purpose is re-ordering.

What about packet loss detection?
This has two sub-functions.
For L1 real-time services we use the number of lost packets
to fill-in for the clock recovery. This only makes sense when
there are a VERY small number of lost packets. Once
there is a gaping whole, no TDM service can be kept running anyway.

On the other hand, if all one wants to do is to detect that
something is lost, then once again only a small sequence number
is needed. Since we don't have to know precisely how many
packets were lost, the only thing of importance is the 
probability that the number of packets lost is precisely
the length of the sequence minus one. This may be a problem
for 1-bit or 3-bit sequence numbers in AAL2 or AAL1,
but the probability of this happening even with uniformly distributed
packet loss lengths for one byte sequence numbers is 1/256,
i.e. 4 per thousand.  I think that is pretty good already.

So if you think about things this way, 
do we really need 16 bit sequence numbers?
Isn't sequencing really used mainly as a "hint"
that something went wrong, in which case
some higher layer mechanism is responsible 
for the clean-up?

Just my thoughts ....

Y(J)S

---

From: Sasha Vainshtein <[email-masked]>
Date: Tue, 26 Nov 2002 15:44:12 +0200

Yaakov and all,

With best regards,
                                   Sasha Vainshtein

> Sent: Tuesday, November 26, 2002 3:04 PM
> To: Alexander Vainshtein
> Cc: [email-masked]
My comment referred to the following text in your mail (snipped in
your response):
<quote>
TDMoIP is a TDM draft, in fact it was the first of all the drafts.
It does NOT require a structure pointer, and the Martini CW is just fine.
<end quote>
My English may be lacking, but, IMHO, there are more straightforward
ways to say that TDMoIP does not require a structure pointer IN THE CW
because it has it elsewhere.
> making the point that such pointers are needed for all L1 services.
I did not say that ALL L1 services require structure pointers, 
either in Atlanta or on the list.  Indeed I believe that UNSTRUCTERED
EMULATION of such services does not require a structure pointer 
(among other things, because it is agnostic to any structures). 
However, STRUCTURED EMULATION of L1 services must somehow 
provide access to structure(s) carried within each packet, and there 
are different ways to do that.  
BTW, another case when such access is required is EMULATION
OF CELL STREAMS - and the common method used for this purpose
both in the ATM encapsulation draft(s) *and* in TDMoIP  (locking the
cell/"subframe" alignment to the packet payload boundary)  has been
*reused* in CESoPSN. The commonality between EMULATION of 
CELL STREAMS and EMULATION of STRUCTURED BIT-STREAMS
(a.k.a. L1 services) with reasonably short structures is recognized in 
draft-ietf-pwe3-arch-01.txt. 
> this opinion, which was my own.
Thank you for an important clarification.

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 26 Nov 2002 13:54:31 +0000

The key principle is surely "no gratuitous differences".

The CW design with the greatest consensus is the one
designed by Luca. If there is a valid reason why a
specific CW needs to be different, that is fine, however
I think that it behooves the author to account for the
differences.

Stewart

---

From: David Zelig <[email-masked]>
Date: Tue, 26 Nov 2002 17:24:51 +0200

Yaakov,
In your analysis you ignore the fact that many impairments in the network
includes miss-order of few mili-seconds (especially when there are fast
re-route capabilities in the PSN), and I would like to be able to use the
miss-ordered packets. These packets are useless if the window of the
sequence number is smaller than the length of buffer I already use anyway
for de-jitter. The length of S/N together combined the structure pointer was
designed to fit this requirement and support much higher rates than
available today while keeping the CW to 32 bit.

David

> Sent: Tuesday, November 26, 2002 3:27 PM
> To: [email-masked]
> Cc: [email-masked]

---

From: "Yaakov Stein" <[email-masked]>
Date: Tue, 26 Nov 2002 18:48:06 +0200

David, 

> 4 bytes unified control word: Here you sacrifice the S/N 
> length to get it squeezed. Especially in circuit emulation, if the S/N is too 
> small you will have uncertainty to where to place the packet in the buffer. 
> The flags in the suggested CEP header are not in the same place as in the 
> suggested CW for packets: where the consistency?

I discussed the sequence number issue in a previous email.

Regarding the flags,  there is some overlap,
but on the whole you are quite right.
However, here I agree with Ron Cohen - 
there is no commonality in the flag meaning,
so there is no single box that will be monitoring
flags for different services and expecting them
to be in the same place.

> Option 2:
> 4 bytes common control word+extra 4 bytes to carry the 
> pointer adjustments in SONET: but here in martini you do not need the extra 
> header, so you need two different treatment anyway (unless you suggest to carry 
> empty 4 bytes on each L2 circuit. -:).

As I said in the draft, most services will require only the first 32 bits
and these will be standardized, and everything in the same place.
For SONET services, and only for these,
there will be an extra 32 bits.

> any communality in CW is for external beauty only.

Well, let's discuss the box I sell.
I want to support both unstructured TDM
(with the uCESoPSN draft using the CEP header)
as well as structured TDM (presently Martini)
and cell mode ATM containing AAL1 (which uses the Martini word).

Please explain to me what to do!!!!!!!!!!!!!!!!!!!!!!

> 
> If you do not like the L1/L2 partition, use <L2, CESoPSN, 
> TDMoIP><SONET> partition, but here I still think the communality in 
> implementation should dictate L1/L2 partition.

I simply don't see the connection between L1 and the SONET structure
pointer being required in the CW.

But my bottom line is very simple.
I don't care which CW is chosen, but I want a single CW.
Since more drafts have chosen Martini to date,
I thought it would be simpler to use it.

However, I am more than open to suggestions
regarding the implementation problem I mentioned above.

Y(J)S

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 26 Nov 2002 12:15:43 -0500

Yaakov,

>Please explain to me what to do!!!!!!!!!!!!!!!!!!!!!!

You use the CW that's appropriate to the application.

Cheers,
Andy

---

From: [email-masked]
Date: Tue, 26 Nov 2002 17:47:56 -0000

Thanks Yakov...I understand your arguments/points, but I have a few more
observations....please find them below.

regards, Neil

> should do.
>  However, as I discuss in my draft, I do not really 
> think that
NH=> OK, I understand your points.  But these are all secondary
considerations IMO based on an assumption we have some agreed metric that
defines up and down states.  Indeed, such actions are only valid in the up
state.  But think now of the general case of simply measuring some QoS
metric.  This has to have clearly defined criteria as to whether such QoS
aggregation is valid, and when it's not to suspend it and start a count of
the down time.  So how is this aspect going to be addressed?

I have suggested the SN ought to be large enough not to wrap for a period
commensurate with an unavailability metric......not for the (very) short
timescale pkt loss/masking actions, but as a general principle of design.

It the TDM case life can be made simpler since we are dealing with a
deterministic continuoius streaming client.....so we could have a shorter
initial evaluation period which can be concatenated to create the required
larger evaluation period (=> a shorter SN field is OK).  In the cell/pkt
client case this is not true.....unless there are periodic deterministic OAM
flows one can rely on being there to ensure some 'traffic' is always present
to differentiate between a quiescent client, a high-rate but heavy pkt-loss
client, and a low-rate but light-loss client.  IMO this is just common
sense, but I get the impression that it has not even been considered.

---

From: "Gray, Eric" <[email-masked]>
Date: Tue, 26 Nov 2002 15:56:37 -0500

Alex,

	The control word might seem to be a good target
for this sort of effort, but it is not.  There seems 
to be some confusion on this issue, but it is really
quite straight-forward, once you look into it.  Andy,
feel free to chime in if I get this wrong.  Here is a 
basic description of the players...

For ATM encapsulation (draft-ietf-pwe3-atm-encap-00.txt):

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Rsvd  | Flags |Res|   Length  |     Sequence Number           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

For Ethernet (draft-ietf-pwe3-ethernet-encap-01.txt):

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Reserved            |       Sequence Number         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

For SONET/SDH CEP (draft-ietf-pwe3-sonet-00.txt):

    0                   1                   2                   3  
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2  
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
    |0|R|D|N|P| Structure Pointer[0:12] |  Sequence Number[0:13]    | 
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  

What appears to be the recommended consolidated Control 
Word format, one which takes into account the needs of
CEP and provides sufficient coverage for the others
(draft-stein-pwe3-controlword-00.txt):

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  RES  |     FLAGS     |  Length   |           Seq #           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |            POINTER            |            RES                |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Applying higher math and using my slide rule, I find that 
the last format has two 32 bit words while the original 3
distinct formats only have one 32 bit word each.  Whichever
format is used would need to be in each encapsulated frame, 
resulting in one 32 bit word additional overhead per frame 
forwarded - if the consolidated format were used.

If this isn't bad enough, there is always the issue that 
it may turn out that one of the encapsulations actually
needs more information once we get more experience with
its use.  Using a single consolidated CW would result in
either going back to the drawing board for all of them
or using two different formats of so-called consolidated
format.  

Kind of defeats the purpose.

Eric W. Gray
Systems Architect
Celox Networks, Inc.
[email-masked]
508 305 7214

> From: Alex Conta [mailto:[email-masked]]
> Sent: Tuesday, November 26, 2002 2:39 PM
> To: Andrew G. Malis
> Cc: Yaakov Stein; David Zelig; [email-masked]
> Subject: Re: [PWE3] Response to Ron Cohen's comments on control words
> 
> Andy,
> 
> I would be concerned if the PWE3 WG would not make the effort to
> minimize the number of differences in mechanisms that have common
> aspects. The CW seems to be a good target.
> 
> Regards,
> Alex
> 
> "Andrew G. Malis" wrote:
> >
> > Yaakov,
> >

---

From: "Gray, Eric" <[email-masked]>
Date: Tue, 26 Nov 2002 17:24:05 -0500

Alex,

	I think that the majority of the working group
has already indicated that they would not agree with
your assertion that there is more value in a single 
standardized format.  As Andy said earlier, there 
certainly seemed to be a consensus at the meeting to
leave things as they were.

Eric W. Gray
Systems Architect
Celox Networks, Inc.
[email-masked]
508 305 7214

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 27 Nov 2002 08:31:02 +0200

Neil, 

> considerations IMO based on an assumption we have some agreed 
> metric that defines up and down states.  Indeed, such actions are only 
> valid in the up state.  But think now of the general case of simply measuring some QoS
> aggregation is valid, and when it's not to suspend it and 
> start a count of the down time.  So how is this aspect going to be addressed?

I totally agree that we need good QoS metrics,
but I am not sure that I agree with your method
of collecting them.

IMO the collection of statistics should be left to OAM flows 
dedicated to this purpose.  If you have read the TDMoIP draft 
you will see that I have defined a statistics gathering mechanism,
although I would much prefer that the WG as a whole
define a single standard method.

In any case, trying to use the sequence number as the 
main source of information for performance metrics
does not seem to be the correct thing to do
from a purely theoretical point of view.

> 
> I have suggested the SN ought to be large enough not to wrap 
> for a period commensurate with an unavailability metric......not for the 
> (very) short timescale pkt loss/masking actions, but as a general 
> principle of design.

For the non-TDM cases, where timing recovery is not an issue,
we are left with two reasons to monitor a sequence number,
namely reordering and what I called the "hint" to the fact
that packet loss has occurred. 

When the hint is detected, then I suggest starting up
the connectivity and one-way delay OAM flows
to see what is happening.

> deterministic continuoius streaming client.....so we could 
> have a shorter initial evaluation period which can be concatenated to create 
> the required larger evaluation period (=> a shorter SN field is OK).  In 
> the cell/pkt client case this is not true.....unless there are periodic 
> deterministic OAM flows one can rely on being there to ensure some 'traffic' is 
> always present to differentiate between a quiescent client, a high-rate but 
> heavy pkt-loss client, and a low-rate but light-loss client.  IMO this is just common

Now I fully agree with you.

It is not only common sense, but mandatory that we be able
to measure the fitness of the network to carry the service.

As Scott Bradner said in Atlanta,
it was precisely due to these issues that PWE3
was put in the transport area in the first place.

Y(J)S

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 27 Nov 2002 08:36:37 +0200

> Yaakov,
> In your analysis you ignore the fact that many impairments in 
> the network includes miss-order of few mili-seconds (especially when 
> there are fast re-route capabilities in the PSN), and I would like to be 
> able to use the miss-ordered packets. These packets are useless if the window of the
> sequence number is smaller than the length of buffer I 
> already use anyway for de-jitter. The length of S/N together combined the 
> structure pointer was designed to fit this requirement and support much higher rates than

David - I agree that the sequence number should be large
enough to disambiguate any packet that can be used.

The CEP header has a 14-bit sequence number.
Do I understand that your jitter buffer holds 16,000 packets?

Y(J)S

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 27 Nov 2002 08:42:53 +0200

> (draft-stein-pwe3-controlword-00.txt):

The extra 32 bits is only needed for the SONET case
(it is what the SONET draft calls an "extended format" control word).

> or using two different formats of so-called consolidated

If anything more than a few more bit flags is needed,
then this would imply new major functionality.
Such functionality would probably better be covered
by OAM mechanisms, than by trying to force everything
into the CW.

Y(J)S

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 27 Nov 2002 08:50:50 +0200

> leave things as they were.

As I already replied, this "consensus" you mention
was simply the comments of two people
(and the chair's explanation that the third CW is now optional).
Although these people spoke with strong conviction,
I do not believe that they amount to consensus.

After the meeting there was a hallway discussion
wherein I heard quite a few remarks to the contrary.

In particular several people were unhappy with the CEP header's
use of the first 4 bits in the CW.

Y(J)S

---

From: "Daniel Feldman" <[email-masked]>
Date: Wed, 27 Nov 2002 09:14:52 +0200

Interesting "consensus". If we were to count the number of
messages debating this subject, we would need at least 6 bits...
	Does not seem like consensus to me...
	Regards,
		Daniel.
PS: This is message number 32, unless I have message loss ;-)

> Sent: Wednesday, November 27, 2002 8:51 AM
> To: Gray, Eric; Alex Conta
> Cc: [email-masked]

---

From: Sasha Vainshtein <[email-masked]>
Date: Wed, 27 Nov 2002 10:05:18 +0200

David, Yaakov and all,
I may be absolutely wrong, but I seem to remember that CEP 
has been designed to deal with up to OC-192 rates. If CEP default payload
size is used (i.e. 783 payload bytes per packet), it would result 
in just below 1.6M packets per second. In other words, 16K packets in the
jitter 
buffer would just suffice for about 10 ms - does not look like asking for
too much
as far as I am concerned. 

Of course this is just the "ultimate worst case"... 

With best regards,
                                   Sasha Vainshtein

> Sent: Wednesday, November 27, 2002 8:37 AM
> To: David Zelig; [email-masked]
> Cc: [email-masked]

---

From: David Zelig <[email-masked]>
Date: Wed, 27 Nov 2002 10:58:30 +0200

The way to support (future) higher rates is by using longer packets. That's
why I said earlier in the thread that the pointer in the CW and the SN
combined length is what important for future growth. 
The current CW support any practical use in the coming years of STS-48c
emulation.

David

> From: Sasha Vainshtein [mailto:[email-masked]]
> Sent: Wednesday, November 27, 2002 10:05 AM
> To: 'Yaakov Stein'; David Zelig
> Cc: [email-masked]

---

From: Sasha Vainshtein <[email-masked]>
Date: Wed, 27 Nov 2002 12:08:04 +0200

David and all,
I agree with the need for longer packets for OC-192.
I just tried to show that, at the rates you deal with, 16K packets 
in the jitter buffer is not an overkill.

With best regards,
                                   Sasha Vainshtein

> From: David Zelig [mailto:[email-masked]]
> Sent: Wednesday, November 27, 2002 10:59 AM
> To: 'Sasha Vainshtein'; 'Yaakov Stein'
> Cc: [email-masked]

---

From: Sasha Vainshtein <[email-masked]>
Date: Wed, 27 Nov 2002 13:23:13 +0200

Neil,
The text below is an attempt to summarize my understanding
of your position.

1. All PWs must support some OAM mechanimsms that allow to define
   availability and finer QoS metrics of emulated services.  In particular,
   a way to distinguish between an unavailable service and a service
   that is available but does not function well must be found, because
   finer QoS metrics relate to the service up-state only.

2. PW availability definitions must be compatible with these defined for
    the native services being emulated over these PWs.

3. PW emulating L1 services may use data packets for OAM purposes 
    because thesearrive at a well predicted rate.

4. PWs emulating L2 services must use some dedicated OAM packets
    to distinguish between an available PW that does not carry any data
    due to CE silence and an unavailable PW.

If this summary correctly reflects your posistion, I would like to ask
you the following additional questions:

Q1: Are OAM mechanisms really necessary for PWs that emulate
      services without any native OAM (e.g., Ethernet)?

Q2: Can the PWs emulating an L2 service that admits some native
      OAM mechanism emulate exactly this mechanism for their
      OAM purposes? E.g., if you have a PW that emulates a FR
      service, would some form of LMI emulation between the endpoints
      of the PW suffice? 

I would highly appreciate your clarifications.

With best regards,
                                   Sasha Vainshtein

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, November 26, 2002 12:42 PM
> To: [email-masked]
> Cc: [email-masked]
> was chosen....I

---

From: "Yaakov Stein" <[email-masked]>
Date: Wed, 27 Nov 2002 14:46:49 +0200

> has been designed to deal with up to OC-192 rates. If CEP 
> default payload size is used (i.e. 783 payload bytes per packet), it would result 
> in just below 1.6M packets per second. In other words, 16K 
> packets in the jitter  buffer would just suffice for about 10 ms - does not look 
> like asking for too much as far as I am concerned. 

That's an interesting and valid point.
If we expect to provide something similar to 
SONET's 50 ms recovery from a fiber cut,
we will need even more.

But your observation leaves me with two questions.

1) How much does 16K * 783B = 12 MB of memory
fast enough for these speeds cost ?

2) If you are going to use this to re-order,
do you parse the entire jitter buffer each time
you have to remove a packet,
or do you sort the packets upon entry?
Either way it sounds like you need one of the
stronger of the Crays just to manage one such jitter buffer.

It still sounds as if the wisest engineering decision
is to consider fast re-routing to be a very infrequent event,
and not to insist that sequencing support its concealment.

Y(J)S

---

From: [email-masked]
Date: Thu, 28 Nov 2002 11:24:12 -0000

Yaakov Stein wrote 27 November 2002 06:43
<snipped>
> by OAM mechanisms, than by trying to force everything

NH=> I am concerned as to where this discussion is heading.  Statements like
'....better covered by OAM mechanisms' are an indication of poorly thought
out functional architecture.

If I ignore the control-plane aspects of instantiating a trail (I make no
apologies for using G.805 language as this is what you need to
properly/unabiguously define what is going on here), then the data-plane
server layer trail created must have the following functions:
-	an adaptation function for how the client signal is mapped into the
payload of the server (and how these are indicated, eg a PID field or by 1:1
mapping inference, ie no muxing of different clients allowed);
-	characteristic information added to the server layer to ensure
transport integrity....so as a minimum this is some form pkt
framing/alignment mechanism and a basic defect detection/handling OAM
mechanism.

So what have we architecturally going on in PWE3?  Well, there are a set of
client layer networks that are going to be carried by some form of server
layer PSN.  *If* both the server layer and the client layer have been
properly specified as layer networks in their own right, then the only thing
that should be required is the approparite adaptation function....as the
characteristic information required to ensure the intergrity of both the
client layer and the server layer are functionally embedded within their own
layer network description.

Further, the defects of each layer network should have been identified
up-front and specified in terms of automatic detection/handling
**independently** of other layer networks....why?.....because if you don't
do this there is no functional independence between layer networks which
means:
-	one can't have flexible/arbitrary client/server relationships;
-	independent layer network evolution cannot happen, as 1 layer
affects another layer
{Aside - Exactly the same issues hold for control-plane and data-plane
protocol relationships, ie they must be decoupled functionally....have a
read of Y.1710 for further details}

For some client layer networks this is true, eg SDH and ATM.....for others
it is not, eg ethernet.  For some server layer networks this is true, eg p2p
LSPs (Y.1711), IP (ICMP)....for others it is not, eg mp2p LSPs, ethernet.
But let me stress again....client/server relationships can be arbitrary (ie
one man's client can be another man's server), ergo need for layer
independence.

IMO the CW should be seen mainly as an adaptation function.  And in this
context, I have to agree with Eric that different solutions will be required
to adapt various clients into various servers.  Where it's not so clear
however, is when functionality is provided that is related to ensuring
integrity....and I am specifically referring to the SN field here.  However,
this is addressing a 'deficiency' of the server layer (or at least a
'behaviour' of the server layer) that is inconsistent with what the client
layer would expect from the (more 'usual') server layer, eg pkt misordering.

Now, as far as I am concerned, if such (SN) functionality is introduced (in
the CW) then it must be designed to be consistent with what the client layer
would expect. For PDH/SDH/ATM clients there is a very clear demarcation
point between up-state QoS specification and down-state unavailability
specification based on a 10s evaluation period.  So, not only should this be
addressed in the sever layer OAM per se as an independent layer function
(else client/server inconsistencies....apples? = oranges?), but it should
also be reflected in any such intergrity verifying functionality built-in to
the CW.

So this is why I said that the SN should ideally be designed not to wrap
under all expected cases for the order of 10s.

This observation is perhaps not so critical *if* the server layer is
providing automatic defect detection/handling *and* under real defect
conditions of the server layer......because here we should get an automatic
FDI feed to the client which tell us that things like SNs will be lost and
their processing is not valid.  However, it is more critical for the cases:
-	where the server does not have proper defect detection/handling and
real server layer defects are present;
-	where there are no real server layer defects present but, for
whatever reason, the server layer is persistently misordering/losing pkts.
Further, one really needs to differentiate between a quiescent client (or a
low-loss low-rate client) and a heavy-loss heavy-rate client.....but this
would not seem possible without some form of deterministic OAM flow.

Note also that in these latter cases (ie persistent SN anomalies), there
really ought to be 'SN processing' defects and consquent actions
specified.....though I don't hold much hope out for seeing this properly
addressed either.

regards, Neil

---

From: [email-masked]
Date: Thu, 28 Nov 2002 17:16:16 -0000

Hi Sasha....please see my responses below.  regards, Neil

>    finer QoS metrics relate to the service up-state only.
NH=> This is not how I would phrase it.  All layer networks must have
independent (see Note) data-plane fault-management.
Note - 'Independent' here means there are no specific client layer or server
layer (technology) data-plane dependancies, and no control-plane protocol
(be it routing or signalling) dependancies.
As regards the defect stuff, this is just a logical flow of consequences.
That is:
-	you can't measure QoS unless the entity is known to be 'up'
-	you can't know what is 'up' unless you can specify/measure up/down
transitions
-	you can't specify up/down transitions unless you have identified all
defects and specified their entry/exit criteria (and also consequent
actions, eg to protect client layers from alarm storms)
>     the native services being emulated over these PWs.
NH=> Again, its not a PW specific issue.....the obvious objective here is
harmonised (as far as practically possible) perf metrics/objectives over all
layer networks.  Its no good someone saying its broken at some client layer
if the problems are arising at a lower layer and, according to the perf
specification of the lower layer, people there say its behaving within
specification.  Also think of where the different vertical client/server
layer network relationships are owned by different commercial
entities....clearly as much harmonisation as possible is desirable.  The
same also applies to horizontal partitions within a given layer network,
which again can belong to different commercial entities......and
additionally here, fair apportinoment rules are needed that are consistent
with some larger end-end set of metric objectives.  Again this is all common
sense IMO.
>     because thesearrive at a well predicted rate.
NH=> Not sure what you mean here.  There ought to be independent OAM
functions in the server layer network and client layer network.....these
have nothing to do with the PW case.  
>     due to CE silence and an unavailable PW.
NH=> See previous comment.
>       services without any native OAM (e.g., Ethernet)?
NH=> The issue is nothing to do with PWs and all to do with the layer
networks themselves....unless of course one is advocating (and its very easy
to start moving in this direction) that a PW is creating a new layer network
(which I don't think is the case).  If a layer network has no OAM then it
has no OAM period.  You can't do much about that other than re-spin the
technology....though one could carry it on something that has decent
OAM....not ideal, but possible, eg ethernet->GFP->SDH_VC-4 (concat as
necessary) could be done, and such a model would make a great deal of sense
for carriers who have large SDH/Sonet deployments. Note that GFP here is
performing the adaptation function, rather like I imagined the CW would be
doing.   
>       service, would some form of LMI emulation between the endpoints
NH=> See previous remark.
> I would highly appreciate your clarifications.
NH=> Please also see my last response to Yakov on these matters.  I always
regarded the CW as a form of client<=>server adaptation function and not a
generic integrity function (which is step 1 on the road to defining a new
layer network).

---

From: Sasha Vainshtein <[email-masked]>
Date: Thu, 28 Nov 2002 19:35:05 +0200

Nail,
Thank you for a prompt and clarifying response.

With best regards,
                                   Sasha Vainshtein

> Sent: Thursday, November 28, 2002 7:16 PM
> To: [email-masked]
> Cc: [email-masked]

---

From: "Cheng-Yin Lee" <[email-masked]>
Date: 29 Nov 2002 14:07:06 -0500

Neil,

> LSPs (Y.1711), IP (ICMP)....for others it is not, eg mp2p LSPs, ethernet.

Is the implication of this then one could leverage the server layer integrity
mechanism to manage these services :
- Ethernet over p2p LSPs,  SDH/SONET, ATM
- Ethernet over IP
but perhaps not for:
- Ethernet over mp2p LSPs or Ethernet
- X over Ethernet?

thanks
cheng-yin

---

From: [email-masked]
Date: Fri, 29 Nov 2002 19:35:02 -0000

Hi Cheng-Yin....please see my (towards) end of mail response.

regards, Neil

> From: Cheng-Yin Lee [mailto:[email-masked]]
> Sent: 29 November 2002 19:07
> To: Harrison,N,Neil,IKL2 R
> Cc: [email-masked]
> > **independently** of other layer 
> networks....why?.....because if you don't
NH=> The trite answer is 'yes' (but see Note).  Any operator who has lots of
SDH/Sonet deployment would, IMO, be rather silly to miss the obvious
efficacy of an ethernet->GFP->SDH/Sonet solution for certain services. I
would, however, not want to constrain what others want to do by this
observation.
Note - It is not generally a good idea IMO to rely on (or force) the 'good'
functional specification of 1 layer network to proxy for missing functional
components in another layer network (be this either as a client or server
relationship).  However, short of re-spinning the deficient layer network
concerned, what is suggested above is a pragmatic tactical way forward.

regards Neil