Subject: Draft on Advice for Internet Subnetwork Designers

From: "Falk, Aaron" <[email-masked]>
Date: Thu, 24 Feb 2000 12:02:08 -0500

Folks--

A very important piece of work for PILC is the LINK draft that Phil Karn is
editing. There has been a strong consensus that such a document is needed to
give link designers a clue about how to carry Internet traffic. As Phil has
eloquently put it, "most errors are those of commission, rather than
omission." So, there is a need to express to folks designing links what
services Internet protocols provide in a way that they can make design
decisions. I've had a lot of these conversations on the satellite systems
that worked on and I know many of the subscribers on this list have also.
What is needed is to capture the design principles in a single place so that
the gap between the physical layer community (particularly RF) and the
Internet community (particularly Transport) can be bridged. 

I would like to encourage those who haven't done so recently to read
http://www.ietf.org/internet-drafts/draft-ietf-pilc-link-design-01.txt and
comment to this list. Also, there are some holes in the document that need
filling. (see below) I don't believe any individual can do this whole thing
but we need someone to commit some time to generate a few paragraphs so that
we can converge on some recommendations.

The deadline for draft submission in time for the next IETF is March 10th.
So, to incorporate your comments and turn around an iteration of the draft,
a rapid dialog is needed. 

Thanks in advance for your help.

--aaron

=================

LINK placeholder sections:

Quality of Service, Fairness vs Performance, Congestion signalling

   [subnet hooks for QOS bits]

Delay Characteristics

   [self clocking TCP, (re)transmission shaping]

Buffering, flow & congestion control

   [atm dropping individual cells in a packet means the entire packet
   must be dropped unless EPD/PPD is used]

Mobility

   [best provided at a higher layer, for performance and flexibility
   reasons, but some subnet mobility can be a convenience as long as
   it's not too inefficient with routing]

Routing

   [what is proper division between routing at the Internet layer and
   routing in the subnet? Is it useful or helpful to Internet routing to
   have subnetworks that provide their own internal routing?]

Security

   [Security mechanisms should be placed as close as possible to the
   entities that they protect. E.g., mechanisms that protect host
   computers or users should be implemented at the higher layers and
   operate on an end-to-end basis under control of the users. This makes
   subnet security mechanisms largely redundant unless they are to
   protect the subnet itself, e.g., against unauthorized use.]

---
Aaron Falk
Director - Internet Services Development
PanAmSat Corporation
Greenwich, Connecticut, USA
*	Tel: 203.861.8326
*	Fax: 203.861.8677
*	mailto:[email-masked] 
*	http://www.panamsat.com

---

From: Dr G Fairhurst <[email-masked]>
Date: Mon, 28 Feb 2000 15:06:57 +0100

Here are some comments on Link Level Issues:

"Falk, Aaron" wrote:

> editing.
> http://www.ietf.org/internet-drafts/draft-ietf-pilc-link-design-01.txt

PAGE 4: MTU Size

It may be useful to note that using IPSEC in tunnel mode can increase
the size of a packet as it travels across the network.  This can result
in a smaller MTU.

PAGE 6: ALTERNATE FRAMING TECHNIQUES

In AAL5, the 1-bit per segment flag, carried in the ATM header,
indicates the end of a PDU.  The PDU control information (trailer)
is located at the end of the segment.  Placing the trailer in a
fixed position may simplify hardware reassembly.

Another framing technique is to insert per segment overhead
which indicates the presence of a segment option.  When present,
the option carries a pointer to the end of the PDU.  This differs
from AAL5, in that it permits another PDU to follow within the
same segment.  MPEG-2 supports this style of fragmentation, and
may utilise either padding (limiting each transport stream packet
to carry only part of one PDU with padding, or to allow a second
PDU to start (no padding)).

Reference: MPEG-2 or DVB reference.

PAGE 6: SEGMENTATION AND CRCs

I can find no recommendations on integrity checks.

I enclose some text below:

Most packet protocols rely upon error control at the data link layer using a
Cyclical Redundancy Check (CRC) at the end of the frame.  Some character-based
protocols rely on less powerful error detection algorithms (e.g. parity, block
checksum character); these are not well suited to a link which may be subject
to varying levels of error.

The IPv4 packet header contains a header checksum that verifies the integrity
of the header. This checksum was originally design to protect from router
software errors, however this function is no longer necessary.  IPv6 does
not therefore implement a network level integrity checksum. The role of the
IP checksum has therefore been subsumed into the TCP and UDP checksums.

This has two implications for satellite links:
(1)  Transport protocols should use a payload checksum when using IPv4
satellite links and must implement a checksum when used with IPv6,
(previously UDP had an option to use no checksum (value=0) which had for
a time been widely used for NFS in LANs).

(2)  Link layer protocols should provide adequate protection from
transmission errors.  Whereas in previous cases, small CRCs or no CRC
(in the case of SLIP) have been tolerated.  A CRC-32 is desirable for
each packet (as in Ethernet).  Links that perform transparent fragmentation
may use apply the CRC to the PDU prior to segmentation and check after
reassembly.

Reference on suitability of CRC-32:

"M. Greenwald, C. Partridge, and J. Hughes, Performance of Checksums
and CRC's over Real Data, IEEE/ACM Transactions on Networking, 6(5),
529-543 (1998)"

PAGE 13: Link ARQ

I think it may be wise to note that there are two approaches to TCP
friendly link ARQ.  One has been put forward by Balakrishnan:

"H. Balakrishnan, V.N. Padmanabhan, S Seshan, and R H Katz, A
Comparison of Mechanisms for Improving TCP Performance over
Wireless Links. IEEE Trans Net, 1997. 5(6): p. 756-759."

In this case, TCP headers are read and the information used to tune
the link protocol.

The other approach is to bound link behaviour to match TCP
"requirements" i.e. A receiver may calculate the variation
of the recovery delay and then give up, if the variation
which will be introduced is more than that can be tolerated
by the TCP transport layer.  This latter
approach was summarised in one of our papers:

"N Samaraweera and G Fairhurst Robust Data Link Protocols for
Connection-less Service over Satellite Links. Int J. Sat Coms,
1996. 14(5): p. 427-437."

PAGE 13: PACKET or FRAME?

The use of "Packet" with ARQ, could be confused here....

Consider the statement:
 "Note that the packet size plays an important role.  Larger packet
   sizes will allow for improved performance at the same *packet loss*
   rate."

This is only true if packet equate to MSS or transparent fragmentation
is used. Not so, for IP segmentation...

In the following calculations MSS is not correct, you mean MSS+all headers
including TCP, IP, Link and Physical pre/mid ambles.

PAGE 13: Hybrid ARQ

You do not mention that hybrid ARQ may combine the properties of ARQ and FEC.

PAGE 14: GEO Satellite Delay

 "Links running over geostationary repeater satellites have one-way
   times of around 250ms (125ms up to the satellite, 125ms down) so the
   RTT of an end-to-end TCP connection that includes such a link can be
   expected to be greater than 250ms."

Well, often a duplex satellite link is used. So the delay added is 2x250 ms
(one to get there, and one to get back).

------------

Best wishes,

Gorry Fairhurst

---

From: Dr G Fairhurst <[email-masked]>
Date: Mon, 28 Feb 2000 15:07:30 +0100

Here are some comments on TCP Issues:

> http://www.ietf.org/internet-drafts/draft-ietf-pilc-link-design-01.txt

PAGE 9: New-Reno

The ID says...

 "Taking this as a hint that the network is congested, TCP
   waits for the retransmission to be ACKed before it continues, and it
   gradually increases the number of packets in flight as long as a
   timeout does not occur again."

New Reno modifies this statement slightly, by adding the concept of
a retransmission phase:

   Hoe noted that a smaller value for ssthresh causes premature termination
   of the slow start phase and subsequent slow (linear) increase of the cwnd.
   A larger value causes the sender to over-feed packets to the network (i.e.
   transmit too long a burst of data packets) causing congestion.  Since
   most TCP sessions last only for a short period of time, the initial slow
   start period is significant for the over all performance.

New Reno also introduces the concept of a Fast Retransmission Phase which
improves performance when two or more segments have been lost (from the
same window of data).  In Reno, the Fast Retransmission and Fast Recovery
algorithms will not be able to recover such losses without waiting for
retransmission time out.  New-Reno overcomes this problem by introducing
a Fast Retransmission Phase, which starts on detection of a packet loss
(receiving 3 duplicate ACKs) and ends when the receiver acknowledges
reception of all data transmitted at the start of the Fast Retransmission
phase.  If more than one packet is missing within the same window, a
retransmission only recovers the first lost packet from the window.
The receiver then ACKs reception of the retransmitted segment and all
following segments up to the next lost segment.  This ACK is called a
"partial ACK", because it has not ACKed all the packets which were
transmitted prior to the start of the current Fast Retransmission Phase.

The transmitter assumes reception of a partial ACK during the Fast
Retransmission phase as an indication that another packet has been
lost within the window and retransmits that packet immediately to
prevent expiry of the retransmission timer. New Reno sets the cwnd
to one segment on reception of 3 duplicate ACKs (i.e. when entering
the Fast Retransmission Phase) and unacknowledged data are retransmitted
using the Slow Start algorithm. The transmitter is also allowed to
transmit a new data packet on receiving 2 duplicate ACKs. While the
transmitter is in the Fast Retransmission Phase, it continues to
retransmit packets using Slow Start until all packets have been
recovered (without starting a new retransmission phase for partial
ACKs). Although this modification may cause some unnecessary retransmissions,
it reduces transmitter time outs and efficiently recovers multiple packet
loss using partial ACKs.

PAGE 10: SACK

It is important to stress the role of SACK for TCP.  This has benefits in
terms of improved retransmission, but also in improved congestion
assessment when used in combination with new-reno.

For a discussion of SACK under satellite error conditions, see

"N Samaraweera, and G Fairhurst, Reinforcement of TCP/IP Error Recovery
for Wireless Communications. Computer Communications Review (CCR),
1998. 28(2): p. 30-38."
- Extended copy under review with IEEE since Feb 1999.

It may be useful to note that retransmission timer expiry can have a
very profound effect on performance.  I have found that some types
of interference can lead to multiple loss which results in
corruption of the retransmitted segment.  In normal implementations
of SACK, this results in timer expiry, even if there is information
to suggest the link RTT has not changed and no congestion is present
(see above paper for details). I can write more, if you like.

PAGE 15: Asymmetry

On asymmetry of capacity and ACK modification schemes you may also like to
consider quoting the alternate viewpoints in:

"N Samaraweera, Return Link Optimization for Internet Service Provision using
DVB-S Networks. ACM CCR, 1999. 29(3)."

or

"G Fairhurst, N. Samaraweera, H Harun, K Hodson and R Donadio, Performance
Issues
in Asymmetric Service Provision  using Broadband Satellite, IEE Proc (Comms),
2000."

------------

Best wishes,

Gorry Fairhurst

---

From: Lloyd Wood <[email-masked]>
Date: Mon, 28 Feb 2000 15:32:22 +0000 (GMT)

On Mon, 28 Feb 2000, Dr G Fairhurst wrote:
[..]
> New Reno sets the cwnd
> using the Slow Start algorithm.

setting cwnd to 1 segment and doing slow start describes timeout,
surely? The fast recovery algorithm would involve halving cwnd...

See RFC2581 section 3.2, equation 3, and RFC2582, equation 1.

Your text is slightly confused by discussing this after discussing
additional partial acks, which doesn't help.

> assessment when used in combination with new-reno.

How, exactly? For SACK to work you need both SACK receivers and SACK
senders. A SACK receiver with a NewReno sender doesn't gain you
anything.

> - Extended copy under review with IEEE since Feb 1999.

http://www.acm.org/sigcomm/ccr/archive/1998/apr98/ccr-9804-samaraweera.html
if you're going to cite it, give the URL in addition.

> DVB-S Networks. ACM CCR, 1999. 29(3)."

that would be:
http://www.acm.org/sigs/sigcomm/ccr/archive/1999/jul99/ccr-9907-samaraweera4.html

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: Dr G Fairhurst <[email-masked]>
Date: Mon, 28 Feb 2000 16:14:42 +0100

Thanks for the Lloyd for the URL's...

> senders. A SACK receiver with a NewReno sender doesn't gain you

Sure you need a SACK implementation at both sender and receiver. I take
that as granted.

I think one should look at the two together, SACK operates as a loss repair
mechanism, by informing the sender that packets have been received, it
pre-empts possible erroneous retransmission. It also gives explicit information
about segments leaving a network (i.e. ones which have made it all through
the network path and  therefore no longer occupy router buffers.) That is, with
SACK information received from the remote, the sender is able
to accurately know which segments have left, and therefore to inflate
cwnd at the transmitter during the recovery phase. In the latter way, it
may act in conjunction with the congestion avoidance algorithm.

Gorry

---

From: Lloyd Wood <[email-masked]>
Date: Mon, 28 Feb 2000 17:03:06 +0000 (GMT)

On Mon, 28 Feb 2000, Dr G Fairhurst wrote:

> I think one should look at the two together,

"when used in combination with new-reno"? Oh hang on, you mean NewReno
*and* SACK? 

http://src.doc.ic.ac.uk/rfc/rfc2582.txt

   This document applies only for TCP connections that are unable to
   use the TCP Selective Acknowledgement (SACK) option.

Sorry, you've lost me completely.

> SACK operates as a loss repair
> pre-empts possible erroneous retransmission.

Not always. (Thinking of multipath scenarios leading to reordering
here.)

 It also gives explicit information
> may act in conjunction with the congestion avoidance algorithm.

I thought recovery and congestion avoidance were entirely separate
transmission phases?

L.

---

From: Lloyd Wood <[email-masked]>
Date: Mon, 28 Feb 2000 17:02:04 +0000 (GMT)

> the size of a packet as it travels across the network.  This can result

Doesn't RFC2401 already cover this safely in PMTU advertisement?

> software errors, however this function is no longer necessary. 

bit strong.

Rather, header corruption is assumed sufficiently rare and hosts
sufficiently robust at coping with any misdelivery that endhost
handling and subsuming into TCP/UDP checksums was considered feasible.

> IPv6 does

This (the draft, the implication...) is not at all satellite-link
specific. It's general to anything seeking to replace TCP or UDP. (I'm
expecting the UDP-Lite crew to break cover round about... now.)

> transmission errors.

It's a tradeoff dependent on the characteristics and use of the link,
since the end-to-end checksum catches a number of errors. If the
reader thinks 'adequate protection' means 100% reliability means ARQ,
you've got problems. How about:

 Link layer protocols should provide adequate protection from
 transmission errors. This is necessary since:
 1.  there is as yet no internetwork framework for detecting or
     handling link transmission errors effectively
 2.  it is desirable from a performance viewpoint to avoid triggering
     congestion control algorithms needlessly as result of errored
     link-layer frames being discarded.
 3.  the TCP and UDP checksums are neither strong nor foolproof.

>  Whereas in previous cases, small CRCs or no CRC
> each packet (as in Ethernet).

...or as in ATM AAL5. The best choice of CRC might be dependent on
advertised MTU and link characteristics; probably need a note on
transparent fragmentation here. I can't think of any references as to
why CRC-32 is considered adequate/sufficient offhand, but some would
be nice.

> In this case, TCP headers are read and the information used to tune

this breaks layering. Ugh. And it's a fat lot of use with e.g. IPSec.
Not my first or recommended choice.

> by the TCP transport layer. 

and this assumes that the link in question is the only link in the
path doing such calculations. ugh again.

Really, you should imo be saying that if the variation is any more
than an order of magnitude less than the limit of transparency to TCP,
give up on the resend requests. If it's likely to be regularly more
than an order of magnitude less than the transparency limit, be very
skeptical of ARQ.

(500ms RTO gives 50ms per-link limit, which is probably a bit on the
 high side, given that the draft talks about 10-20ms delay already.
 But two orders of magnitude would be 5ms. I have to stop myself
 thinking in binary magnitudes too...)

cheers,

L.

---

From: Phil Karn <[email-masked]>
Date: Mon, 28 Feb 2000 09:58:28 -0800

Gorry,

Many thanks for your extensive comments! I'll work them into the next
revision of the document. I'll be at the IAB Wireless workshop most of
this week, so it may have to wait until this weekend.

Phil

---

From: Reiner Ludwig <[email-masked]>
Date: Tue, 29 Feb 2000 11:30:01 +0100

to throw in some papers from our side ... ;-)

>1996. 14(5): p. 427-437."

I third approach is to have the link layer

1. distinguish between flow types by using the protocol-id field in the IP 
header or the ds-field (if appropriate codepoints were defined which is not 
the case today).
See http://iceberg.cs.berkeley.edu/papers/Ludwig-Infocom99/index.html

2. Do little or no link layer ARQ for real-time flows

3. Do highly persistent link layer ARQ (delaying packets up to 2 minutes 
(MSL) !) for reliable flows such as TCP-based flows.
See http://iceberg.cs.berkeley.edu/papers/Ludwig-WINET99/

The latter paper shows that this approach:

A. Leads to optimal e2e performance

B. Renders obsolete any effort to make TCP/IP compression more robust

Also note that:

i. this approach rarely causes spurious timeouts
See http://iceberg.cs.berkeley.edu/papers/Ludwig-Sigmetrics99/index.html
which is due to TCP's overly conservative retransmission timer
See http://iceberg.cs.berkeley.edu/papers/Ludwig-Eifel-Xmit/index.html

ii. TCP can easily be made robust against spurious timeouts and packet 
re-orderings should they occur
See http://iceberg.cs.berkeley.edu/papers/Ludwig-Eifel-Alg/index.html

///Reiner