Subject: Re: [Sipping] Last Call: draft-ietf-sipping-overload-reqs

From: Volker Hilt <[email-masked]>
Date: Thu, 15 May 2008 23:29:57 -0400

Matt,

> I have serious concerns about this document because it explicitly excludes the 
> only approach for coping with overload that is guaranteed to be robust under 
> all conditions.  Although I know it is considered bad form to describe 
> solutions while debating requirements, I think a sketch of a solution will 
> greatly clarify the discussion of the requirements.
> 
It seems reasonable not to exclude an implicit mechanism from the 
solution space. We know that with the current SIP protocol, simply 
dropping messages as a response to overload performs poorly and may lead 
to congestion collapse. But there may of course be other implicit 
mechanisms. The requirement 6 might need a slight rewording to address 
this issue:

    REQ 6:  Any response codes, header fields, or other protocol
       machinery defined for overload control shall be used exclusively
       for overload handling, and not be used to indicate other failure
       conditions.  This is meant to avoid some of the problems that have
       arisen from the reuse of the 503 response code for multiple
       purposes.

I don't think a solution sketch will be helpful in a requirements document.

> The only robust overload signal is the natural implicit signal - silently 
> discarding excess requests.  Explicit overload messages (code 503) should be 
> optional, and must have an explicit rate limit.  The error message may be 
> cached (e.g. in proxies, etc) but must not be required to be cached.  All 
> retransmissions in all parts of the protocol must back off exponentially 
> (which I am told is already true for SIP).
> 
No, silently discarding excess messages will not lead to a good 
performance of a SIP server. If incoming messages are randomly discarded 
(i.e. without processing them at all), the number of successful INVITE 
transactions will eventually drop to close to zero. The probability that 
all messages of a transaction will go through will become very small. If 
messages are parsed and selected, the capacity a server is able to spend 
on successful calls will reach zero as you describe blow.

An explicit signal can help to tune the offered load to a level that is 
close to but not above the capacity limit of a SIP server. REQ 15 adds 
to robustness and addresses the case in which a proxy is unable to 
generate feedback and prevents a server from being overloaded when 
things go wrong.

> Sending additional messages to explicitly indicate overload is intrinsically 
> fragile.  If the overload management mechanism consumes any shared resource 
> that might be needed to complete other calls, then there exists some operating 
> point where any additional requests will cause a decline in the number of 
> successfully completed calls.  This is likely to be regenerative, with each 
> successive error using more resources and preventing more calls, until the 
> throughput crashes to zero.  This phenomena was readily apparent in all of the 
> plots shown in the tsvwg meeting at IETF 71.
> 
What you are describing is a local overload mechanism in which an entity 
has no way of pushing back and needs to somehow process all incoming 
messages by either forwarding or rejecting them. This case is indeed 
shown in the graphs of the TSVAREA presentation in the first part, where 
we describe the problems of today's mechanisms. Scenario 2 and 3 show 
the use of 503 responses for overload control.

If you move ahead in this presentation you'll see graphs for the 
solutions that use feedback mechanisms as discussed by the design team.

> Note that if the explicit overload management mechanism is very complicated, 
> the situation that triggers this failure might also be very complicated. 
> Asserting that this hazard does not exist is probably equivalent to proving 
> that explicit overload notifications never cause additional calls to fail, for 
> all combinations of implementations under all operating conditions.  It would 
> not be an easy task to prove that the standards are sufficient to guarantee 
> this for all possible implementations.
> 
I agree that a mechanism should be as simple as possible - but it has to 
work too.

> My specific objections to the document are as follows: Requirement 6 calls for 
> explicit overload messages and forbids silently discarding requests, since 
> they are not unambiguous in their meaning.  

See rephrased REQ 6 above.

> Requirement 15 seems to provide a 
> loophole (allowing complete failures) but seems to forbid using it as the 
> preferred mechanism.  

I don't think an implicit mechanism would be in violation of REQ 15.

> Requirement 8 does not make sense without explicit 
> notification.  Requirements 7, 8 and 9 should note that they can be (are
> already?) equivalently satisfied by properly structured exponential 
> retransmission backoff timers in SIP itself.
> 
No. The exponential backoff timers in SIP do NOT satisfy these 
requirements.

> I would like to point out that TCP, IP and several other transport protocols 
> have evolved in the same direction as I am advocating for SIP: the only robust 
> indication that an error has occurred is connection failure.  Error messages 
> are cached and sometimes accelerate timers (e.g. retransmit now, or go to the 
> next IP address now), but do not change basic protocol behavior.  Error 
> messages are most often rate limited at the sender and the saved error codes 
> are used to provide a clue why something failed, but the fact that it failed 
> most likely comes from a timer, not the message itself.  The number of error 
> massages that are required for correct operation is declining (note that 4821 
> makes ICMP can't fragment optional), and may be zero.
> 
> Rate limiting all errors messages and treating them as advisory improves 
> robustness in several ways: fraudulent messages have less impact, error 
> messages can not be used an DDOS attack magnifiers, and overload is addressed 
> implicitly by silently discarding requests.
> 
> Note that the normal, non-crisis, behavior has not changed significantly: 
> error message are sent, cached and reported to the application.  However, in a 
> crisis, the error reporting degrades gracefully, while the throughput goes 
> flat, without any negative slope.  This is where SIP (and all other protocols) 
> should strive to be.
> 
> Treating all errors as soft should have been an Internet Architectural 
> Principle.

These are certainly all good design considerations to take into account.

Thanks,

Volker

_______________________________________________
Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Matt Mathis <[email-masked]>
Date: Thu, 22 May 2008 14:29:13 -0400 (EDT)

I think I need a SIP tutorial - there are things in your message that I 
do not understand.

On Thu, 15 May 2008, Volker Hilt wrote:

> ... We know that with the current SIP protocol, simply
> to congestion collapse.

This really scares me - This statement implies that if I have an uncongested 
but lossy network and a constant call volume, then increasing the loss rate 
will increases the network load by more than the loss rate.  Is this true? 
If so SIP is fundamentally unstable and will be prone to congestion collapse 
in all sorts of unexpected ways.  Other protocols have not been permitted to 
do such things because they are so blatantly risky.

Additional protocol to fix one form of this problem is just a band-aid - I 
strongly suggest looking at the base protocol and figuring out why it 
amplifies the effects of losses.  Is this behavior really a requirement for 
the protocol or is it simply over aggressive retransmission?

If you find a way to fix this, a whole lot of things will just get better.

I think the rest of your message follows the above remarks, so I will deffer 
further questions until I understand these better.

I hope we are making progress,

Thanks,
--MM--

---

From: Henry Sinnreich <[email-masked]>
Date: Thu, 22 May 2008 16:41:18 -0500

> I think I need a SIP tutorial

More than that, you need to _use_ SIP as well.
We are having _only_ SIP VoIP in our home for years and its works just as
good as the PSTN. So why not replace such statements coming out of lack of
personal experience with just using SIP on a daily basis?

Never mind that SIP is a signaling protocol and voice packets run over
RTP...

Henry

---

From: Matt Mathis <[email-masked]>
Date: Thu, 22 May 2008 19:58:13 -0400 (EDT)

I have no doubt that SIP works a lot better then POTS.  I have used it on 
occasion.  But I was also around at the time of the SS7 meltdown in North 
America due to congestive collapse caused by regenerative overload errors (mid 
70s I think). We are not talking about normal operations, we are talking about 
when something has already gone wrong and you are are suddenly critically 
dependent on the correctness of relatively lightly tested congestion behavior.

If congestion is regenerative, such that losses cause increasing presented 
load, the system is intrinsically unstable and prone to congestion collapse. 
If you add a control system to prevent congestion, you are ok as long as the 
causes of the congestion are consistent with the assumptions of the control 
system.  If the control system is well designed this might be just fine, but 
proving that the control system is robust requires proving that it works for 
all types and causes of congestion, including events that are out side of 
standard behavior (e.g. bugs and denial-of-service attacks).   Otherwise SIP
has the potential to magnify the harm caused by the non-standard behavior.

On the other hand, if congestion is not regenerative, the system can be 
designed to be intrinsically stable, as are TCP and other transport protocols. 
Then it does not matter what caused the congestion, the system will do the 
best it can in a crisis.  It might be improved with a control system (for 
example to explicitly allocate scarce resources), but one is not required to 
assure stability, avoid congestive collapse, and assure continued "forward 
progress", even in a crisis.

When the stability comes from the intrinsic behavior of the base protocol, it 
is much easier and more convincing to make the case that it will be well 
behaved in the presence of congestion caused by arbitrary events.

Can somebody explain why SIP congestion is regenerative?  (e.g. why losses 
cause increasing load?)
Can somebody explain why it has to be this way?

(The control plane/data plane discussion is really tangential, since SIP does 
not make any specific assumptions about traffic isolation between the control 
and data planes).

Thanks,
--MM--
-------------------------------------------
Matt Mathis     http://staff.psc.edu/mathis
Work:412.268.3319    Home/Cell:412.654.7529
-------------------------------------------
Evil is defined by mortals who think they know
"The Truth" and use force to apply it to others.

---

From: Henry Sinnreich <[email-masked]>
Date: Thu, 22 May 2008 19:20:39 -0500

If I remember well, the SS7 meltdown was actually due to operator error:
Someone made typos while provisioning something.

Now p2p on the contrary gets better the larger the p2p network is, except
that to my understanding the recent Skype outage was also due to some
misconfiguration. 

The SIP Torture Tests are designed to check the robustness of SIP
implementations.

ftp://ftp.rfc-editor.org/in-notes/rfc4475.txt

Given that no protocol is 100% proof against software faults, maybe we
should get back to the simple notion of SIP overload. To my understanding,
this has not been a problem to some of the popular SIP servers I happen to
know.

I have taken the liberty to copy their developers, in case they would like
to comment from deployment experience.

Henry

---

From: Henry Sinnreich <[email-masked]>
Date: Thu, 22 May 2008 19:23:19 -0500

Henry

---

From: Matt Mathis <[email-masked]>
Date: Thu, 22 May 2008 21:21:26 -0400 (EDT)

> Someone made typos while provisioning something.

Yes that was the trigger, but I believe the congestion did not subside until 
the presented load fell below some small fraction of the capacity, even though 
the typo had been long corrected.

SIP is exposed to exactly this sort of failure if there is any state where 
increased losses cause increased presented load.

It may be that it can't be fixed in this version of SIP, but as a community we 
really need to understand what protocol details cause this behavior, and how 
they might be eliminated in a future version.

If I were to guess, the exponential timers to not back off steeply enough to 
offset the multiplicative participation of additional nodes.  It may be that 
something as simple as tweaking the timer scaling functions would be 
sufficient to prevent congestion collapse.

Thanks,
--MM--

---

From: "DOLLY, MARTIN C, ATTLABS" <[email-masked]>
Date: Thu, 22 May 2008 21:24:14 -0400

You both have understanding of neither.... 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Matt Mathis
Sent: Thursday, May 22, 2008 9:21 PM
To: Henry Sinnreich
Cc: Volker Hilt; Adrian Georgescu; Adam Roach; [email-masked]
Subject: Re: [Sipping] Last Call: draft-ietf-sipping-overload-reqs

> Someone made typos while provisioning something.

Yes that was the trigger, but I believe the congestion did not subside
until 
the presented load fell below some small fraction of the capacity, even
though 
the typo had been long corrected.

SIP is exposed to exactly this sort of failure if there is any state
where 
increased losses cause increased presented load.

It may be that it can't be fixed in this version of SIP, but as a
community we 
really need to understand what protocol details cause this behavior, and
how 
they might be eliminated in a future version.

If I were to guess, the exponential timers to not back off steeply
enough to 
offset the multiplicative participation of additional nodes.  It may be
that 
something as simple as tweaking the timer scaling functions would be 
sufficient to prevent congestion collapse.

Thanks,
--MM--

---

From: "Dwight, Timothy M (Tim)" <[email-masked]>
Date: Fri, 23 May 2008 06:32:26 +0000

There was a period in the early 1990's when, over a period of about 2
weeks, the networks of several North American operators experienced
overload on a daily basis, during busy hour.  Is that the "SS7 meltdown"
of which we're speaking?

If so, the cause wasn't actually provisioning, it was a couple of bugs
in the STPs of one supplier whose product was used in all the affected
networks; the net effect of which was to disable the MTP3 "load
shedding" function.  The result was as Matt recalls below - persistent,
wide-spread congestion until the offerred load abated.  This isn't
because SS7 lacks robust congestion management capabilities - a couple
of decades of operational experience argue to the contrary.  It was
because an implementation happened to get broken in a very unfortunate
way.

I don't think this episode sheds any light on the issue of whether SIP
is or is not or can or cannot be made to be, robust in the face of
congestion.

Tim

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Matt Mathis
> Sent: Thursday, May 22, 2008 8:21 PM
> To: Henry Sinnreich

---

From: Henry Sinnreich <[email-masked]>
Date: Fri, 23 May 2008 04:05:01 -0500

Timothy Dwight wrote:

>it was a couple of bugs
>This isn't because SS7 lacks robust congestion management capabilities

Oh, the good old SS7.
This could make an entertaining discussion during a break, if we meet at the
72 IETF :-)

Henry

---

From: Volker Hilt <[email-masked]>
Date: Fri, 23 May 2008 09:06:48 -0400

Matt,

I fully agree that it is important to fix the problem and not just to 
add a band aid.

For an INVITE transaction to be successful at minimum three messages 
need to be processed, often five or more. Once you start randomly 
discarding messages out of these transaction, the chances that a 
transaction is successful will drop and therefore your goodput will drop 
even if the message throughput is still up.

To your point of too aggressive timers - even if most retransmission 
timers are disabled, the goodput will still eventually approach zero. 
E.g., if the SIP stack is run in TCP mode (for the shake of the argument 
I'm assuming TCP is not used for pushback) the goodput will approach 
zero, even though, the drop is more gradual as there are no retransmissions.

In other words, I don't see how simply changing the retransmission 
timers is going to fix this.

Volker

Matt Mathis wrote:
> This really scares me - This statement implies that if I have an 
> uncongested but lossy network and a constant call volume, then 
> increasing the loss rate will increases the network load by more than 
> the loss rate.  Is this true? If so SIP is fundamentally unstable and 
> will be prone to congestion collapse in all sorts of unexpected ways.  
> Other protocols have not been permitted to do such things because they 
> are so blatantly risky.
> 
> Additional protocol to fix one form of this problem is just a band-aid - 
> I strongly suggest looking at the base protocol and figuring out why it 
> amplifies the effects of losses.  Is this behavior really a requirement 
> for the protocol or is it simply over aggressive retransmission?
> 
> I think the rest of your message follows the above remarks, so I will 
> deffer further questions until I understand these better.
>

---

From: "DOLLY, MARTIN C, ATTLABS" <[email-masked]>
Date: Fri, 23 May 2008 09:36:36 -0400

Henry,

If the SIP torture tests were so good, then why is every vendor
implementation so non-compliant in one area or another. If they were all
non-compliant in the same area that would be great, but sadly not.

And it was bad code that caused the SS7 failures. 

Lastly, those who do not learn from the failures of the past will repeat
them. 

Martin 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Henry Sinnreich
Sent: Thursday, May 22, 2008 8:23 PM
To: Matt Mathis
Cc: Volker Hilt; Adrian Georgescu; Adam Roach; [email-masked]
Subject: Re: [Sipping] Last Call: draft-ietf-sipping-overload-reqs

Now p2p on the contrary gets better the larger the p2p network is,
except
that to my understanding the recent Skype outage was also due to some
misconfiguration. 

Given that no protocol is 100% proof against software faults, maybe we
should get back to the simple notion of SIP overload. To my
understanding,
this has not been a problem to some of the popular SIP servers I happen
to
know.

I have taken the liberty to copy their developers, in case they would
like
to comment from deployment experience.

Henry

> not make any specific assumptions about traffic isolation between the
control
uncongested

---

From: Henry Sinnreich <[email-masked]>
Date: Fri, 23 May 2008 12:13:27 -0500

Martin,

> And it was bad code that caused the SS7 failures.
Truth be told that there was also elsewhere a similar big Frame Relay data
network failure that lasted for about 10 days. It was a software failure as
well.

> Lastly, those who do not learn from the failures of the past will repeat
Fully agree

> implementation so non-compliant in one area or another.
The SIP Torture tests are at least a good start, if everyone would comply.
Beyond this, there are commercial products with 1000's of automatically
generated test cases. Of course, vendors would have to spend the resources
to use them in the first place.

I believe our discussion here is not off-topic, since it puts the SIP
overload into some perspective. SIP overload protection is a useful tool as
well, but not obviously a complete insurance policy. There is none IMO
though sound design (simplicity, loose coupling, open code that can be
audited, etc.) and testing principles go a long way to building stable
networks. These principles have served the Internet very well.

Henry

---

From: [email-masked]
Date: Fri, 23 May 2008 16:03:31 -0400

From: Matt Mathis <[email-masked]>

IMHO, the important question at this point is to have someone who is
thoroughly versed in these concepts review SIP and determine whether
SIP congestion is indeed regenerative.  If it is, then a critical
requirement (within this draft) is adding mechanisms to ensure SIP
congestion is not regenerative.

(Perhaps these questions have been answered already, but I don't think
I've seen that on this list.)

This may require more work by an outside expert.

I respectfully request the chairs to address this matter.

Dale

---

From: "Mary Barnes" <[email-masked]>
Date: Fri, 23 May 2008 15:32:34 -0500

Matt Mathis is the outside expert to whom you refer.  Matt is from the
TSV area and was asked specifically by the TSV ADs to review this
specific document, thus all the comments.  There are also overload
design team members that have years (i.e., decades) of experience
dealing with overload in telecommunication systems.  So, I don't think
we're lacking in experts.  And, I'm wondering if you missed that Volker
did respond to Matt on this topic:
http://www.ietf.org/mail-archive/web/sipping/current/msg15691.html
The thread did get slightly off topic (IMHO), but I think the folks that
should be communicating are.

And, I think the concerns Matt raised are being resolved given
Jonathan's update to the document and Matt's subsequent response:  
http://www.ietf.org/mail-archive/web/sipping/current/msg15675.html

Mary.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, May 23, 2008 3:04 PM
To: [email-masked]; [email-masked]; Barnes, Mary
(RICH2:AR00)

   Can somebody explain why SIP congestion is regenerative?  (e.g. why
losses 
   cause increasing load?)
   Can somebody explain why it has to be this way?

IMHO, the important question at this point is to have someone who is
thoroughly versed in these concepts review SIP and determine whether SIP
congestion is indeed regenerative.  If it is, then a critical
requirement (within this draft) is adding mechanisms to ensure SIP
congestion is not regenerative.

Dale

---

From: "Bharrat, Shaun" <[email-masked]>
Date: Fri, 23 May 2008 20:36:14 -0400

This may be but the question about whether SIP congestion is
regenerative was raised subsequent to this. I agree with Dale that it
is important we get a definitive expert opinion as to whether
this is in fact true for SIP, and, if so, if there are simple
tweaks (retransmission schedule?) that can avoid this.

(and if this question has already been answered, can someone please
repost to the mailing list)

Thanks and Cheers,
Shaun

-----Original Message-----
From: [email-masked] on behalf of Mary Barnes
Sent: Fri 5/23/2008 4:32 PM
To: [email-masked]; [email-masked]; [email-masked]
Cc: Volker Hilt
Subject: Re: [Sipping] Last Call: draft-ietf-sipping-overload-reqs

Mary.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, May 23, 2008 3:04 PM
To: [email-masked]; [email-masked]; Barnes, Mary
(RICH2:AR00)

Dale

---

From: [email-masked]
Date: Sat, 24 May 2008 23:01:58 -0400

From: "Mary Barnes" <[email-masked]>

   So, I don't think we're lacking in experts.

No, I have no worry that we do not have experts.  I'm concerned
whether a particular problem has been suitably examined.

   And, I'm wondering if you missed that Volker did respond to Matt on
   this topic:
   http://www.ietf.org/mail-archive/web/sipping/current/msg15691.html
   The thread did get slightly off topic (IMHO), but I think the folks
   that should be communicating are.

I saw the response, but in regard to the specific question

> This statement implies that if I have an 

I haven't seen what I take to be a specific affirmation or denial as
to whether baseline SIP is "fundamentally unstable" in this way.
(Although it is possible that I have not fully understood the
implications of what people have said subsequently.)  And if SIP has
this problem, are Matt et al. satisfied that the requirements in the
I-D are sufficient to remedy the consequences of this "fundamental
instability"?  The wording of msg15677.html suggests that Matt had not
considered this possibility before reading the I-D; if so, he had not
analyzed the I-D as a remedy for fundamental instability.

These two requirements seem to be relevant, but I'm not sure if they
cover all implications "fundamental instability":

   REQ 1:  The overload mechanism shall strive to maintain the overall
      useful throughput (taking into consideration the quality-of-
      service needs of the using applications) of a SIP server at
      reasonable levels even when the incoming load on the network is
      far in excess of its capacity.  The overall throughput under load
      is the ultimate measure of the value of an overload control
      mechanism.

   REQ 21:  The overload mechanism should ensure that the system remains
      stable.  When the offered load drops from above the overall
      capacity of the network to below the overall capacity, the
      throughput should stabilize and become equal to the offered load.

Dale

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sun, 25 May 2008 07:53:04 -0400

I think Volker and Matt, you guys are talking about different metrics here.

I think Matt is not concerned about goodput, but the question is about 
load. The question is, if there are a total of L calls per second 
constantly arriving into the network which causes a load of M SIP 
messages per second overall through the network under no packet loss, 
and a packet loss of r messages per second is introduced, does the total 
number of messages per second increase above M?

That is different from the question of goodput, which is maintaining a 
rate of successfully completed calls even in the face of failure.

Matt, am I understanding your questions correctly?

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                   499 Thornall St.
Cisco Fellow                                   Edison, NJ 08837
Cisco, Voice Technology Group
[email-masked]
http://www.jdrosen.net                         PHONE: (408) 902-3084
http://www.cisco.com
_______________________________________________
Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Matt Mathis <[email-masked]>
Date: Sun, 25 May 2008 10:29:31 -0400 (EDT)

(was Re: [Sipping] Last Call: draft-ietf-sipping-overload-reqs)

On Sun, 25 May 2008, Jonathan Rosenberg wrote:

> I think Matt is not concerned about goodput, but the question is about load. 
> The question is, if there are a total of L calls per second constantly 
> arriving into the network which causes a load of M SIP messages per second 
> overall through the network under no packet loss, and a packet loss of r 
> messages per second is introduced, does the total number of messages per 
> second increase above M?

John is correctly interpreting my words, however, I have since been 
reeducated that the problem is CPU at the receiver and not link load.   This 
does not change the nature of the failure, but it might make a fix much more 
difficult.

John is also correct that an explicit mechanism (e.g. a flag in messages akin 
to TCP ECN) is likely to be more efficient than an implicit mechanism (e.g. 
tossing messages when overloaded).  My point in some of the earlier email, is 
that if the core congestive behavior is regenerative, then any explicit 
mechanism is fragile in the presence of bugs or unanticipated events. 
Furthermore there has been some real life experience with systemic failures of 
this sort in prior generations of telecommunications gear.

Therefor it is appropriate to pursue both an efficient explicit mechanism, and 
to reduce the extent to which SIP congestion is regenerative.  I believe 
that if SIP congestion can be made to be non-regenerative, you get the
extremely robust implicit mechanism for free.

BTW: The graphs shown at TSV showed the unmistakable signature of regenerative 
congestion collapse.  When loaded to 99% of capacity, goodput was 99% of 
capacity, push the load up by another 1% and the goodput crashes to near zero. 
I'm guessing from the slope of the fall that 503 errors are somehow twice as 
expensive than completing calls.  If this behavior had been observed when 3261 
was in draft, it probably would not have passed TSV review.

So the next question is why (or what are all the causes) of regenerative 
congestion in SIP?   This requires SIP expertise.....

Here is a simple experiment for whom ever ran the simulations shown in TSV:
If the data is collected as a single run by ramping the load, what happens if 
the load ramps up to collapse, and then is ramped down until the collapse 
subsides?  At what load does it subside?  If you tinker with details of 
the protocol and topology, which details affect the load to subside?
These give direct clues as to what parts of the protocol are causing the 
regeneration.  I am particularly curious to know what happens if you just don't 
send 503 errors at all and rely on timers instead.

Thanks,
--MM--

---

From: [email-masked]
Date: Sun, 25 May 2008 23:08:13 -0400

Therefor it is appropriate [...] to reduce the extent to which SIP
   congestion is regenerative.  I believe that if SIP congestion can
   be made to be non-regenerative, you get the extremely robust
   implicit mechanism for free.

This is the point I want to see incorporated into the requirements.

Dale

---

From: Matt Mathis <[email-masked]>
Date: Mon, 26 May 2008 08:13:11 -0400 (EDT)

> This is the point I want to see incorporated into the requirements.

This is a good goal, but it might require draconian changes to 3261.  How much 
are we (you) willing to bite off?

Thanks,
--MM--

---

From: [email-masked]
Date: Mon, 26 May 2008 10:14:27 -0400

> This is the point I want to see incorporated into the requirements.

   This is a good goal, but it might require draconian changes to
   3261.  How much are we (you) willing to bite off?

My understanding of your messages is that this is nearly mandatory if
we're going to make SIP stable under overload.

Dale

---

From: Volker Hilt <[email-masked]>
Date: Mon, 26 May 2008 23:40:53 -0400

Matt Mathis wrote:
> John is also correct that an explicit mechanism (e.g. a flag in messages 
> akin to TCP ECN) is likely to be more efficient than an implicit 
> mechanism (e.g. tossing messages when overloaded).  My point in some of 
> the earlier email, is that if the core congestive behavior is 
> regenerative, then any explicit mechanism is fragile in the presence of 
> bugs or unanticipated events. Furthermore there has been some real life 
> experience with systemic failures of this sort in prior generations of 
> telecommunications gear.
> 
> Therefor it is appropriate to pursue both an efficient explicit 
> mechanism, and to reduce the extent to which SIP congestion is 
> regenerative.  I believe that if SIP congestion can be made to be 
> non-regenerative, you get the
That sounds like a good goal to go after.

> BTW: The graphs shown at TSV showed the unmistakable signature of 
> regenerative congestion collapse.  When loaded to 99% of capacity, 
> goodput was 99% of capacity, push the load up by another 1% and the 
> goodput crashes to near zero. I'm guessing from the slope of the fall 
> that 503 errors are somehow twice as expensive than completing calls.  
> If this behavior had been observed when 3261 was in draft, it probably 
> would not have passed TSV review.
> 
The TSVAREA presentation had two graphs for 503 responses: without 
request retries (scenario 2) and with request retries (scenario 3). The 
poor performance shown for 503 responses without retries is actually 
caused by the control algorithm. These simulations were based on a 
watermark-based control algorithm, which does not perform well.

We did some more experiments with other control algorithms and found 
that it is possible to get a linear degradation of goodput with 503 
responses. Thus, a collapse is not inherent to the 503 response 
mechanism. However, the goodput will eventually reach zero at the point 
where a proxy is completely occupied rejecting requests and no requests 
get processed. In our experiments, these control algorithms could not 
prevent a collapse when 503 retries were used.

> If the data is collected as a single run by ramping the load, what 
> happens if the load ramps up to collapse, and then is ramped down until 
> the collapse subsides?  At what load does it subside?  If you tinker 
> with details of the protocol and topology, which details affect the load 
> to subside?
> regeneration.  I am particularly curious to know what happens if you 
> just don't send 503 errors at all and rely on timers instead.
> 
We have done some experiments on this scenario. Here is what we found: 
assuming SIP runs over UDP (i.e., retransmission timers are enabled) and 
503 responses are not used, then goodput will collapse as we ramp up 
load beyond the capacity. This collapse persists if the load is lowered 
back below the capacity limit and only resolves if the the load is 
lowered further to some fraction of the original capacity. A main 
contributor to this behavior are retransmissions generated for the 
dropped requests.

Is this the behavior you are worried about? I believe this where 
modifications to the retransmission timers can help to improve SIP 
performance and robustness under overload.

Even if no retransmissions are generated, the goodput of a SIP proxy 
(i.e. the number of calls established) will still degrade and reach zero 
under overload. I think that it is important to avoid this behavior and 
ensure that the goodput stays at the capacity limit during periods of 
overload.

Thanks,

Volker

---

From: Matt Mathis <[email-masked]>
Date: Tue, 27 May 2008 10:57:17 -0400 (EDT)

> we're going to make SIP stable under overload.

I may be overly paranoid: explicit mechanisms can work well, especially under 
the conditions they are designed and tested for: too much presented load. 
They may also work well under unanticipated conditions such as bugs, but this 
is really hard to predict.

As it stands SIP can not use the same mechanism used to protect TCP from a SYN 
flood.  In TCP, if a request is for a new connection (a SYN) arrives and the 
queue of unprocessed SYN's is too long, the message is simply discarded.  TCP 
retries later with an exponential backoff.

This does not naturally migrate to SIP, because the expensive step is parsing 
the message.  As I understand it, by the time you get to the point where you 
know that a message is a new request, you have already expended most of the 
effort needed to process it, so discarding the message is an insignificant 
savings over processing it.  (And worse things happen if you discard messages 
without parsing them because then you loose replies to your own requests, 
that you have already committed resources to and are then going to stall). 
As long as the cost of deciding if a message can be tossed is a large fraction 
of the cost to actually process it, the best you can hope for is steeply 
declining goodput under increasing overload.

On the other hand, for example, if there was a fast way to peak into incoming 
messages to identify new requests and put them into a separate queue, you 
might then give priority to requests that already have committed resources, 
and cheaply shed excessive new requests before committing any significant 
resources to them. This leads to nearly flat goodput under overload (declining 
only by the cost of peeking into messages that will ultimately be discarded).

My goal in reviewing -overload-reqs- was to make sure that the implicit 
solution is within scope.  The WG is going to have to decide if it has the 
chops to mandate potentially incompatible changes to the base protocol.  It 
seemed like it would be unnecessarily contentions for -overload-reqs- to 
mandate a robust implicit solution, if it might not interoperate with existing 
SIP implementations.

I hope this helps,
THanks,
--MM--

---

From: [email-masked]
Date: Wed, 28 May 2008 16:33:08 -0400

As it stands SIP can not use the same mechanism used to protect TCP
   from a SYN flood.  In TCP, if a request is for a new connection (a
   SYN) arrives and the queue of unprocessed SYN's is too long, the
   message is simply discarded.  TCP retries later with an exponential
   backoff.

   This does not naturally migrate to SIP, because the expensive step
   is parsing the message.  As I understand it, by the time you get to
   the point where you know that a message is a new request, you have
   already expended most of the effort needed to process it, so
   discarding the message is an insignificant savings over processing
   it.

   On the other hand, for example, if there was a fast way to peak
   into incoming messages to identify new requests and put them into a
   separate queue, you might then give priority to requests that
   already have committed resources, and cheaply shed excessive new
   requests before committing any significant resources to them. This
   leads to nearly flat goodput under overload (declining only by the
   cost of peeking into messages that will ultimately be discarded).

I do not believe that the current SIP intrinsically makes "cheap
discarding" impossible.  The key would be to extract the necessary
information for the discard decision without completely parsing the
message.  I think there are 3 actions needed:

- Extracting the SIP messages from the transport stream.  This is
  trivial for UDP and not so difficult for TCP (because the
  Content-Length header can be found with a finite-state machine).

- Discerning out-of-dialog requests.  This can be done by checking for
  the presence of a to-tag, which can be done with a finite-state
  machine.

- Discriminating requests from responses.  This can also be done with
  a finite-state machine.

In addition, a FSM can also isolate the Call-Id and CSeq headers, and
the to- and from-tags, which can then be used to look up in a hash
table whether the request/response is part of a transaction that the
agent is aware of.

All of the above are order(size of SIP message).

   My goal in reviewing -overload-reqs- was to make sure that the
   implicit solution is within scope.  The WG is going to have to
   decide if it has the chops to mandate potentially incompatible
   changes to the base protocol.  It seemed like it would be
   unnecessarily contentions for -overload-reqs- to mandate a robust
   implicit solution, if it might not interoperate with existing SIP
   implementations.

   I hope this helps,

Yes, it clarifies things greatly.

My goal was to ensure that this issue was *considered* as thoroughly
as necessary, whether or not we decided to take any action on it.

Dale