Subject: [Wish] How to detect a crashed client?

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 09 Sep 2021 17:18:35 +0200

Hi,

When a client terminates correctly, the server can detect that it has left
either when it receives DELETE on the resource URL, or because the client
closes DTLS.

When a client crashes, on the other hand, it just stops sending RTP.  At
what point can the server get rid of the client and destroy (now frozen)
video?

-- Juliusz

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 9 Sep 2021 17:30:24 +0200

As stated in the draft:

   Once a session is setup ICE consent freshness [RFC7675
<https://datatracker.ietf.org/doc/html/rfc7675>] will be used
   to detect abrupt disconnection and DTLS teardown for session
   termination by either side.

Which means that the server has to monitor the ICE requests and
timeout any client which hasn't issued a new ICE request to renovate
the consent in 30s.

Best regards

Sergio

El jue, 9 sept 2021 a las 17:18, Juliusz Chroboczek (<[email-masked]>)
escribió:

> --
> Wish mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/wish
>

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 09 Sep 2021 18:10:12 +0200

> Which means that the server has to monitor the ICE requests and timeout
> any client which hasn't issued a new ICE request to renovate the consent
> in 30s.

So we're assuming that the client has a shorter ICE timeout than the
server, right?  Otherwise the client won't get a chance to send an ICE
restart.

Should that be mentioned in the document?

-- Juliusz

---

From: Nils Ohlmeier <[email-masked]>
Date: Thu, 9 Sep 2021 10:52:30 -0700

> On Sep 9, 2021, at 09:10, Juliusz Chroboczek <[email-masked]> wrote:
> 

[With my individual contributor hat on]

I think Sergios description above is actually not fully correct.

With ICE consent the client and the server both run their own independent ICE consent checks.
So the client sends consent requests and server answers them. And the server sends consent requests which the client has to answer.
If the client crashes, the servers consent checks will get no responses and then after 30 seconds can delete the PeerConnection.

But yes if the server then would delete the PeerConnection right after consent timed out I assume it would make it hard to execute an ICE restart if the client requests that N seconds later. This is different from the browser case, where the ICE consent timeout only changes the ICE connection state, but does not directly affect the lifetime of the PeerConnection itself.

I would suggest that instead of messing with the timers for ICE consent on the two different sides of the connection - BTW for browser endpoints the JS client can’t change the ICE consent timeout value - the spec should mandate that server endpoints need to keep the PeerConnection around for another N seconds after ICE consent timed out.

Best
  Nils Ohlmeier

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 9 Sep 2021 22:10:34 +0200

El jue, 9 sept 2021 a las 19:52, Nils Ohlmeier (<[email-masked]>)
escribió:

> So the client sends consent requests and server answers them. And the
> server sends consent requests which the client has to answer.
> If the client crashes, the servers consent checks will get no responses
> and then after 30 seconds can delete the PeerConnection.
>

That's true in case of full ice servers, but servers may be ice lite which
shouldn't send ice requests.

>
> But yes if the server then would delete the PeerConnection right after
> consent timed out I assume it would make it hard to execute an ICE restart
> if the client requests that N seconds later. This is different from the
> browser case, where the ICE consent timeout only changes the ICE connection
> state, but does not directly affect the lifetime of the PeerConnection
> itself.
>
> I would suggest that instead of messing with the timers for ICE consent on
> the two different sides of the connection - BTW for browser endpoints the
> JS client can’t change the ICE consent timeout value - the spec should
> mandate that server endpoints need to keep the PeerConnection around for
> another N seconds after ICE consent timed out.
>

I am not sure if we should mandate it in the spec, setting a timeout grace
period once the ice timer has expired could be implementation specific. The
client has no way of knowing the exact timeout on the server side, so it
would have the same effect to raise the timeout server side, or have a
grace period deleting the session server side.

Also, in case of ice restart the behaviour would be the same, the whip
client would have to try an ice restart, and if the session is not found,
do a full restart (based reconnection policies on client side).

IMHO, doing an ice restart after 30s or 40s of disconnection does not offer
any benefit over full restart, but others may not think alike.

Best regards
Sergio

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 09 Sep 2021 22:58:23 +0200

> period once the ice timer has expired could be implementation specific.

Fine with me, I'm happy with just some vague implementation guidance in
the draft.

In the meantime: what do you want me to do in my server?  Please advise.

-- Juliusz

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 10 Sep 2021 09:17:35 +0200

> In the meantime: what do you want me to do in my server?  Please advise.
Surprise us! 😂

I think in terms of the hackhaton it is better to just implement what fits
best for you and get feedback on the interoperability issues it may cause
and solve then later in the spec.

Best regards
Sergio

---

From: Nils Ohlmeier <[email-masked]>
Date: Fri, 10 Sep 2021 13:42:05 -0700

> On Sep 9, 2021, at 13:10, Sergio Garcia Murillo <[email-masked]> wrote:
> El jue, 9 sept 2021 a las 19:52, Nils Ohlmeier (<[email-masked] <mailto:[email-masked]>>) escribió:
> That's true in case of full ice servers, but servers may be ice lite which shouldn't send ice requests.

Good point. Although I want to highlight for folks not that familiar with ICE specs that technically nothing forbids an implementer from claiming his server is an ice-lite implementation, but then to send out consent requests once the connection is established. I would argue that this still makes your implementation considerably easier compared to a full ICE stack.

> I am not sure if we should mandate it in the spec, setting a timeout grace period once the ice timer has expired could be implementation specific. The client has no way of knowing the exact timeout on the server side, so it would have the same effect to raise the timeout server side, or have a grace period deleting the session server side.
> 
> Also, in case of ice restart the behaviour would be the same, the whip client would have to try an ice restart, and if the session is not found, do a full restart (based reconnection policies on client side).
> 
> IMHO, doing an ice restart after 30s or 40s of disconnection does not offer any benefit over full restart, but others may not think alike. 

I fully agree on the 30s here.
So maybe the easiest is then to let the client side decide when to give up with ICE consent. And it needs to be prepared to handle a 404 on it’s ICE restart request, to then do a full restart of the connection.

Best
  Nils

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 10 Sep 2021 22:53:18 +0200

El vie., 10 sept. 2021 22:42, Nils Ohlmeier <[email-masked]>
escribió:

> Good point. Although I want to highlight for folks not that familiar with
> ICE specs that technically nothing forbids an implementer from claiming his
> server is an ice-lite implementation, but then to send out consent requests
> once the connection is established. I would argue that this still makes
> your implementation considerably easier compared to a full ICE stack.
>

that's exactly what I have implemented in my media server, I call it ICE
lite+ 🤣

>> I would suggest that instead of messing with the timers for ICE consent
>> on the two different sides of the connection - BTW for browser endpoints
>> the JS client can’t change the ICE consent timeout value - the spec should
> IMHO, doing an ice restart after 30s or 40s of disconnection does not
> offer any benefit over full restart, but others may not think alike.
>
>
> So maybe the easiest is then to let the client side decide when to give up
> with ICE consent. And it needs to be prepared to handle a 404 on it’s ICE
> restart request, to then do a full restart of the connection.
>

Completely agree.

Best regards
Sergio

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 10 Sep 2021 21:52:59 +0000

Hi,

The main purpose of ICE consent freshness is not to detect crashed endpoints. For that ICE keepalives are enough.

Regards,

Christer

From: Wish <[email-masked]> On Behalf Of Sergio Garcia Murillo
Sent: perjantai 10. syyskuuta 2021 23.53
To: Nils Ohlmeier <[email-masked]>
Cc: WISH List <[email-masked]>; Juliusz Chroboczek <[email-masked]>
Subject: Re: [Wish] How to detect a crashed client?

El vie., 10 sept. 2021 22:42, Nils Ohlmeier <[email-masked]<mailto:[email-masked]>> escribió:

On Sep 9, 2021, at 13:10, Sergio Garcia Murillo <[email-masked]<mailto:[email-masked]>> wrote:
El jue, 9 sept 2021 a las 19:52, Nils Ohlmeier (<[email-masked]<mailto:[email-masked]>>) escribió:
[With my individual contributor hat on]

that's exactly what I have implemented in my media server, I call it ICE lite+ 🤣

Completely agree.

Best regards
Sergio

---

From: Nils Ohlmeier <[email-masked]>
Date: Fri, 10 Sep 2021 15:16:45 -0700

Hi Christer,

> On Sep 10, 2021, at 14:52, Christer Holmberg <[email-masked]> wrote:
> 

While it is true that the purpose of ICE consent is not to detect crashes, it does allow to detect that is no longer useful to waste bandwidth by sending data to an endpoint which isn’t listening any more for what ever reason. Which I guess in case of Wish would mean only the client would implement ICE consent to detect that the server is no longer receiving the data.

But at least in the browser WebRTC world ICE consent simply has replaced ICE keepalives. Mostly because ICE keepalives worn’t working really well. So I would recommend to stay away from ICE keepalives for Wish.

Best
  Nils

> El vie., 10 sept. 2021 22:42, Nils Ohlmeier <[email-masked] <mailto:[email-masked]>> escribió:
>  
> On Sep 9, 2021, at 13:10, Sergio Garcia Murillo <[email-masked] <mailto:[email-masked]>> wrote:

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 11 Sep 2021 09:47:54 +0000

Hi,

> While it is true that the purpose of ICE consent is not to detect crashes, it does allow to detect that is no longer useful to waste bandwidth by sending data to an endpoint which isn’t listening any more for what ever reason. 
> Which I guess in case of Wish would mean only the client would implement ICE consent to detect that the server is no longer receiving the data.
>

Sure, but I just think it needs to be clear that, as far as the standard is concerned, ICE consent is not required for making sure the peer is alive. But, if you are anyway going to do ICE consent, you can of course also use it for check-that-the-peer-is-alive purpose.

However, it was also suggested that the server would use ICE consent towards the client. That sounds a little weird to me, as there will be no media in that direction to begin with.

Regards,

Christer

From: Wish <mailto:[email-masked]> On Behalf Of Sergio Garcia Murillo
Sent: perjantai 10. syyskuuta 2021 23.53
To: Nils Ohlmeier <mailto:[email-masked]>
Cc: WISH List <mailto:[email-masked]>; Juliusz Chroboczek <mailto:[email-masked]>
Subject: Re: [Wish] How to detect a crashed client?

El vie., 10 sept. 2021 22:42, Nils Ohlmeier <mailto:[email-masked]> escribió:
 
On Sep 9, 2021, at 13:10, Sergio Garcia Murillo <mailto:[email-masked]> wrote:
El jue, 9 sept 2021 a las 19:52, Nils Ohlmeier (<mailto:[email-masked]>) escribió:
[With my individual contributor hat on]

 

 

 

 
 
Completely agree.
 
Best regards 
Sergio

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sat, 11 Sep 2021 12:14:15 +0200

> However, it was also suggested that the server would use ICE consent
> towards the client. That sounds a little weird to me, as there will be
> no media in that direction to begin with.

I think that one of the desirable properties of WHIP is that it can easily
be implemented with minimal changes to the WebRTC stack -- in the current
draft, the only place where you need to delve into the internals of the
WebRTC stack is to generate and parse the somewhat baroque RFC 8840 syntax
for trickled ICE candidates.

My undetstanding of Nils' point is that while ICE keepalives are the
natural thing to do in the server->client direction, if your stack is
doing ICE consent instead, then it's fine to keep using that.  Does anyone
see a problem with that?

-- Juliusz

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 11 Sep 2021 10:21:24 +0000

> I think that one of the desirable properties of WHIP is that it can easily be implemented with minimal changes to the WebRTC stack -- in the current draft, the only place where you need to delve into
> the internals of the WebRTC stack is to generate and parse the > somewhat baroque RFC 8840 syntax for trickled ICE candidates.
>
> My undetstanding of Nils' point is that while ICE keepalives are the natural thing to do in the server->client direction, if your stack is doing ICE consent instead, then it's fine to keep using that.  Does anyone see a problem with that?

Even with "normal" bi-directional media cases, how often does a network server use ICE consent towards a client?

Network servers are often (please correct me if I'm wrong) ICE lite, and now there is suggestion that those should do ICE consent. How is that keeping things simple? :)

Regards,

Christer

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sat, 11 Sep 2021 12:42:36 +0200

> Network servers are often (please correct me if I'm wrong) ICE lite,
> and now there is suggestion that those should do ICE consent. How is
> that keeping things simple? :)

I don't think the suggestion is that they SHOULD do ICE consent.  The
suggestion is that they MAY do ICE consent.

-- Juliusz

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 11 Sep 2021 11:29:39 +0000

> I don't think the suggestion is that they SHOULD do ICE consent.  The suggestion is that they MAY do ICE consent.

Sure, but then I don't think we should talk about ICE lite. If we see the need for a new ICE type ("ICE lite++", or whatever we want to call it) I think that needs a broader discussion, beyond WHIP.

Of course, an implementation can do whatever, but in my opinion we should not specify things that are not aligned with the standard - even if it's a MAY.

Also note that ICE consent is sent no matter whether there is media flowing or not. But, if you are only interested in knowing that the client is alive, I assume receiving media from it is enough? So, the server would only need to check whether it's alive or not when it does not receive media.

Regards,

Christer

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sat, 11 Sep 2021 14:20:36 +0200

My original idea was that as clients are mandated to send ice consent, the
server would be able to detect disconnection based on the reception of the
ice request sent by the clients.

ICE lite servers should not be mandated to send ice requests in any way.
ICE lite+ or full ice servers may use the consent sent by the servers
instead.

I don't like the idea of using media for this,  as I have never liked the
webrtc approach of sending black frames when media is paused.

Best regards
Sergio

El sáb., 11 sept. 2021 13:29, Christer Holmberg <
[email-masked]> escribió:

> Sure, but then I don't think we should talk about ICE lite. If we see the
> need for a new ICE type ("ICE lite++", or whatever we want to call it) I
> think that needs a broader discussion, beyond WHIP.
>
> Of course, an implementation can do whatever, but in my opinion we should
> not specify things that are not aligned with the standard - even if it's a
> MAY.
>
> Also note that ICE consent is sent no matter whether there is media
> flowing or not. But, if you are only interested in knowing that the client
> is alive, I assume receiving media from it is enough? So, the server would
> only need to check whether it's alive or not when it does not receive media.
>
> Regards,
>
> Christer
>
>

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 11 Sep 2021 18:22:05 +0000

>My original idea was that as clients are mandated to send ice consent, the server would be able to detect disconnection based on the reception of the ice request sent by the clients.

Correct. I have no issue with that.

I just don’t think it’s correct to say that ICE consent is required in order to detect disconnection.

>ICE lite servers should not be mandated to send ice requests in any way. ICE lite+ or full ice servers may use the consent sent by the servers instead.
>
>I don't like the idea of using media for this,  as I have never liked the webrtc approach of sending black frames when media is paused.

I did not suggest that. What I said was that ICE keepalives are only sent when there is no media, because when media IS sent it can be used as keepalive - while ICE consent is used no matter if there is media or not.

Regards,

Christer

Regards,

Christer

---

From: Adam Roach <[email-masked]>
Date: Sat, 11 Sep 2021 14:32:30 -0500

On 9/11/2021 7:20 AM, Sergio Garcia Murillo wrote:
> I don't like the idea of using media for this,  as I have never liked 
> the webrtc approach of sending black frames when media is paused.

Even if you don't send black frames, you should still be sending RTCP 
Sender Reports, right?

I'm not necessarily advocating for this approach, but it does seem 
perfectly viable.

/a