Subject: [Dots] Behavior when keep-alives fail (RE: Mirja Kühlewind's Discuss on draft-ietf-dots-signal-channel-31: (with DISCUSS and COMMENT)

From: <[email-masked]>
Date: Wed, 3 Jul 2019 07:53:37 +0000

Hi Mirja, 

(Focusing on individual issues) 

Please see inline. 

Cheers,
Med

> -----Message d'origine-----
> De : Mirja Kuehlewind [mailto:[email-masked]]
> Envoyé : mardi 2 juillet 2019 16:00
> À : BOUCADAIR Mohamed TGI/OLN
> Cc : Konda, Tirumaleswar Reddy; Benjamin Kaduk; draft-ietf-dots-signal-
> [email-masked]; [email-masked]; [email-masked]; The IESG;
> [email-masked]
> Objet : Re: [Dots] Mirja Kühlewind's Discuss on draft-ietf-dots-signal-
> channel-31: (with DISCUSS and COMMENT)
> 
...
> >>>>> 10) The document should more explicitly provide more guidance about
> >>>>> when a client should start a session and what should be done (from
> the
> >>>>> client side) if a session is detected as inactive (other than during
> >>>>> migration which is discussed a bit in 4.7). Is the assumption to
> have
> >>>>> basically permanently an active session or connect for migration and
> >>>>> configuration requests separately at a time?
> >>>>
> >>>> I think there was some clarifying text added, but please confirm if
> you
> >> think it
> >>>> is sufficient.
> >>
> >> Sorry, don’t see where text was added. Can you provide a pointer?
> >
> > [Med] We do have this text, for example:
> >
> >   The DOTS signal channel can be established between two DOTS agents
> >   prior or during an attack.  The DOTS signal channel is initiated by
> >   the DOTS client.  The DOTS client can then negotiate, configure, and
> >   retrieve the DOTS signal channel session behavior with its DOTS peer
> >   (Section 4.5).  Once the signal channel is established, the DOTS
> >   agents periodically send heartbeats to keep the channel active
> >   (Section 4.7).  At any time, the DOTS client may send a mitigation
> >   request message (Section 4.4) to a DOTS server over the active signal
> >   channel.  While mitigation is active (because of the higher
> >   likelihood of packet loss during a DDoS attack), the DOTS server
> >   periodically sends status messages to the client, including basic
> >   mitigation feedback details.  Mitigation remains active until the
> >   DOTS client explicitly terminates mitigation, or the mitigation
> >   lifetime expires.  Also, the DOTS server may rely on the signal
> >   channel session loss to trigger mitigation for pre-configured
> >   mitigation requests (if any).
> 
> Okay thanks for for the pointer. What I think is missing are some
> sentences about what the client (or server) should do if the keep-alive
> fails. Try to reconnect directly or just with the next request or
> whatever. Basically who should reconnect and when?

[Med] This is discussed in details in Section 4.7, in particular. 

As a generic rule, it is always the client who connects (see the excerpt above). 

The server may use the failure to initiate automated mitigation (see the excerpt above). More details are provided in other sections. 

There are several heartbeat failure cases to handle by the client. Examples from 4.7 are provided below, fwiw: 

      The DOTS client MUST NOT consider the DOTS signal channel session
      terminated even after a maximum 'missing-hb-allowed' threshold is
      reached.  The DOTS client SHOULD keep on using the current DOTS
      signal channel session to send heartbeat requests over it, so that
      the DOTS server knows the DOTS client has not disconnected the
      DOTS signal channel session.

      After the maximum 'missing-hb-allowed' threshold is reached, the
      DOTS client SHOULD try to resume the (D)TLS session.  The DOTS
      client SHOULD send mitigation requests over the current DOTS
      signal channel session, and in parallel, for example, try to
      resume the (D)TLS session or use 0-RTT mode in DTLS 1.3 to
      piggyback the mitigation request in the ClientHello message.

      As soon as the link is no longer saturated, if traffic from the
      DOTS server reaches the DOTS client over the current DOTS signal
      channel session, the DOTS client can stop (D)TLS session
      resumption or if (D)TLS session resumption is successful then
      disconnect the current DOTS signal channel session.

Do you think additional text is needed?

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 3 Jul 2019 12:25:31 +0200

Hi Med,

See below.

> On 3. Jul 2019, at 09:53, [email-masked] wrote:
> 
> Hi Mirja, 
> 

Actually to my understanding this will not work. Both TCP heartbeat and Coap Ping are transmitted reliably. If you don’t receive an ack for these transmissions you are not able to send any additional messages and can only choose the connection.

Mirja

---

From: <[email-masked]>
Date: Wed, 3 Jul 2019 10:48:16 +0000

Mirja, 

> Actually to my understanding this will not work. Both TCP heartbeat and
> Coap Ping are transmitted reliably. If you don’t receive an ack for these
> transmissions you are not able to send any additional messages and can
> only choose the connection.

This behavior is implemented and tested between two implementations. The exact procedure is described in the draft, fwiw: 

==
   When a Confirmable "CoAP Ping" is sent, and if there is no response,
   the "CoAP Ping" is retransmitted max-retransmit number of times by
   the CoAP layer using an initial timeout set to a random duration
   between ack-timeout and (ack-timeout*ack-random-factor) and
   exponential back-off between retransmissions.  By choosing the
   recommended transmission parameters, the "CoAP Ping" will timeout
   after 45 seconds.  If the DOTS agent does not receive any response
   from the peer DOTS agent for 'missing-hb-allowed' number of
   consecutive "CoAP Ping" Confirmable messages, it concludes that the
   DOTS signal channel session is disconnected.  A DOTS client MUST NOT
   transmit a "CoAP Ping" while waiting for the previous "CoAP Ping"
   response from the same DOTS server.
==

Cheers,
Med

> Envoyé : mercredi 3 juillet 2019 12:26
> Objet : Re: Behavior when keep-alives fail (RE: [Dots] Mirja Kühlewind's
> Discuss on draft-ietf-dots-signal-channel-31: (with DISCUSS and COMMENT)
> 
> Hi Med,
> 
> See below.
> 
> > The server may use the failure to initiate automated mitigation (see the
> excerpt above). More details are provided in other sections.
> >
> > There are several heartbeat failure cases to handle by the client.
> Examples from 4.7 are provided below, fwiw:
> >
> Actually to my understanding this will not work. Both TCP heartbeat and
> Coap Ping are transmitted reliably. If you don’t receive an ack for these
> transmissions you are not able to send any additional messages and can
> only choose the connection.
> 
> Mirja
> 
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 3 Jul 2019 14:46:04 +0200

Hi Med,

See below.

> On 3. Jul 2019, at 12:48, <[email-masked]> <[email-masked]> wrote:
> 
> Mirja, 
> 

First, can you explain why you need 'missing-hb-allowed’? If the ping is transmitted reliably, one “missed” should be enough to conclude that the session is disconnected.

Yes, as Coap Ping is used, the agent should not only conclude that the DOTS signal session is disconnected but also the Coap session and not send any further Coap messages anymore. 

If you want to send further UDP datagram you should it unreliability and not more often then one per 3 seconds. 

Mirja

---

From: <[email-masked]>
Date: Wed, 3 Jul 2019 13:46:04 +0000

Re-,

Please see inline. 

Cheers,
Med

> Envoyé : mercredi 3 juillet 2019 14:46
> > This behavior is implemented and tested between two implementations. The
> exact procedure is described in the draft, fwiw:
> >
> > ==
> First, can you explain why you need 'missing-hb-allowed’?

[Med] because we need to make sure this a "real/durable" session defunct, not a false positive. For example, this would have implications on the server as it may erroneously start automated mitigations (because it concludes the session is lost).

 If the ping is
> transmitted reliably, one “missed” should be enough to conclude that the
> session is disconnected.

[Med] Hmm, under some DDoS attacks, both endpoints may be sending/replying to confirmable ping messages, but the reply may get  dropped. The session is not disconnected in such case.

> 
> Yes, as Coap Ping is used, the agent should not only conclude that the
> DOTS signal session is disconnected but also the Coap session and not send
> any further Coap messages anymore.
> 
> >>> As a generic rule, it is always the client who connects (see the
> excerpt
> >> above).
> >>>

---

From: "Jon Shallow" <[email-masked]>
Date: Wed, 3 Jul 2019 15:04:25 +0100

Hi Mirja,

As an implementer of DOTS I have the following comments to make to try and help understand what is going on with the Heartbeats

In the peace time scenario, it is assumed that the heartbeats function through the network in both directions and that it is possible to disable one or both of the heartbeat directions.

Heartbeats can use UDP or TCP depending on the session set up based on the initial connection (UDP is preferred over TCP).  With UDP, there is a CoAP Ping (Empty request CON 0.00) and a CoAP RST response.  With TCP, there is a Coap Ping (7.02) and a Coap Pong (7.03) response.
For UDP, https://tools.ietf.org/html/rfc7252#section-4.8.2 defines the ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT which map onto the DOTS heartbeat parameters ack-timeout, ack-random-factor and max-retransmit.  These 3 parameters determine the elapsed time when there has been transmission failure.   There is an additional DOTS parameter missing-hb-allowed to support more than one heartbeat loss should it be needed before determining that a DOTS agent has really gone away (instead of, say, going through a reboot or a restart cycle).
 
When handling an attack scenario, there is a good chance that the inbound (DOTS server to DOTS client) data path is flooded /overloaded and hence packet loss (but not the case with all DDoS type scenarios).

A significant purpose of the DOTS client generating a heartbeat is to make sure that any NAT devices in the path maintain their NAT associations and allow any returning responses (which could be unsolicited if the observe of a mitigation is active).

Even in the attack scenario, the DOTS server will see these heartbeat messages, but can only deduce that the connection from the DOTS client to the DOTS server is good - but cannot make any assumptions about traffic flowing in the other direction.

However, the DOTS client may not get a ping response due to the flooded inbound pipe.  If the DOTS client has initiated a mitigation request, then it is unsafe for the DOTS client to close down the session - it will need to refresh the mitigation requests / create new ones even if the mitigation is not being that effective as traffic can still flow to the server.  It is possible that the DOTS server has just restarted - hence the requirement to try and open up a new session in parallel.

If the DOTS server also initiates heartbeat messages, sees the DOTS client pings, but does not see any response to the DOTS server ping, the DOTS server can now deduce that the outbound pipe is good, but the inbound pipe to the DOTS client is failing.  The DOTS server then does not need to close down the session as it will be expecting additional mitigation requests from the DOTS client - even though the DOTS server Coap Ping is failing.

Furthermore, if the DOTS server initiates its CoAP ping on receipt of the DOTS client Coap Ping, then there is a good chance that the NAT sessions are "warm" on any intervening NAT devices.  If the DOTS server initiates the Coap Ping on its own cycle, there is a chance that it may not get through and confuse the logic.

Regards

Jon

> -----Original Message-----
> From: Dots [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: 03 July 2019 14:46
> To: Mirja Kuehlewind
> Cc: [email-masked]; Konda, Tirumaleswar Reddy;
> [email-masked]; [email-masked]; The IESG; [email-masked];
> Benjamin Kaduk
> Subject: Re: [Dots] Behavior when keep-alives fail (RE: Mirja Kühlewind's
> [Med] because we need to make sure this a "real/durable" session defunct,
> not a false positive. For example, this would have implications on the server
> as it may erroneously start automated mitigations (because it concludes the
> session is lost).
> 
>  If the ping is
> [Med] Hmm, under some DDoS attacks, both endpoints may be
> sending/replying to confirmable ping messages, but the reply may get
> dropped. The session is not disconnected in such case.
> 
> >
> > >>>>>>>>> 10) The document should more explicitly provide more
> guidance
> > >> about
> > >>>>>>>>> basically permanently an active session or connect for
> migration
> > >> and
> _______________________________________________
> Dots mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dots

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 3 Jul 2019 18:49:32 +0200

Hi Med,

See below.

> On 3. Jul 2019, at 15:46, <[email-masked]> <[email-masked]> wrote:
> 
> Re-,
> 
> Please see inline. 
> 
> Cheers,
> Med
> 

If the reliable ping fails, this means already that there where X number of reties which exponential backoff. This really should be an indication of a real failure!

> [Med] Hmm, under some DDoS attacks, both endpoints may be sending/replying to confirmable ping messages, but the reply may get  dropped. The session is not disconnected in such case.

For confirmable massage an ack is send. If the ack is dropped, the message will be retransmitted several times (with exponential backoff).

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 3 Jul 2019 19:12:13 +0200

Hi Jon,

Thanks for extended explanation. Please see questions inline.c

> On 3. Jul 2019, at 16:04, Jon Shallow <[email-masked]> wrote:
> 
> Hi Mirja,
> 

Why do you need this additional parameter? Why is max-retransmit not enough? Isn’t the results the same, you send more ping frames before you finally give up?

> Even in the attack scenario, the DOTS server will see these heartbeat messages, but can only deduce that the connection from the DOTS client to the DOTS server is good - but cannot make any assumpt…

However, if the client does never receive a Coap RST or Coap pong, it will sooner or later give up and not send any ping messages anymore. In this case the server will receive no ping anymore and can decide to send own pings. Important is the idle time out is known to both ends.

Further if your really want to be sure if the RST was received or not, I’d recommend you to use an own application ping that indicated if the ping is a retransmission or not. 

Detection one-way congestion is a different function than keep-alive testing and it is better to use an explicit mechanism for that then trying to infer something from a mechanism that was designed for a different purpose.

> Furthermore, if the DOTS server initiates its CoAP ping on receipt of the DOTS client Coap Ping, then there is a good chance that the NAT sessions are "warm" on any intervening NAT devices.  If the…

This also sounds to me that you should rather design your own testing during mitigation in the dots layer, e.g. don’t use the Coap Ping, but send a non-confirmable Coap message which contains a dots layer “ping" and an indication if a dots-layer “pong" has been received or not.

However, note that this still might not work with TCP as messages cannot be transmitted unreliably and not-transmitted/no-acked application layer data will block all other traffic on the same connection at some point because TCP will try to retransmit and shrink the congestion window to the minimum.

Mirja

---

From: <[email-masked]>
Date: Wed, 3 Jul 2019 18:05:10 +0000

Re-,

Please see inline. 

Cheers,
Med

> Envoyé : mercredi 3 juillet 2019 18:50
> Cc : [email-masked]; Konda, Tirumaleswar Reddy;
> > [Med] because we need to make sure this a "real/durable" session
> defunct, not a false positive. For example, this would have implications
> on the server as it may erroneously start automated mitigations (because
> it concludes the session is lost).
> 
> If the reliable ping fails, this means already that there where X number
> of reties which exponential backoff. This really should be an indication
> of a real failure!
> 

[Med] We spent in the WG a lot of cycles and time to discuss this particular point. The conclusion of the WG is that 4 retransmits (that corresponds to the default value of MAX_RETRANSMIT) are not sufficient to detect the session is disconnected during volumetric DDoS attack saturating the incoming link, and which will lead to high packet loss.

This is why multiple heartbeat losses are required to detect the session is defunct.

Please note that 'missing-hb-allowed' is configurable parameter that can be negotiated with the server. Also, the HB procedure can even be deactivated during idle time.

---

From: "Jon Shallow" <[email-masked]>
Date: Wed, 3 Jul 2019 20:15:53 +0100

Hi Mirja,

In one sense Med has answered your queries below separately, but I will try to expand on them from my implementation perspective.

Regards

Jon

> From: Dots [mailto: [email-masked]] On Behalf Of Mirja
> Kuehlewind
> Sent: 03 July 2019 18:12
> To: Jon Shallow
> [email-masked]; Benjamin Kaduk
> > As an implementer of DOTS I have the following comments to make to try
> and help understand what is going on with the Heartbeats
> >
> > In the peace time scenario, it is assumed that the heartbeats function
> through the network in both directions and that it is possible to disable one
> or both of the heartbeat directions.
> >
> > Heartbeats can use UDP or TCP depending on the session set up based on
> the initial connection (UDP is preferred over TCP).  With UDP, there is a CoAP
> Ping (Empty request CON 0.00) and a CoAP RST response.  With TCP, there is
> a Coap Ping (7.02) and a Coap Pong (7.03) response.
> > For UDP, https://tools.ietf.org/html/rfc7252#section-4.8.2 defines the
> ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT which map
> onto the DOTS heartbeat parameters ack-timeout, ack-random-factor and
> max-retransmit.  These 3 parameters determine the elapsed time when
> there has been transmission failure.   There is an additional DOTS parameter
> missing-hb-allowed to support more than one heartbeat loss should it be
> needed before determining that a DOTS agent has really gone away (instead
> of, say, going through a reboot or a restart cycle).
> 
> Why do you need this additional parameter? Why is max-retransmit not
> enough? Isn’t the results the same, you send more ping frames before you
> finally give up?

Well no - we are running in a failing network environment due to the DDoS attacks and need to be as robust as possible before finally giving up.

In RFC7252 ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT have the suggested values of 2, 1.5 and 4 respectively.  Following https://tools.ietf.org/html/rfc7252#section-4.8.2, the time between the first and last retransmit with these values is 45 seconds - which means that the request times out after 93 seconds (MAX_TRANSMIT_WAIT) at the point when the MAX_RETRANSMIT+1 is retried.

This final time of 48 seconds (93sec - 45sec) is bigger than a NAT refresh timeout that we are comfortable with, so the recommended DOTS value  for max-retransmit is 3.  We have added in this additional missing-hb-allowed to increase the timeout before failure is determined.

However, as mentioned below, this determined failure may not be sufficient to cause a session to be closed down as there could be network losses due to the DDoS attacks.

> 
> >
> > When handling an attack scenario, there is a good chance that the inbound
> (DOTS server to DOTS client) data path is flooded /overloaded and hence
> packet loss (but not the case with all DDoS type scenarios).
> >
> > A significant purpose of the DOTS client generating a heartbeat is to make
> sure that any NAT devices in the path maintain their NAT associations and
> allow any returning responses (which could be unsolicited if the observe of a
> mitigation is active).
> >
> > Even in the attack scenario, the DOTS server will see these heartbeat
> messages, but can only deduce that the connection from the DOTS client to
> the DOTS server is good - but cannot make any assumptions about traffic
> flowing in the other direction.
> 
> However, if the client does never receive a Coap RST or Coap pong, it will
> sooner or later give up and not send any ping messages anymore. In this case
> the server will receive no ping anymore and can decide to send own pings.
> Important is the idle time out is known to both ends.

Agreed, but as mentioned below the DOTS client must continue to transmit if there has been a mitigation request (i.e. running in under attack mode and networks could be flaky).

The retransmission parameters are negotiated between the DOTS agents at client session startup ( https://tools.ietf.org/html/draft-ietf-dots-signal-channel-34#section-4.5.2 ) so both ends know what the timeouts are.

> 
> Further if your really want to be sure if the RST was received or not, I’d
> recommend you to use an own application ping that indicated if the ping is a
> retransmission or not.

Well actually, while  the CoAP layer handles the transmission of the Empty/RST, the ping is initiated by the DOTS layer (and is told either a RST or timeout occurred), not the CoAP layer and so the DOTS application is in control of what is going on here.

The Ping is not handled in the same as, say, TCP keep-alive packets which are handled completely by the TCP layer.

> 
> Detection one-way congestion is a different function than keep-alive testing
> and it is better to use an explicit mechanism for that then trying to infer
> something from a mechanism that was designed for a different purpose.
> 

Agreed, but the DOTS layer is triggering the CoAP ping to do this work for the one-way congestion testing.

> >
> > However, the DOTS client may not get a ping response due to the flooded
> inbound pipe.  If the DOTS client has initiated a mitigation request, then it is
> unsafe for the DOTS client to close down the session - it will need to refresh
> the mitigation requests / create new ones even if the mitigation is not being
> that effective as traffic can still flow to the server.  It is possible that the
> DOTS server has just restarted - hence the requirement to try and open up a
> new session in parallel.
> >
> > If the DOTS server also initiates heartbeat messages, sees the DOTS client
> pings, but does not see any response to the DOTS server ping, the DOTS
> server can now deduce that the outbound pipe is good, but the inbound
> pipe to the DOTS client is failing.  The DOTS server then does not need to
> close down the session as it will be expecting additional mitigation requests
> from the DOTS client - even though the DOTS server Coap Ping is failing.
> >
> > Furthermore, if the DOTS server initiates its CoAP ping on receipt of the
> DOTS client Coap Ping, then there is a good chance that the NAT sessions are
> "warm" on any intervening NAT devices.  If the DOTS server initiates the
> Coap Ping on its own cycle, there is a chance that it may not get through and
> confuse the logic.
> 
> This also sounds to me that you should rather design your own testing
> during mitigation in the dots layer, e.g. don’t use the Coap Ping, but send a
> non-confirmable Coap message which contains a dots layer “ping" and an
> indication if a dots-layer “pong" has been received or not.

As stated above, the CoAP ping is not initiated by the CoAP layer like TCP keep-alives, but is triggered by the DOTS application by sending the Coap Ping packet - which in effect is what you are suggesting (apart from the non-confirmable aspect).  If using TCP, then what you describe is correct (except Confirmable/Non-Confirmable are out of the picture).

The CoAP Empty packet must be confirmable to solicit a RST response (Table 1: RFC7252).  I would rather not move away from RFC7252 here.

> 
> However, note that this still might not work with TCP as messages cannot be
> transmitted unreliably and not-transmitted/no-acked application layer data
> will block all other traffic on the same connection at some point because TCP
> will try to retransmit and shrink the congestion window to the minimum.

TCP CoAP Ping/Pong does work as we are initiating it from the DOTS layer.

~Jon

> >>>> This behavior is implemented and tested between two
> implementations.
> >> The
> defunct,
> >>>>>>>>>>> I think there was some clarifying text added, but please
> confirm
> >>> if
> >>>>>>> you
> >>>>>>>>> think it
> >>>>>>>>>>> is sufficient.
> >>>>>>>>>

---

From: "Konda, Tirumaleswar Reddy" <[email-masked]>
Date: Mon, 15 Jul 2019 12:40:53 +0000

Hi Mirja,

We have updated the draft to address your Discuss and comments (https://tools.ietf.org/rfcdiff?url2=draft-ietf-dots-signal-channel-35.txt).  Please have a look and approve the draft. 

Best Regards,
-Tiru

> From: Jon Shallow <[email-masked]>
> Sent: Thursday, July 4, 2019 12:46 AM
> To: [email-masked]; 'Mirja Kuehlewind'
> <[email-masked]>; [email-masked]; Konda,
> Tirumaleswar Reddy <[email-masked]>;
> [email-masked]; [email-masked]; 'The IESG' <[email-masked]>; dots-
> [email-masked]; 'Benjamin Kaduk' <[email-masked]>
> Subject: RE: [Dots] Behavior when keep-alives fail (RE: Mirja Kühlewind's
> This email originated from outside of the organization. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
> 
> Hi Mirja,
> 
> In one sense Med has answered your queries below separately, but I will try
> to expand on them from my implementation perspective.
> 
> Regards
> 
> Jon
> 
> > Cc: [email-masked]; Konda, Tirumaleswar
> > Reddy; [email-masked]; [email-masked]; The IESG;
> > [email-masked]; [email-masked]; Benjamin Kaduk
> > Subject: Re: [Dots] Behavior when keep-alives fail (RE: Mirja
> > Kühlewind's Discuss on draft-ietf-dots-signal-channel-31: (with
> > DISCUSS and COMMENT)
> >
> > Hi Jon,
> >
> > through the network in both directions and that it is possible to
> > disable one or both of the heartbeat directions.
> > >
> > the initial connection (UDP is preferred over TCP).  With UDP, there
> > is a CoAP Ping (Empty request CON 0.00) and a CoAP RST response.  With
> > TCP, there is a Coap Ping (7.02) and a Coap Pong (7.03) response.
> > ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT which
> map onto the
> > DOTS heartbeat parameters ack-timeout, ack-random-factor and
> > missing-hb-allowed to support more than one heartbeat loss should it
> > be needed before determining that a DOTS agent has really gone away
> > (instead of, say, going through a reboot or a restart cycle).
> >
> > enough? Isn’t the results the same, you send more ping frames before
> > you finally give up?
> 
> Well no - we are running in a failing network environment due to the DDoS
> attacks and need to be as robust as possible before finally giving up.
> 
> In RFC7252 ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT
> have the suggested values of 2, 1.5 and 4 respectively.  Following
> https://tools.ietf.org/html/rfc7252#section-4.8.2, the time between the first
> and last retransmit with these values is 45 seconds - which means that the
> request times out after 93 seconds (MAX_TRANSMIT_WAIT) at the point
> when the MAX_RETRANSMIT+1 is retried.
> 
> This final time of 48 seconds (93sec - 45sec) is bigger than a NAT refresh
> timeout that we are comfortable with, so the recommended DOTS value  for
> max-retransmit is 3.  We have added in this additional missing-hb-allowed to
> increase the timeout before failure is determined.
> 
> However, as mentioned below, this determined failure may not be sufficient
> to cause a session to be closed down as there could be network losses due to
> the DDoS attacks.
> 
> >
> > >
> > (DOTS server to DOTS client) data path is flooded /overloaded and
> > hence packet loss (but not the case with all DDoS type scenarios).
> > >
> > sure that any NAT devices in the path maintain their NAT associations
> > and allow any returning responses (which could be unsolicited if the
> > observe of a mitigation is active).
> > >
> > messages, but can only deduce that the connection from the DOTS client
> > to the DOTS server is good - but cannot make any assumptions about
> > traffic flowing in the other direction.
> >
> > However, if the client does never receive a Coap RST or Coap pong, it
> > will sooner or later give up and not send any ping messages anymore.
> > In this case the server will receive no ping anymore and can decide to send
> own pings.
> Agreed, but as mentioned below the DOTS client must continue to transmit if
> there has been a mitigation request (i.e. running in under attack mode and
> networks could be flaky).
> 
> The retransmission parameters are negotiated between the DOTS agents at
> client session startup ( https://tools.ietf.org/html/draft-ietf-dots-signal-
> channel-34#section-4.5.2 ) so both ends know what the timeouts are.
> 
> >
> > Further if your really want to be sure if the RST was received or not,
> > I’d recommend you to use an own application ping that indicated if the
> > ping is a retransmission or not.
> 
> Well actually, while  the CoAP layer handles the transmission of the
> Empty/RST, the ping is initiated by the DOTS layer (and is told either a RST or
> timeout occurred), not the CoAP layer and so the DOTS application is in
> control of what is going on here.
> 
> > Detection one-way congestion is a different function than keep-alive
> > testing and it is better to use an explicit mechanism for that then
> > trying to infer something from a mechanism that was designed for a
> different purpose.
> >
> 
> > inbound pipe.  If the DOTS client has initiated a mitigation request,
> > then it is unsafe for the DOTS client to close down the session - it
> > will need to refresh the mitigation requests / create new ones even if
> > the mitigation is not being that effective as traffic can still flow
> > to the server.  It is possible that the DOTS server has just restarted
> > - hence the requirement to try and open up a new session in parallel.
> > >
> > pipe to the DOTS client is failing.  The DOTS server then does not
> > need to close down the session as it will be expecting additional
> > mitigation requests from the DOTS client - even though the DOTS server
> Coap Ping is failing.
> > >
> > DOTS client Coap Ping, then there is a good chance that the NAT
> > sessions are "warm" on any intervening NAT devices.  If the DOTS
> > server initiates the Coap Ping on its own cycle, there is a chance
> > that it may not get through and confuse the logic.
> >
> > during mitigation in the dots layer, e.g. don’t use the Coap Ping, but
> > send a non-confirmable Coap message which contains a dots layer “ping"
> > and an indication if a dots-layer “pong" has been received or not.
> 
> As stated above, the CoAP ping is not initiated by the CoAP layer like TCP
> keep-alives, but is triggered by the DOTS application by sending the Coap
> Ping packet - which in effect is what you are suggesting (apart from the non-
> confirmable aspect).  If using TCP, then what you describe is correct (except
> Confirmable/Non-Confirmable are out of the picture).
> 
> The CoAP Empty packet must be confirmable to solicit a RST response (Table
> 1: RFC7252).  I would rather not move away from RFC7252 here.
> 
> >
> > However, note that this still might not work with TCP as messages
> > cannot be transmitted unreliably and not-transmitted/no-acked
> > application layer data will block all other traffic on the same
> > connection at some point because TCP will try to retransmit and shrink the
> congestion window to the minimum.
> 
> > >> Cc: [email-masked]; Konda, Tirumaleswar
> > >> Reddy; [email-masked]; [email-masked]; The IESG; dots-
> > [email-masked];
> > >> Benjamin Kaduk
> > >> Subject: Re: [Dots] Behavior when keep-alives fail (RE: Mirja
> > >> Kühlewind's Discuss on draft-ietf-dots-signal-channel-31: (with
> > >> DISCUSS and
> > COMMENT)
> > >>
> > >> Re-,
> > >>
> > >> Please see inline.
> > >>
> > >> Cheers,
> > >> Med
> > >>
> > >>> De : Mirja Kuehlewind [mailto:[email-masked]] Envoyé :
> > >>> mercredi 3 juillet 2019 14:46 À : BOUCADAIR Mohamed TGI/OLN Cc :
> > >>> Konda, Tirumaleswar Reddy; Benjamin Kaduk; draft-ietf-dots-signal-
> > >>> [email-masked]; [email-masked]; [email-masked]; The
> > >>> IESG; [email-masked] Objet : Re: Behavior when keep-alives
> > >>> fail (RE: [Dots] Mirja Kühlewind's Discuss on
> > >>> draft-ietf-dots-signal-channel-31: (with DISCUSS and
> > >> COMMENT)
> > >>>
> > >>> Hi Med,
> > >>>
> > >>> See below.
> > >>>
> > >>>>> Actually to my understanding this will not work. Both TCP
> > >>>>> heartbeat
> > and
> > >>>>> Coap Ping are transmitted reliably. If you don’t receive an ack
> > >>>>> for
> > >>> these
> > >>>>> transmissions you are not able to send any additional messages
> > >>>>> and
> > can
> > >>>>  When a Confirmable "CoAP Ping" is sent, and if there is no
> > >>>> response,  the "CoAP Ping" is retransmitted max-retransmit number
> > >>>> of times by  the CoAP layer using an initial timeout set to a
> > >>>> random duration  between ack-timeout and
> > >>>> (ack-timeout*ack-random-factor) and  exponential back-off between
> > >>>> retransmissions.  By choosing the  recommended transmission
> > >>>> parameters, the "CoAP Ping" will timeout  after 45 seconds.  If
> > >>>> the DOTS agent does not receive any response  from the peer DOTS
> > >>>> agent for 'missing-hb-allowed' number of  consecutive "CoAP Ping"
> > >>>> Confirmable messages, it concludes that the  DOTS signal channel
> > >>>> session is disconnected.  A DOTS client MUST NOT  transmit a "CoAP
> Ping" while waiting for the previous "CoAP Ping"
> > >> not a false positive. For example, this would have implications on
> > >> the
> > server
> > >> as it may erroneously start automated mitigations (because it
> > >> concludes
> > the
> > >> session is lost).
> > >>
> > >> If the ping is
> > >>> transmitted reliably, one “missed” should be enough to conclude
> > >>> that
> > the
> > >>> session is disconnected.
> > >>
> > >> sending/replying to confirmable ping messages, but the reply may
> > >> get dropped. The session is not disconnected in such case.
> > >>
> > >>>
> > >>> Yes, as Coap Ping is used, the agent should not only conclude that
> > >>> the DOTS signal session is disconnected but also the Coap session
> > >>> and not
> > send
> > >>> If you want to send further UDP datagram you should it
> > >>> unreliability and not more often then one per 3 seconds.
> > >>>
> > >>> Mirja
> > >>>
> > >>>
> > >>>>
> > >>>> Cheers,
> > >>>> Med
> > >>>>
> > >>>>> De : Mirja Kuehlewind [mailto:[email-masked]] Envoyé :
> > >>>>> mercredi 3 juillet 2019 12:26 À : BOUCADAIR Mohamed TGI/OLN Cc :
> > >>>>> Konda, Tirumaleswar Reddy; Benjamin Kaduk; draft-ietf-dots-
> > signal-
> > >>>>>>> De : Mirja Kuehlewind [mailto:[email-masked]] Envoyé :
> > >>>>>>> mardi 2 juillet 2019 16:00 À : BOUCADAIR Mohamed TGI/OLN Cc :
> > >>>>>>> Konda, Tirumaleswar Reddy; Benjamin Kaduk; draft-ietf-dots-
> > >>> signal-
> > >>>>>>> [email-masked]; [email-masked]; [email-masked];
> > >>>>>>> The
> > >> IESG;
> > >>>>>>> [email-masked]
> > >>>>>>> Objet : Re: [Dots] Mirja Kühlewind's Discuss on
> > >>>>>>> draft-ietf-dots-
> > >>> signal-
> > >>>>>>>>>>>> when a client should start a session and what should be
> > >>>>>>>>>>>> done
> > >>> (from
> > >>>>>>> the
> > >>>>>>>>>>>> client side) if a session is detected as inactive (other
> > >>>>>>>>>>>> than
> > >>>>> during
> > >>>>>>>> prior or during an attack.  The DOTS signal channel is
> > >>>>>>>> initiated by the DOTS client.  The DOTS client can then
> > >>>>>>>> negotiate, configure,
> > and
> > >>>>>>>> retrieve the DOTS signal channel session behavior with its
> > >>>>>>>> DOTS
> > peer
> > >>>>>>>> (Section 4.5).  Once the signal channel is established, the
> > >>>>>>>> DOTS agents periodically send heartbeats to keep the channel
> > >>>>>>>> active (Section 4.7).  At any time, the DOTS client may send
> > >>>>>>>> a mitigation request message (Section 4.4) to a DOTS server
> > >>>>>>>> over the active
> > >>> signal
> > >>>>>>>> likelihood of packet loss during a DDoS attack), the DOTS
> > >>>>>>>> server periodically sends status messages to the client,
> > >>>>>>>> including basic mitigation feedback details.  Mitigation
> > >>>>>>>> remains active until the DOTS client explicitly terminates
> > >>>>>>>> mitigation, or the mitigation lifetime expires.  Also, the
> > >>>>>>>> DOTS server may rely on the signal channel session loss to
> > >>>>>>>> trigger mitigation for pre-configured mitigation requests (if any).
> > >>>>>>>
> > >>>>>>> Okay thanks for for the pointer. What I think is missing are
> > >>>>>>> some sentences about what the client (or server) should do if
> > >>>>>>> the keep-
> > >>> alive
> > >>>>>>> fails. Try to reconnect directly or just with the next request
> > >>>>>>> or whatever. Basically who should reconnect and when?
> > >>>>>>
> > >>>>>> The server may use the failure to initiate automated mitigation
> > >>>>>> (see
> > >>> the
> > >>>>> Actually to my understanding this will not work. Both TCP
> > >>>>> heartbeat
> > and
> > >>>>> Coap Ping are transmitted reliably. If you don’t receive an ack
> > >>>>> for
> > >>> these
> > >>>>> transmissions you are not able to send any additional messages
> > >>>>> and
> > can

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 16 Jul 2019 17:36:15 +0200

Hi Jon, hi all,

Thanks for your replies. And sorry for my late reply.

Again my higher layer point is that the function you want is different than what Coap Ping was designed for (and I thing you agreed to that) and therefore it is a much cleaner design to specify an own way to realise this function with in your protocol layer.

Please see further below.

> On 3. Jul 2019, at 21:15, Jon Shallow <[email-masked]> wrote:
> 
> Hi Mirja,
> 
> In RFC7252 ACK_TIMEOUT, ACK_RANDOM_FACTOR, and MAX_RETRANSMIT have the suggested values of 2, 1.5 and 4 respectively.

Yes but you use/recommend different values.

>  Following https://tools.ietf.org/html/rfc7252#section-4.8.2, the time between the first and last retransmit with these values is 45 seconds - which means that the request times out after 93 seconds (MAX_TRANSMIT_WAIT) at the point when the MAX_RETRANSMIT+1 is retried.
> 
> This final time of 48 seconds (93sec - 45sec) is bigger than a NAT refresh timeout that we are comfortable with, so the recommended DOTS value  for max-retransmit is 3.

If the waiting time between the retransmission is seen as to small, you can adjust ACK_TIMEOUT or ACK_RANDOM_FACTOR.

>  We have added in this additional missing-hb-allowed to increase the timeout before failure is determined.

However, again, instead of trying to hack the ping mechanism in coap you should really rather design your own application layer mechanism, given you have a different goal and scenario that the coap ping was designed for.
> However, as mentioned below, this determined failure may not be sufficient to cause a session to be closed down as there could be network losses due to the DDoS attacks.

Understood.
> Agreed, but as mentioned below the DOTS client must continue to transmit if there has been a mitigation request (i.e. running in under attack mode and networks could be flaky).

The DOTS client will also stop after missing-hb-allowed has been reached? So it will stop at some point just later.
> 
> 
> The retransmission parameters are negotiated between the DOTS agents at client session startup ( https://tools.ietf.org/html/draft-ietf-dots-signal-channel-34#section-4.5..2 ) so both ends know what the timeouts are.
> 
>> 
> Well actually, while  the CoAP layer handles the transmission of the Empty/RST, the ping is initiated by the DOTS layer (and is told either a RST or timeout occurred),

It initiated by the application but it still uses the Coap layer mechanism. I strongly recommend to not do that.

> not the CoAP layer and so the DOTS application is in control of what is going on here.

DOTS initiates the ping but then Coap takes control and handles retransmission.
> The Ping is not handled in the same as, say, TCP keep-alive packets which are handled completely by the TCP layer.

It’s not the same but the application still has only partial control.

> As stated above, the CoAP ping is not initiated by the CoAP layer like TCP keep-alives, but is triggered by the DOTS application by sending the Coap Ping packet - which in effect is what you are suggesting (apart from the non-confirmable aspect).

Not it’s not. What I suggest is to take fill control. Design an own dots ping or let’s say probing message (that also indicates if the previous message was confirmed or not) and send this as non-confirmable message over Coap. While a dots receiver would the send a (non-confirmable) dots-layer pong.

>  If using TCP, then what you describe is correct (except Confirmable/Non-Confirmable are out of the picture).
> 

I don’t understand how this works. If a message is not transmitted over TCP the connection will close at some point and you have to reconnect. If you can’t do a TCP handshake, you will never get a new connection.

Mirja

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 16 Jul 2019 17:41:08 +0200

Hi Tiru,

Thanks for the updates. I think there is one remaining issue on the use of ping/heart-beats (see also my other message). However, I believe all other discuss points have been addressed now. Thanks for that!

Mirja

> On 15. Jul 2019, at 14:40, Konda, Tirumaleswar Reddy <[email-masked]> wrote:
> 
> Hi Mirja,
>

---

From: <[email-masked]>
Date: Wed, 17 Jul 2019 12:15:42 +0000

Hi Mirja,

Glad to see that we are progressing. With regards to the remaining issue, let's recap on the intended functionality and general observations.

* What is the intended functionality?

(1) What we need is a check to assess the liveness of the peer DOTS agent + maintain NAT/FW state on on-path devices. 
 
(2) CoAP does not impose how CoAP Ping messages should be sent, how their response (or lack of) should be interpreted, etc. This is left to the application.

(3) In a DDoS context, failure to receive heartbeats ** should not ** be systematically interpreted as the session is defunct. To that aim, a specific behavior is followed at the DOTS layer (not CoAP layer). The DOTS layer instructs the CoAP layer (DOTS session, and mitigation operations) accordingly.   

* Why CoAP Pings?

CoAP Pings naturally achieve (1). These messages are interesting because they have small message sizes. CoAP Ping uses Empty Confirmable message (no request) and the response is a Reset message (which is empty).

CoAP Ping can be used ** as-is ** during idle time (no attack). Multiple Ping failures and Ping exchange in both directions is required only during the attack time to really detect if the peer is dead or alive (because of (3)).

This is aligned with (2).

* Why not DOTS-specific messages?

If we have to define such message, it needs to be the smallest footprint as practical. An Empty message is inexpensive, nevertheless empty Non-confirmable messages are not allowed (See rfc7252#section-4.3) and sending the Reset message is optional. 

Furthermore, if heartbeats were non-confirmable that means that we would need to get the application layer to do any recovery work. It is much simpler to make them confirmable and delegate the recovery work to CoAP layer. 

So, even if we did it in the DOTS layer it will be the same as "CoAP ping".

The WG did not see a reason to re-do the same in the application layer. In DOTS, the CoAP ping is initiated by the application (DOTS layer itself) and CoAP layer only handles the retransmissions. The proposed mechanism does not require an update to the CoAP stack other with the assumption that callback handlers are supported to give visibility of when a ping or pong was received by the CoAP layer in the application layer. As such, the current design is pragmatic. 

The DOTS layer has the ** full control required for the intended functionality **.

* Why separating heartbeats parameters from CoAP transmission ones?

We need to allow for a differentiated handling of heartbeats during idle/mitigation times. To that aim, we defined specific heartbeat parameters for each of these times without touching the CoAP connection parameters. All the message transmission parameters including missing-hb-allowed are configurable using the DOTS signal channel (see draft-ietf-dots-signal-channel-35#section-4.5) and these message transmission parameter including the missing-hb-allowed is only used for UDP transport.

One last comment about adjusting CoAP parameter values, we spent in the WG a lot of cycles to discuss this particular point. The conclusion of the WG is that 4 retransmits (that corresponds to the default value of MAX_RETRANSMIT) are not sufficient to detect the session is disconnected during volumetric 
DDoS attack saturating the incoming link, and which will lead to high packet loss. This is why multiple heartbeat losses are required to detect the session is defunct.

Unless there is a flaw with the current design (fail to achieve the intended functionality, implementation complexity, side effects, ...), we don't see a reason to change the current approach. 

Thank you. 

Cheers,
Jon/Tiru/Med

> Envoyé : mardi 16 juillet 2019 17:41
> À : Konda, Tirumaleswar Reddy
> Cc : Jon Shallow; BOUCADAIR Mohamed TGI/OLN; draft-ietf-dots-signal-
> [email-masked]; [email-masked]; [email-masked]; The IESG;
> [email-masked]; Benjamin Kaduk
> Objet : Re: [Dots] Behavior when keep-alives fail (RE: Mirja Kühlewind's
> Thanks for the updates. I think there is one remaining issue on the use of
> ping/heart-beats (see also my other message). However, I believe all other
> discuss points have been addressed now. Thanks for that!
> 
> Mirja
> 
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 18 Jul 2019 17:24:49 +0200

Hi Med,

Thanks for this nice summary. Please see inline.

> On 17. Jul 2019, at 14:15, [email-masked] wrote:
> 
> Hi Mirja,
> 
> CoAP Ping can be used ** as-is ** during idle time (no attack).

Agreed to hear.

> Multiple Ping failures and Ping exchange in both directions is required only during the attack time to really detect if the peer is dead or alive (because of (3)).

This is understood but that’s not what ping is designed for.

> If we have to define such message, it needs to be the smallest footprint as practical.

I think I don’t quite understand this requirement. But let’s go on...

> An Empty message is inexpensive, nevertheless empty Non-confirmable messages are not allowed (See rfc7252#section-4.3) and sending the Reset message is optional. 
> 
> Furthermore, if heartbeats were non-confirmable that means that we would need to get the application layer to do any recovery work.

It’s not really recovery work, it’s how ping usually works. But yes, the application would need to care about anything like this here, but actually as you want to send frequently messages even if no confirmation is received, you actually don’t want to send them reliably and there it’s not really a retransmission. It’s rather sending a new message at a defined time frame.

> It is much simpler to make them confirmable and delegate the recovery work to CoAP layer.

I you want a mechanism that checks both side of the connectivity using a non-confirmable message with ping and pong at the application makes more sense and actually has a “smaller footprint” because you only send one message in each direction instead of an empty message + ack in both.

>  
> 
> So, even if we did it in the DOTS layer it will be the same as "CoAP ping”.

It’s not because it will cover both directions and you can directly indicate if a pong was received or not. I think that’s a functionality that your are actually looking for and is not provided with the lower layer ping.

> 
> The WG did not see a reason to re-do the same in the application layer.

It’s usual to have keep-alive mechanism at multiple layers to cover different needs. Naturally if you implement a keep-alive at a higher layer (which send messages often enough), you can enable also keep-alive as the lower layer(s), however, which will normally not become active as you have messages on the higher layer before the idle timer expires. Therefore it would be easy do design a more dedicated mechanism for the DDoS case and use COAP pings for the “normal” case to address your function (1).

> In DOTS, the CoAP ping is initiated by the application (DOTS layer itself) and CoAP layer only handles the retransmissions. The proposed mechanism does not require an update to the CoAP stack other with the assumption that callback handlers are supported to give visibility of when a ping or pong was received by the CoAP layer in the application layer.

Which probably is an update to any COAP implementation. And usually something that should not be exposed because in this case it’s solely a lower layer function. The app should activate keep-alive, configure the idle time and get notified when the connection has been detected to be not alive anymore.

> As such, the current design is pragmatic. 
> 
> We need to allow for a differentiated handling of heartbeats during idle/mitigation times. To that aim, we defined specific heartbeat parameters for each of these times without touching the CoAP connection parameters.

If you define a separate parameter that needs to be implemented and handled, you can also implement an own mechanism for the DDoS case.

> All the message transmission parameters including missing-hb-allowed are configurable using the DOTS signal channel (see draft-ietf-dots-signal-channel-35#section-4.5) and these message transmission parameter including the missing-hb-allowed is only used for UDP transport.
> 
I believe you spend so much discussion time because you are misusing a mechanism that was not designed for your purpose instead of design a proper one.

> Unless there is a flaw with the current design (fail to achieve the intended functionality, implementation complexity, side effects, ...), we don't see a reason to change the current approach. 

I believe there are flaws in the design. First it’s a layer violation, but if more an idealistic concern but usually designing in layers is a good approach. But more importantly, you end up with un-frequent messages which may still terminate the connection at some point, while what you want is to simply send messages frequently in an unreliable fashion but a low rate until the attack is over.

Please have a discussion with the working group about an alternative mechanism. The meeting is next week and I hope it would be possible to get some time for that in the dots session.

I unfortunately won’t be able to make it to the session but I will sync up with Ben.

Mirja