Subject: Re: [DNSOP] Lars Eggert's Discuss on draft-ietf-dnsop-dns-tcp-requirements-13: (with DISCUSS and COMMENT)

From: "Wessels, Duane" <[email-masked]>
Date: Mon, 29 Nov 2021 23:53:48 +0000

> On Oct 26, 2021, at 4:35 AM, Lars Eggert via Datatracker <[email-masked]> wrote:
> 
> Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. 
> 
> Lars Eggert has entered the following ballot position for
> draft-ietf-dnsop-dns-tcp-requirements-13: Discuss
> 
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> 
> 
> Please refer to https://secure-web.cisco.com/1kXvipN4VaKHO3AYxNHSk3_n8uHuI5gukOoaWs_9cG3yENUfEij_EahsVifsuqDvJ87tOMzqfsQvSNVDrlS_-uD93gYFrH-W0nErz-3dDIJpel5Zl7MmuWBdfniJzujsocAV1lsSsYtX8awWBkQ8eb_GhRen4BPENNuz1f9DU8scOGmh_6XvDTl2h2Ut3BN2YuNmDbmfWLYWKn2ljBjy70M3-N-vnFroRv7U3a3Kq-iNDmhKR53kaMlSwzI1NM_rK/https%3A%2F%2Fwww.ietf.org%2Fblog%2Fhandling-iesg-ballot-positions%2F
> for more information about how to handle DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://secure-web.cisco.com/1V7ehlyVXKiQcvhjjSAc0dm4x7jce6oRNiVmeJVwrJmYfNJTQCXozSCiVsTTDTMA31OsPaLi5ktIBX_1SJTMwPOMjHkyejN20CFahmTm4V-mFwr3n3zhznbcttOwt49mZNQ24MwFa4SFXIhHivGRuI65YKsZUQDdEJj2ORiTP9kCyMMSw6uGu5eE_JQtlH1M3Q7rqSm33c6SaE0h2NlmjlKGPa6zzXngPE8McI90Hbd47Q3rc4CuANJZLqNdhgNwu/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-dnsop-dns-tcp-requirements%2F
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Section 4.2. , paragraph 3, discuss:
>>   Since host memory for TCP state is a finite resource, DNS clients and
>>   servers MUST actively manage their connections.  Applications that do
>>   not actively manage their connections can encounter resource
>>   exhaustion leading to denial of service.  For DNS, as in other
>>   protocols, there is a tradeoff between keeping connections open for
>>   potential future use and the need to free up resources for new
>>   connections that will arrive.
> 
> For it to contain a MUST-level requirement, this section needs to give a lot
> more concrete guidance on what it means to "actively" manage connections. Most
> operating systems by default impose some application limits that usually
> effectively prevent DOS or other resource exhaustion issues. Is the intent here
> that DNS implementations need to do more? If so, what?

I can easily be convinced that SHOULD is more appropriate than MUST here.  

I dont necessarily agree that operating systems alone do a very good job
of preventing DOS conditions.  It is possible that Im not up-to-date on
the latest and greatest in terms of operating system features, but I think
historically applications have fared better when they manage their own
connections.  For example, can we realistically expect the OS to know which
idle connections should be closed?

> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> Section 2.4. , paragraph 1, comment:
>> 2.4.  Fragmentation and Truncation
> 
> Fragmentation and IP fragments getting dropped is one reason for needing more
> retries with EDNS(0). But IIRC, a larger contributing factor is that EDNS(0)
> doesn't detect or recover from loss of any UDP packets making up the overall
> message. That means that a normal packet loss (due to congestion or other
> reasons) amplifies into loss of the entire DNS message.

This new paragraph has been added:

   Note that a receiver is unable to differentiate between packets lost
   due to congestion and packets (fragments) intentionally dropped by
   firewalls or middleboxes.  Over network paths with non-trival amounts
   of packet loss, larger, fragmented DNS responses are more likely to
   never arrive and time out compared to smaller, unfragmented
   responses.  Clients might be misled into retrying queries with
   different EDNS(0) UDP packet size values for the wrong reason.

> 
> Section 3. , paragraph 1, comment:
>> 3.  DNS over TCP Requirements
> 
> While the history preceding this section is interesting for context, I think
> moving this section up would increase readability significantly.

Okay with me.  I would like to hear from the Chairs.

> 
> Section 4.2. , paragraph 3, comment:
>>   DNS server software SHOULD provide a configurable limit on the total
>>   number of established TCP connections.  If the limit is reached, the
>>   application is expected to either close existing (idle) connections
>>   or refuse new connections.  Operators SHOULD ensure the limit is
>>   configured appropriately for their particular situation.
> 
> Again, the OS generally already imposes limits. Why recommend that DNS
> implementations self-impose other (lower?) limits?
Perhaps the below text is better?  It is directed more to operators (the
intended audience) rather than implementors, and acknowledges the reality
that implementations already have such limits today.

   Operators of DNS server software SHOULD be aware that operating
   system and application vendors MAY impose a limit on the total number
   of established connections.  These limits may be designed to protect
   against DDoS attacks or performance degradation.  Operators SHOULD
   understand how to increase these limits if necessary, and the
   consequences of doing so.

> 
> Section 4.2. , paragraph 3, comment:
>>   DNS server software MAY provide a configurable limit on the number of
>>   established connections per source IP address or subnet.  This can be
>>   used to ensure that a single or small set of users cannot consume all
>>   TCP resources and deny service to other users.  Operators SHOULD
>>   ensure this limit is configured appropriately, based on their number
>>   and diversity of users.
> 
> This limit only begins to establish fairness once the overall number of
> permitted connections is reached. Until that point, it possibly limits client
> performance without any benefit.

I suppose this depends on how it is implemented and there are lessons to be
learned from the world of IP QOS.  

I surveyed open source DNS server code and their developers and found that only
one implements this limit and ships with no limit by default.  

Perhaps this current (updated) paragraph is better:

   DNS server software MAY provide a configurable limit on the number of
   established connections per source IP address or subnet.  This can be
   used to ensure that a single or small set of users cannot consume all
   TCP resources and deny service to other users.  Note, however, that
   if this limit is enabled, it possibly limits client performance while
   leaving some TCP resources unutilized.  Operators SHOULD be aware of
   these tradeoffs and ensure this limit, if configured, is set
   appropriately based on the number and diversity of their users, and
   whether users connect from unique IP addresses or through a shared
   Network Address Translator [RFC3022].

> 
> Section 4.2. , paragraph 3, comment:
>>   DNS server software SHOULD provide a configurable timeout for idle
>>   TCP connections.  For very busy name servers this might be set to a
>>   low value, such as a few seconds.  For less busy servers it might be
>>   set to a higher value, such as tens of seconds.
> 
> Ditto.

In this case all of the open source implementations I surveyed have this 
limit enabled by default.  

> 
> Section 4.2. , paragraph 3, comment:
>>   DNS server software MAY provide a configurable limit on the number of
>>   transactions per TCP connection.
> 
> What does that limit protect against?

proposed new text:

   DNS server software MAY provide a configurable limit on the number of
   transactions per TCP connection.  This can help protect against
   unfair connection use (e.g., not releasing connection slots to other
   clients) and network evasion attacks.

> 
> Section 4.2. , paragraph 2, comment:
>>   Similarly, DNS server software MAY provide a configurable limit on
>>   the total duration of a TCP connection.
> 
> What does that limit protect against?

Proposed new text:

   Similarly, DNS server software MAY provide a configurable limit on
   the total duration of a TCP connection.  This can help protect
   against unfair connection use, slow read attacks, and network evasion
   attacks.

> 
> Section 4.5. , paragraph 3, comment:
>>   Most open source DNS server implementations provide a configurable
>>   limit on the total number of established connections.  Default values
>>   range from 20 to 150.  In most cases, where the majority of queries
>>   take place over UDP, 150 is a reasonable limit.  For services or
>>   environments where most queries take place over TCP or TLS, 5000 is a
>>   more appropriate limit.
> 
> 20-150 is orders of magnitude less than I expected. Even 5000 seems very low,
> given the capabilities of even low-end servers.

Im not sure what else to say.  20-150 is what we find in current
implementations, and I suspect it must be sufficient for most use cases
(today at least).  I find it difficult to make recommendations for the
range of systems where DNS software can run (from large & expensive servers
on down to the raspberry pi and home routers).

DW

---

From: Lars Eggert <[email-masked]>
Date: Tue, 30 Nov 2021 09:51:11 +0200

Hi,

thanks for the detailed response. I have a few comments below, and I've trimmed everything from this reply where I agree with the resolution you proposed in your response.

> I can easily be convinced that SHOULD is more appropriate than MUST here.

I think that would be better, esp. for clients and because the "actively manage" term is still unclear to me - it seems to mean "must implement (some of) the limits below"?

> idle connections should be closed?

The OS will certainly try to close sufficient connections under DDoS to remain operational. But if an application wants to see connections closed according to a certain policy - and DNS servers probably would - they need to actively engage. Maybe that's the rationale here?

(Also, Linux and other major OSs these days handle very large numbers of TCP connections just fine, I think I recall seeing numbers up to a million for Linux (assuming sufficiently beefy hardware). And deployments that needs to handle such connection counts would normally load-balance into a server cluster anyway. So I remain a bit unconvinced that the limits described in this doc are all that important. But that's just a comment and I'm certainly not going to block the document over this.)

> Okay with me.  I would like to hear from the Chairs.

I'm OK with whatever resolution they prefer.

> limit enabled by default.

It might be useful to add a brief note similar to the one above here as well.

Thanks,
Lars