Subject: [dns-privacy] Working Group Last Call for draft-ietf-dprive-xfr-over-tls

From: Tim Wicinski <[email-masked]>
Date: Thu, 21 Jan 2021 08:54:06 -0500

All

The authors on the xfr-over-tls draft have published an update to their
document and feel they have addressed all issues.

This starts a Working Group Last Call for draft-ietf-dprive-xfr-over-tls

Current versions of the draft is available here:
https://datatracker.ietf.org/doc/draft-ietf-dprive-xfr-over-tls/

The Current Intended Status of this document is: Standards Track

Please review the draft and offer relevant comments.
If this does not seem appropriate please speak out.
If someone feels the document is *not* ready for publication, please speak
out with your reasons.

This starts a two week Working Group Last Call process, and ends on:  4
February 2021

thanks
tim

---

From: Peter van Dijk <[email-masked]>
Date: Sun, 24 Jan 2021 20:56:14 +0100

Hello,

> Please review the draft and offer relevant comments.

I like most of the draft. Some of it seems 'obvious' but often 'obvious' differs from reader to reader, so it's good to be explicit about a lot of things. So, I'm not here to say 'all of this is so obvious that the draft does not need to exist at all'.

However, there's one part that bothers me. The 'intermingling' of multiple XFRs over a single connection (which is a SHOULD, not a MUST, so the document 'gets away with it', or conversely, implementations will get away with not implementing it) feels to me like a lot of additional code complexity for a gain that is not clear to me. XFRs are not latency sensitive to the point that TLS setup is an impediment.

In other words, I do not think the goal 'minimize the number of TCP connections between a primary and secondary' warrants the complexity of intermingling XFRs. I'd much rather have a few more TCP connections, a few more TLS states, and a much simpler code base.

We (PowerDNS) do not think we will implement the parts of the draft that call for such code complexity, and thus, we will serialise XFRs (serial connection reuse does make sense to me) and/or in fact open multiple TLS connections (like we do with TCP today). At least in our code base, TLS setup would never be the bottleneck for transfers anyway.

Do any other vendors actually plan to implement such thorough connection reuse, with primaries actually intermingling multiple XFR responses over a single TCP/TLS session, and with secondaries actually firing off several XFR requests on a single connection to receive the zone chunks in whichever order they may come?

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: Brian Dickson <[email-masked]>
Date: Mon, 25 Jan 2021 10:28:37 -0800

On Sun, Jan 24, 2021 at 11:56 AM Peter van Dijk <[email-masked]>
wrote:

> I like most of the draft. Some of it seems 'obvious' but often 'obvious'
> differs from reader to reader, so it's good to be explicit about a lot of
> things. So, I'm not here to say 'all of this is so obvious that the draft
> does not need to exist at all'.
>
> However, there's one part that bothers me. The 'intermingling' of multiple
> XFRs over a single connection (which is a SHOULD, not a MUST, so the
> document 'gets away with it', or conversely, implementations will get away
> with not implementing it) feels to me like a lot of additional code
> complexity for a gain that is not clear to me. XFRs are not latency
> sensitive to the point that TLS setup is an impediment.
>

I'm not one of the authors, but re-reading the draft with "intermingling"
in focus, I think the issues are:
- The intermingling aspect originates in RFC5936 (over TCP)
- They want parity between TCP and XoT (which is reasonable)
- Interoperability is assumed and not negotiated

>
> In other words, I do not think the goal 'minimize the number of TCP
> connections between a primary and secondary' warrants the complexity of
> intermingling XFRs. I'd much rather have a few more TCP connections, a few
> more TLS states, and a much simpler code base.
>
> We (PowerDNS) do not think we will implement the parts of the draft that
> call for such code complexity, and thus, we will serialise XFRs (serial
> connection reuse does make sense to me) and/or in fact open multiple TLS
> connections (like we do with TCP today). At least in our code base, TLS
> setup would never be the bottleneck for transfers anyway.
>

The relevant sections appear to be the text at the top of section 6.2, and
all of 6.3.2.

Servers MUST be able to handle multiple XFR requests.
Clients MUST be able to handle intermingled responses.
However, Servers MAY intermingle, rather than must, on the responses.

I concur with the code base complexity concern, particularly on the client
side.

I think this might be possible to accommodate with guidance (i.e. suggested
text to the following effect):
"NB: If clients do not want to support intermingled responses, a compliant
client could serialize requests to avoid multiple outstanding XFR requests.
If only one XFR is outstanding, no intermingling will ever occur."
Similarly, server-side guidance could be added:
"NB: if servers do not want to implement intermingled responses, they can
serialize the full XFR responses to individual XFR requests, and this would
be compliant. Similarly, reordering of XFR responses versus XFR requests is
allowed but not required (in order to be compliant)."

>
> Do any other vendors actually plan to implement such thorough connection
> reuse, with primaries actually intermingling multiple XFR responses over a
> single TCP/TLS session, and with secondaries actually firing off several
> XFR requests on a single connection to receive the zone chunks in whichever
> order they may come?

It is not clear if our (internal) implementations will do so, or how
(intermingling out-of-order serialized-but-not-interleaved, versus fully
interleaving of multiple XFR responses).

One possibility is the use of TLS proxies, which could multiplex multiple
XFR over TCP streams, such as from multiple TCP clients to multiple TCP
servers, across "shared" TLS XFR sessions (site to site). But that is
conjecture only at this point.

Brian

---

From: Peter van Dijk <[email-masked]>
Date: Wed, 27 Jan 2021 21:35:56 +0100

Hi Brian,

On Mon, 2021-01-25 at 10:28 -0800, Brian Dickson wrote:
> I'm not one of the authors, but re-reading the draft with "intermingling" in focus, I think the issues are:

Several people have pointed that out to me - I totally missed that
since forever. 5936 specified the intermingling in a very loose way,
though.

> - They want parity between TCP and XoT (which is reasonable)

The document goes further than parity, by adding 'optimize the use of
TCP connections and SHOULD ... edns-tcp-keepalive ... to manage
persistent connections'. It also adds 'The number of TCP connections
between a secondary and primary SHOULD be minimized (also see Section
6.4).' While I think reducing the number of connections is always good,
this should be part of a good tradeoff in code complexity, and the
risks of head of line blocking.

(Of course, head of line blocking becomes much worse when the number of
TCP connections is limited and intermingling does not happen. I do get
where the document is coming from here.)

> - Interoperability is assumed and not negotiated

Indeed.
 
> The relevant sections appear to be the text at the top of section 6.2, and all of 6.3.2.
> 
> I concur with the code base complexity concern, particularly on the client side.

My worries are mostly split between the client and server side.

> 
> I think this might be possible to accommodate with guidance (i.e. suggested text to the following effect):
> "NB: If clients do not want to support intermingled responses, a compliant client could serialize requests to avoid multiple outstanding XFR requests. If only one XFR is outstanding, no intermingling will ever occur."

Indeed, the document allows this but it would be good to call out.

> "NB: if servers do not want to implement intermingled responses, they can serialize the full XFR responses to individual XFR requests, and this would be compliant. Similarly, reordering of XFR responses versus XFR requests is allowed but not required (in order to be compliant)."

Indeed, that more clearly reduces the scope to 'servers do not close
TCP connections after sending out a single XFR' (which is the most
basic requirement currently specified in the document).
 
> It is not clear if our (internal) implementations will do so, or how (intermingling out-of-order serialized-but-not-interleaved, versus fully interleaving of multiple XFR responses).
> 
> One possibility is the use of TLS proxies, which could multiplex multiple XFR over TCP streams, such as from multiple TCP clients to multiple TCP servers, across "shared" TLS XFR sessions (site to site). But that is conjecture only at this point.

Yes, but definitely possible. Such a proxy would need to be aware of DNS TCP framing, of course.
 
Kind regards,

---

From: Peter van Dijk <[email-masked]>
Date: Wed, 27 Jan 2021 21:39:36 +0100

> since forever. 5936 specified the intermingling in a very loose way,

And just now it was pointed out to me that 7766, which updates 5936,
has text like 'DNS clients MUST take care to minimize the number of
concurrent TCP connections made to any individual server.  It is
RECOMMENDED that for any given client/server interaction there SHOULD
be no more than one connection for regular queries, one for zone
transfers, ...'

But, it also goes on to say 'certain primary/secondary configurations
with many busy zones might need to use more than one TCP connection for
zone transfers for operational reasons (for example, to support
concurrent transfers of multiple zones).' and I feel the XoT document
is taking that to the next level :-)

Kind regards,

---

From: Sara Dickinson <[email-masked]>
Date: Fri, 29 Jan 2021 17:34:43 +0000

> On 27 Jan 2021, at 20:35, Peter van Dijk <[email-masked]> wrote:
> 
> Hi Brian,
> 

Hi Peter/Brian, 

Thanks for all the comments and the follow up mail about RFC7766. Yes, those documents already state among other things that 
* open connections SHOULD be re-used (RFC5936)
* connections SHOULD be minimised (RFC7766)
* queries should be pipelined and OOOR handled (RFC7766)

so this draft isn’t adding anything new, just stating the implications of the above for XFRs much more explicitly. However, I think it is true that most open source XFR implementations are not strictly compliant with those existing specs today :-)

As you will see from later comments, this was in fact relatively straight forward to implement on the client side in NSD, so getting feedback that this is a bigger implementation challenge in other code bases is much appreciated (the BIND folks have mentioned some similar concerns, particularly with pipelining). 

> basic requirement currently specified in the document).

To pick apart what exact changes would be needed to the document to support these 2 relaxations explicitly for XFRs, it would include:

CONNECTION RE-USE:
* Adding a reason to section 6.3.1 for not re-using an existing connection something like: “as noted in RFC7766, separate connections for different zones might be preferred for operational reasons. In this case the number of concurrent connections for zone transfers SHOULD be limited to the total number of zones transferred between the client and server.”
* If there is consensus the above is the right way to go, then since the main goal of limiting the connections is to reduce load on the server, I would also suggest adding some text to say “servers MAY limit the number of concurrent connections accepted from a given secondary”. One concern is that there is no easy way for the server to signal a preferred behaviour (for example if it has a very large number of zones and a very large number of secondaries it might really want to minimise connections), so I think this is the best that can be done? 

PIPELINING:
* Updates to sections 6, 6.1 and 6.2 and 6.3.2 to add an exception for XFRs wrt pipelining. For section 6, this would be something like:

“RFC7766 states that 'DNS clients SHOULD pipeline their queries’ on TCP connections, however it did not distinguish between XFRs and other queries for this behaviour. XFRs are not as latency sensitive as other queries, and can be significantly more complex for clients to handle both because of the large amount of state that must be kept and because there may be multiple messages in the responses. For these reasons this requirement is relaxed for XFR requests: clients sending XFRs MAY choose not to pipeline queries. Clients that do not pipeline XFR queries, therefore, have no additional requirements to handle out-of-order or intermingled responses as they will never receive them. "

Would that address your concerns? It would also be good to hear from other folks if they agree with this balance.

> Yes, but definitely possible. Such a proxy would need to be aware of DNS TCP framing, of course.

Note that NSD can already support connection re-use as a secondary in some situations and one of the patches reference in the Implementation section adds a configuration option to make this the preferred mode. It also pipelines queries and handles OOOR/intermingled responses in this mode.

In testing, servers seem able to cope with receiving pipelined queries, but all the open source implementations I know of serialise AXFR responses. There is an overview of current behaviour in different implementations here for reference (please let me know of any inaccuracies):
https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Implementation+Status#DNSPrivacyImplementationStatus-XFR/XoTImplementationstatus <https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Implementation+Status#DNSPrivacyImplementationStatus-XFR/XoTImplementationstatus>

Regards

Sara.