Subject: [nfsv4] questions w.r.t RPC-over-TLS draft

From: Rick Macklem <[email-masked]>
Date: Mon, 20 Jan 2020 00:30:49 +0000

Hi,

I've started implementing RPC-over-TLS for NFS and have run into a couple
of questions related to the draft (#4, I haven't downloaded #5).

1 - Given this description...
   The flavor value of the verifier received in the reply message from
   the server MUST be AUTH_NONE.  The bytes of the verifier's string
   encode the fixed ASCII characters "STARTTLS".

Is the verifier coded as:
A:   verifier length: 8
      bytes STARTTLS
OR
B:   verifier length: 12
      string length: 8
      bytes STARTTLS

ie. Is there supposed to be a string length as coded by xdr_string() in the
verifier? (It is the words "verifier string" the above that I find confusing.)

Then there is this sentence...
   AUTH_ERROR.  If the client sends a STARTTLS after it has sent other
   non-encrypted RPC traffic or after a TLS session has already been
   negotiated, the server MUST silently discard it.

Does "other non-encrypted RPC traffic" refer specifically to traffic between
the NULL RPC with AUTH_TLS and the STARTTLS or does it refer to non-NULL RPC traffic or??

Thanks for any clarification of this, rick

---

From: Rick Macklem <[email-masked]>
Date: Mon, 20 Jan 2020 02:26:09 +0000

I wrote:
>the NULL RPC with AUTH_TLS and the STARTTLS or does it refer to non-NULL RPC >traffic or??
I am also confused about what "sends a STARTTLS" actually means?
(I'll admit I know nothing about TLS and can only find STARTTLS mentioned w.r.t.
 an SMTP email command.)

Does this mean that the 8 bytes STARTTLS goes on the wire immediately after
a NULL RPC with AUTH_TLS or does it mean the 8 bytes STARTTLS goes in a
TCP RPC message (an 8byte message as indicated by the RPC over TCP record
mark, which would normally be too small) or does "sends a STARTTLS" just saying
that the TLS handshake should come immediately after the NULL RPC with AUTH_TLS?

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Mon, 20 Jan 2020 13:18:35 +0100 (CET)

Hi Rick

----- Original Message -----
> From: "Rick Macklem" <[email-masked]>
> To: "NFSv4" <[email-masked]>
> Sent: Monday, January 20, 2020 3:26:09 AM
> Subject: Re: [nfsv4] questions w.r.t RPC-over-TLS draft

> (I'll admit I know nothing about TLS and can only find STARTTLS mentioned w.r.t.

STARTTLS is a protocol specific signal that a client send to the server to start
TLS handshake. If I read spec correctly, then for RPC it's an operation NULL
with auth flavor AUTH_TLS and empty verifier. Server responds with verifier
that has body "STARTTLS" encoded as opaque bytes.

This is at least what our implementation does.

Regards,
   Tigran.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 20 Jan 2020 10:08:41 -0500

Hi Rick-

May I add the FreeBSD implementation of RPC/TLS to Section 6? Did I
already ask you this and simply forgot to add it?

> On Jan 19, 2020, at 7:30 PM, Rick Macklem <[email-masked]> wrote:
> 
> Hi,
> 

I agree, "string" is confusing. It should rather say "fixed-length
opaque".

Section 7.2 of RFC 1831 has this:
 
enum auth_flavor {
      AUTH_NONE = 0,
      AUTH_SYS = 1,
      AUTH_SHORT = 2
      /* and more to be defined */

};

struct opaque_auth {
      auth_flavor flavor;
      opaque body<400>;

};

The flavor field contains AUTH_NONE (0), the opaque's length field
contains 8, and the opaque's data contains the ASCII characters
"STARTTLS".

> the NULL RPC with AUTH_TLS and the STARTTLS or does it refer to non-NULL RPC traffic or??

The client can't start a TLS session on a connection after it
has sent non-NULL RPC traffic.

> that the TLS handshake should come immediately after the NULL RPC with AUTH_TLS?

"sends a STARTTLS" means "initiates a TLS session."

--
Chuck Lever

---

From: Rick Macklem <[email-masked]>
Date: Mon, 20 Jan 2020 23:08:16 +0000

>already ask you this and simply forgot to add it?
Definitely a "work in progress" at this time, so you can decide if it should be
mentioned. (I am making pretty good progress, but the FreeBSD kernel TLS
only works for send at this time, unless you have hardware that does TOE.
It may be a few months before the kernel TLS can do receive, so that it
actually works.)

>I agree, "string" is confusing. It should rather say "fixed-length
This might still hint at an extra length field.
How about something like "The body of the verifier consists of the 8 ASCII bytes STARTTLS"?
(Btw, this is how I had coded it and then, when I read "verifier string" I wasn't sure.)

>> the NULL RPC with AUTH_TLS and the STARTTLS or does it refer to non-NULL RPC traffic >or??
>
Cool. Maybe it needs to say that the NULL RPC with authentication flavor of AUTH_TLS
constitutes a STARTTLS operation.

That still doesn't really clarify if a NULL RPC with authentication flavor of AUTH_NULL
is also allowed before the NULL RPC with authentication flavor of AUTH_TLS.
(FreeBSD currently uses NULL RPCs with AUTH_NULL as a "ping" to figure out if
 the server is up, however it uses a separate TCP connection to do this, so it shouldn't
 matter if it allowed on the same connection.)

>"sends a STARTTLS" means "initiates a TLS session."
That clarifies it. It might be even clearer if it said "initiates a TLS handshake"?
Alternately, defining the NULL RPC with authentication flavor AUTH_TLS as a STARTTLS
command would do so as well.

One slight problem here is that the above would seem to be TCP specific, since it assumes
ordering, but is not in the TCP section. I know nothing about the UDP case and have no intention of implementing it, but it does seem that 
>>   negotiated, the server MUST silently discard it.
is TCP specific? (Would the UDP variant be non-NULL RPC traffic from the same client
IP address or do you just not worry about this for UDP?)

Thanks for the help with this, rick

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Tue, 21 Jan 2020 10:55:40 -0500

> On Jan 20, 2020, at 6:08 PM, Rick Macklem <[email-masked]> wrote:
> 
> Chuck Lever wrote:

A "fixed-length opaque" is explicitly defined in Section 3.9 of RFC 1832.
No length field.

OK, here's what I've changed it to:

>    the server MUST be AUTH_NONE.  The length of the verifier's body
>    field is eight.  The bytes of the verifier's body field encode the
>    ASCII characters "STARTTLS" as a fixed-length opaque.

Thoughts?

> constitutes a STARTTLS operation.

It's meant only to indicate that the sender supports RPC-over-TLS.
Is that really the same as a STARTTLS operation?

The replacement text reads:

>    If an RPC client attempts to use AUTH_TLS for anything other than the
>    NULL RPC procedure, the RPC server MUST respond with a reject_stat of
>    AUTH_ERROR.  If the client initiates a TLS session after it has sent
>    unencrypted non-NULL RPC traffic the server MUST silently discard it.

> is also allowed before the NULL RPC with authentication flavor of AUTH_TLS.

It seems to me the new text clearly allows an unencrypted NULL RPC at any
time. I'm not sure that's even feasible, but it's an implementer's choice.

> matter if it allowed on the same connection.)

The NULL with AUTH_TLS should be able to act as both announcing support
for RPC-over-TLS and a ping for a particular RPC Program.

> ordering, but is not in the TCP section. I know nothing about the UDP case and have no intention of implementing it, but it does seem that is TCP specific? (Would the UDP variant be non-NULL RPC traffic from the same client

Agreed, that language should probably be moved to Section 5.1.1.

--
Chuck Lever

---

From: Rick Macklem <[email-masked]>
Date: Tue, 21 Jan 2020 23:09:20 +0000

>A "fixed-length opaque" is explicitly defined in Section 3.9 of RFC 1832.
Yep. Sounds fine to me.

>>    ASCII characters "STARTTLS" as a fixed-length opaque.
Yep, as above, sounds fine to me.

>>>> the NULL RPC with AUTH_TLS and the STARTTLS or does it refer to non-NULL RPC traffic >>or??
>>>
Well, whatever the correct terminology, the server needs to know when to switch from
receiving Sun RPC messages (with record marks in front of them for TCP) to receiving
TLS records for the handshake.

If the NULL RPC with AUTH_TLS doesn't do that, then something else must be added
that does do it (and it needs to be in a Sun RPC message or start with 4 bytes that
are obviously not a Sun RPC over TCP record mark).
It sounds like Tigran assumes the NULL RPC with AUTH_TLS does this switchover and
I am fine with that too (and have coded it that way).

Admittedly I am not sure if STARTTLS is mentioned anywhere in RFC-8446, so it might not
be the right term for whatever says "a TLS record to start the handshake is coming
immediately after this".

>time. I'm not sure that's even feasible, but it's an implementer's choice.
Yea. I'm not sure if this is feasible either (at least for TCP), but NULL RPCs seem
harmless, so I think the current wording is ok, at least for reliable ordered transports
like TCP.

>for RPC-over-TLS and a ping for a particular RPC Program.
Agreed. However it comes back to whether or not is to me immediately followed by
the first TLS handshake record.
Yes -->can only be done once, just before the TLS handshake and, as such, other
            NULL RPCs with AUTH_NULL before it might be useful.
No-->means something else needs to be defined so that the server knows when to
          switch from "receiving Sun RPC messages" to "receiving TLS handshake records".
I think "yes" is appropriate and what Tigan has already assumed was the case.

As already noted, I think the above wording (which does allow other NULL RPCs) is fine.

>>> "sends a STARTTLS" means "initiates a TLS session."
Yes, but as noted above, something must go on the wire to indicate this.
I think the NULL RPC with AUTH_TLS is fine for doing this, but if not, something else
needs to be defined to do so.

>> ordering, but is not in the TCP section. I know nothing about the UDP case and have no >>intention of implementing it, but it does seem that is TCP specific? (Would the UDP variant >>be >non-NULL RPC traffic from the same client
Yep.

rick

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 21 Jan 2020 15:21:22 -0800

> be the right term for whatever says "a TLS record to start the handshake is coming

8446 is not the right reference for this; 2222 or 3552 might be the best we
can do (though 3207 has actual details on how to do it for SMTP, 3501 for
IMAP, and a number of other protocols have STARTTLS verbs in the RFC
archive).

-Ben

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Wed, 22 Jan 2020 12:52:52 +0100 (CET)

----- Original Message -----
> To: "Chuck Lever" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>
> Sent: Wednesday, January 22, 2020 12:09:20 AM

> harmless, so I think the current wording is ok, at least for reliable ordered

I am pretty sure, that once TLS session is established, you can't sent any other
requests outside of it.

Tigran.

>>> ordering, but is not in the TCP section. I know nothing about the UDP case and
>>> have no >>intention of implementing it, but it does seem that is TCP specific?
>>> (Would the UDP variant >>be >non-NULL RPC traffic from the same client

---

From: Rick Macklem <[email-masked]>
Date: Wed, 22 Jan 2020 22:51:04 +0000

Mkrtchyan, Tigran wrote:
[stuff snipped]
>I am pretty sure, that once TLS session is established, you can't sent any other
Yea, the only case I can think of where it might be possible for TCP is after an
SSL_shutdown is done on the socket. I also haven't looked at the RFC enough to figure out
if unencrypted (I think it calls them cleartext) TLS records can be mixed into encrypted ones.

rick

Tigran.

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 23 Jan 2020 17:54:06 -0800

> SSL_shutdown is done on the socket. I also haven't looked at the RFC enough to figure out

"SSL_shutdown is done on the socket" translates (roughly) to TLS-speak as
"both endpoints send and receive "close_notify" alerts from the peer", and
in theory the protocol can support this but evidence of real-world
functionality is quite sparse.  (Many implementations do not really do
"close_notify" quite according to the spec, and trying to wait for it from
the peer is pretty dicey.)

> if unencrypted (I think it calls them cleartext) TLS records can be mixed into encrypted ones.

They can't.  (If there was a severe need a way could be found, e.g., with a
new record type, but there would be a lot of pushback on this.)

-Ben

---

From: Rick Macklem <[email-masked]>
Date: Fri, 24 Jan 2020 23:03:28 +0000

>the peer is pretty dicey.)
Yes. According to the OpenSSL doc, you can check for a 0 return from the first SSL_shutdown()
call and do a second one in that case, to make sure you get the close_notify from the peer.
None of the sample code I've found (some in the OpenSSL doc) does that, or even checks
the return value from the SSL_shutdown() calls.

Just to be clear, I don't see any need for unencrypted RPC traffic to work on the socket
after the TLS handshake starts.

I also don't care if NULL RPCs with AUTH_NULL are allowed before the NULL RPC with
AUTH_TLS is done.
The draft said "no unencrypted RPC traffic", whereas the NULL RPC with AUTH_TLS is
just that, so it seems there needs to be some rewording done.

I do think that the draft should clearly state:
1 - A NULL RPC with AUTH_TLS that gets a successful reply with "STARTTLS" in the
     reply verifier is a "STARTTLS command, with the TLS handshake following immediately
     after this".
2 - What, if any, unencrypted RPC traffic is allowed to precede the NULL RPC with AUTH_TLS.

>
>> if unencrypted (I think it calls them cleartext) TLS records can be mixed into encrypted >ones.
>
>new record type, but there would be a lot o...
I can't see any reason to do this.

rick

---

From: Chuck Lever <[email-masked]>
Date: Tue, 28 Jan 2020 14:35:44 -0500

> On Jan 27, 2020, at 4:14 PM, David Noveck <[email-masked]> wrote:
> 
> On Mon, Jan 27, 2020, 2:23 PM Chuck Lever <[email-masked]> wrote:
>> 
>> I'm willing to make some alterations. I don't think we're changing any
>> consensus decisions here, so probably the best approach would be for me
>> to submit another revision after we resolve the wording issues.
>> 
>> Does that sound right, NFSv4 chairs?
> 
> This makes sense. The document is waiting for a write-up which is in turn waiting for IPR stuff.
> 
> I don't think you are changing consensus decisions but when posting the new revision, you should give the working group members  an opportunity to object of they disagree. Once those days are over and I have the IPR stuff, I can proceed with the write-up.

I'm proposing the following changes for rpc-tls-06:

https://chucklever.github.io/i-d-rpc-tls/#go.draft-ietf-nfsv4-rpc-tls.diff

These clarifications are the result of Rick and Ben's discussion
earlier in this thread about Sections 4.1 and 5.1.1. The document
is now agnostic about what is permitted on a TCP connection
before TLS negotiation.

I've also documented FreeBSD in the Implementation Status section.

I'll submit this update in a few days if I do not receive further
comments.

--
Chuck Lever