Subject: [nfsv4] NFS over QUIC

From: [email-masked] (J. Bruce Fields)
Date: Thu, 3 Sep 2020 17:52:42 -0400

I've been thinking about what might be required for NFS to run over
QUIC.

Also cc'ing Steve French in case he's thought about this for CIFS/SMB.

I don't have real plans.  For Linux, I don't even know if there's a
kernel QUIC implementation planned yet.

QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:

	https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/

For example, section 4.3, which explains how to authenticate on top of
an already-encrypted session, should also apply to QUIC.

QUIC runs over UDP, so I think all that would be required to negotiate
support would be to attempt a QUIC connection to port 2049.

The "Transport Layers" section in the NFS RFCs:

	https://tools.ietf.org/html/rfc5661#section-2.9

requires transports support reliable and in-order transmission, forbids
clients from retrying a request unless a connection is lost, and forbids
servers from dropping a request without closing a connection.  I'm still
vague on how those requirements interact with QUIC's connection
management and 0-RTT reconnection.

https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks
useful, as a guide for applications running over QUIC.  It warns that
connections can time out fairly quickly.  For timely callbacks over NFS
sessions, that means we need the client to ping the server regularly.
Sounds like that's what they do for HTTP/QUIC to make server push
notifications work:

	https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5

	HTTP clients are expected to use QUIC PING frames to keep
	connections open.  Servers SHOULD NOT use PING frames to keep a
	connection open.  A client SHOULD NOT use PING frames for this
	purpose unless there are responses outstanding for requests or
	server pushes.

QUIC allows multiple streams per connection--I wonder how we might use
that.  RFC 5661 justifies the requirement for an ordered transport with:

	Ordered delivery simplifies detection of transmit errors, and
	simplifies the sending of arbitrary sized requests and responses
	via the record marking protocol.

So as long as we don't try to split a single RPC among streams, I think
we're OK.  Would a stream per session slot be reasonable?  I'm not sure
what the cost of a stream is.

Do we need to add a new universal address type so the protocol can
specify QUIC endpoints when necessary?  (For server-to-server-copy, pnfs
file layouts, fs_locations, etc.)  All QUIC needs is an IP address and
maybe a port, so maybe the existing UDP/TCP addresses are enough?

--b.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 3 Sep 2020 19:48:19 -0400

Hi Bruce-

> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <[email-masked]> wrote:
> 

The link to the latest version of that document is

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

> an already-encrypted session, should also apply to QUIC.

Most of the document's content will be re-used for defining
RPC-over-QUIC, for example the ALPN defined in Section 8.2.
Lars Eggert, a chair of the QUIC WG, has been helping guide
the RPC-over-TLS effort with an eye towards using QUIC for
RPC when QUIC becomes more mature.

I thought the plan was to write a specification of RPC-over-
QUIC as a new RPC transport type with a netid and uaddr along
with a definition of the transport semantics (a la TI-RPC).
The document would need to explain record marking, peer
authentication, how to use multi-path and multi-stream support,
and so on.

Making NFS work on that transport should then be straightforward
enough that perhaps additional standards work wouldn't be
necessary.

--
Chuck Lever
[email-masked]

---

From: Bruce Fields <[email-masked]>
Date: Thu, 3 Sep 2020 20:32:42 -0400

> enough that perhaps additional standards work wouldn't be

Oh, OK, good.  Sounds like you're way ahead of me, then, I didn't know
there was a plan.

--b.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 3 Sep 2020 20:53:23 -0400

> On Sep 3, 2020, at 8:32 PM, Bruce Fields <[email-masked]> wrote:
> 

That's all there is for the moment! :-)

--
Chuck Lever
[email-masked]

---

From: Brian Pawlowski <[email-masked]>
Date: Thu, 3 Sep 2020 18:02:06 -0700

Where the Like button?

On Thu, Sep 3, 2020 at 5:53 PM Chuck Lever <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Steven French <[email-masked]>
Date: Fri, 4 Sep 2020 03:01:14 +0000

We (open source SMB3 developers) have been thinking a lot about SMB3.1.1 over QUIC especially after the interesting talks on this at the last few SDC conferences e.g.

https://www.snia.org/sites/default/files/SDC/2019/presentations/SMB/Li_Frank_Surfing_the_World_Wide_File_SMB3_Improvements_for_Safe_and_Efficient_Internet_Access.pdf (slides 8 and following).

And some of the broader coverage:
https://redmondmag.com/articles/2020/03/02/microsoft-smb-over-quic-to-windows.aspx

I have gotten a chance to talk with some of the developers of the recently open sourced quic libraries (https://github.com/microsoft/msquic) and there are various code style issues that would prevent it being merged into Linux kernel without significant changes (not just to fix camelCase) but ... it is a very very promising place to start.

One of the ideas being bounced around is to do this in stages - first try to cleanup the kernel style issues with the open sourced msquic project and merge it experimentally into the Linux client (cifs.ko) since there should be SMB3.1.1. servers to test it against reasonably soon (presumably we will know a lot more after the 2020 SNIA SDC talk on SMB3.1.1 over QUIC in a few weeks updating us on status https://www.snia.org/events/storage-developer/2020/abstracts#smb-Dantuluri) after we are convinced that the quic driver interoperates ok then move it into the Linux networking stack and let the networking experts clean it up more and remove the 'experimental' tag from it when satisfied.

More discussion is needed on this so we should add samba-technical and some of the network experts as well.

I am very excited about the possibilities opened up by a decent quic kernel driver, but the current cross platform open sourced driver is fairly large - ~90KLOC so will have to be trimmed down (and a lot of boring changes to convert to kernel style needed).

-----Original Message-----
From: Chuck Lever <[email-masked]> 
Sent: Thursday, September 3, 2020 7:53 PM
To: Bruce Fields <[email-masked]>
Cc: Linux NFS Mailing List <[email-masked]>; NFSv4 <[email-masked]>; Steven French <[email-masked]>
Subject: [EXTERNAL] Re: [nfsv4] NFS over QUIC

>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fda
>>> tatracker.ietf.org%2Fdoc%2Fdraft-cel-nfsv4-rpc-tls%2F&amp;data=02%7C
>>> 01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee6
>>> 3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&am
>>> p;sdata=pOnlPJcTkyJiqQm%2BCbo7dbOAqHMKFIi01qrShVZTH%2Bk%3D&amp;reser
>>> ved=0
>> 
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdat
>> atracker.ietf.org%2Fdoc%2Fdraft-ietf-nfsv4-rpc-tls%2F&amp;data=02%7C0
>> 1%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%
>> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&amp;s
>> data=YYBPryN5sFJApWm%2BkQvmjYzGXxjxI1fXRUWVXvVGE24%3D&amp;reserved=0
>> 
>>> For example, section 4.3, which explains how to authenticate on top 
>>> of an already-encrypted session, should also apply to QUIC.
>> 
>> Lars Eggert, a chair of the QUIC WG, has been helping guide the 
>> RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC 
>> becomes more mature.
>> 
>> I thought the plan was to write a specification of RPC-over- QUIC as 
>> a new RPC transport type with a netid and uaddr along with a 
>> definition of the transport semantics (a la TI-RPC).
>> authentication, how to use multi-path and multi-stream support, and 
>> so on.
>> 
>> enough that perhaps additional standards work wouldn't be necessary.
> 

> --b.
> 
>>> QUIC runs over UDP, so I think all that would be required to 
>>> negotiate support would be to attempt a QUIC connection to port 2049.
>>> 
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Frfc5661%23section-2.9&amp;data=02%7C01%7CSteve
>>> n.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%7C72f988
>>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&amp;sdata=ej
>>> gN6OuHrcDMzejeEKbaP5UiD1GAxHfQcPyLZZ45vbo%3D&amp;reserved=0
>>> 
>>> requires transports support reliable and in-order transmission, 
>>> forbids clients from retrying a request unless a connection is lost, 
>>> and forbids servers from dropping a request without closing a 
>>> connection.  I'm still vague on how those requirements interact with 
>>> QUIC's connection management and 0-RTT reconnection.
>>> 
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.ietf.org%2Fid%2Fdraft-ietf-quic-applicability-07.txt&amp;data=02%7
>>> C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee
>>> 63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&amp;sdata=TZeoR%2FT22hg%2BkG0LllDEttlgdzv%2FAsH8PeXbdEHQ9Mk%3D&amp;reserved=0 looks useful, as a guide for applications running over QUIC.  It warns that connections can time out fairly quickly.  For timely callbacks over NFS sessions, that means we need the client to ping the server regularly.
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Fdraft-ietf-quic-http-09%23section-5&amp;data=0
>>> 2%7C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506
>>> cee63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63734777607589360
>>> 4&amp;sdata=rF91Tq8OXHeN7W5iuSbtYotq1XBC4Zc8DGEGzwPUuTg%3D&amp;reser
>>> ved=0
>>> 
>>> QUIC allows multiple streams per connection--I wonder how we might 
>>> use that.  RFC 5661 justifies the requirement for an ordered transport with:
>>> 
>>> So as long as we don't try to split a single RPC among streams, I 
>>> think we're OK.  Would a stream per session slot be reasonable?  I'm 
>>> not sure what the cost of a stream is.
>>> 
>>> specify QUIC endpoints when necessary?  (For server-to-server-copy, 
>>> pnfs file layouts, fs_locations, etc.)  All QUIC needs is an IP 
>>> address and maybe a port, so maybe the existing UDP/TCP addresses are enough?
>> 
>> --
>> Chuck Lever
>> [email-masked]

--
Chuck Lever
[email-masked]