Subject: Re: [nfsv4] Benjamin Kaduk's Discuss on draft-ietf-nfsv4-rfc5661sesqui-msns-03: (with DISCUSS and COMMENT)

From: David Noveck <[email-masked]>
Date: Thu, 2 Jan 2020 10:08:23 -0500

On Sun, Dec 22, 2019 at 9:14 PM Benjamin Kaduk <[email-masked]> wrote:

> On Fri, Dec 20, 2019 at 09:46:22AM -0500, David Noveck wrote:
> > > ----------------------------------------------------------------------
> > > DISCUSS:
> > > ----------------------------------------------------------------------
> > >
> >
> > 11.5.4.1.  File System Trunking Presented as Replication
> >
> >    In some situations, a file system location entry may indicate a file
> >    system access path to be used as an alternate location, where
> >    trunking, rather than replication is to be used.  The situtions in
>
> nit: I think we need a comma after "replication" (or to put all of "rather
> than replication" in parentheses).
>

Went with the comma.

> nit: s/situtions/situations/
>

Fixed.

>
> >    which this is appropriate are limited to those in which both of the
> >    following are true.
> >
> >    o  The two file system locations (i.e the one on which the location
>
> nit: comma after "i.e."
>

OK.

>
> >       attribute is obtained and the one specified in the file system
> >       location entry) designate the same locations within their
> >       respective single-server namespaces.
> >
> >    o  The two server network address(i.e. the one being used to obtain
>
> ditto, and space before open paren.
>

Fixed.

>
> >       the location attribute and the one specified in the file system
> >       location entry) designate the same server (as indicated by the
> >       same value of the so_major_id field of the eir_server_owner field
> >       returned in response to EXCHANGE_ID).
> >
> >    When these conditions hold, access to using both access paths is
>
> nit: s/to // (though the double "access" is still perhaps unfortunate)
>

Went with "operations using both access paths are"

>
> >    genrally trunked, although, when the attribute fs_locations_info is
> >    used, trunking msy be disallowed:
>
> nit: s/msy/may/
>

Fixed.

>
> >    o  When the fs_locations_info attribute shows the two entries as not
> >       having the same simultaneous-use class, trunking is inhibited and
> >       the two access paths cannot be used totgether.
> >      In this case the two paths can be used serially with no transition
> >       activity required on the part of the client.  In this case, any
> >       transition between access paths is transparent, and the client in
> >       transferring access from one to the other is acting as it would in
> >       the event that communication is interrupted, with a new connection
> >       and possibly a new session being established to continue access to
> >       the same file system.
>
> I think I've lost the context for how "no transition activity" meshes with
> "new connection and possibly a new session" -- it it mostly just "no need
> for recovery" and "use the same client ID"?
>

Yes.

I think I will chage "transition activity' to "recovery activity" and add
"using the
existing client id" to the end of the last sentence.

>
> >    o  Note that for two such location entries, any information within
> >       the fs_locations_info attribute that indicates the need for
> >       special transition activity, i.e., the appearance of the two file
> >       system location entries with different handle, fileid, write-
> >       verifier, change, and readdir classes, indicates a serious
> >       problem.  The client, if it allows transition to the file system
> >       instance at all, must not treat any transition as a transparent
> >       one.  The server SHOULD NOT indicate that these entries belong to
> >       different handle, fileid, write-verifier, change, and readdir
>
> Pedagogically, we might want to be more explicit about "these entries",
> perhaps along the lines of "multiple entries for the same file system on
> the same server" (if I understand correctly).
>

RIght.

I'm thinking of "these two entries (for the same file system on the same
server)"

>
> >       classes, whether or not the two entries are shown belonging to the
> >       same simultaneous-use class.
> >
> >    This situation was recognized by [62], even though that document made
> >    no explicit mention of trunking.
> >
> >    o  It treated the situation that we describe as trunking as one of
> >       simultaneous use of two distinct file system instances, even
> >       though, in the explanatory framework now used to describe the
> >       situstion, the case is one in which a single file system is
> >       accessed by two different trunked addresses.
> >
> >    o  It treated the situation in which the two paths are to be used
> >       serially as a special sort of "transparent transition", while in
> >       the descriptve framework now used to categorize transition
> >       situations, this is a case of a "network endpoint transition" (see
> >       Section 11.9.
> >
> > 11.6.  Trunking without File System Location Information
> >
> >    In situation in which a file system is accessed using two server-
>
> nit: missing article ("the situation"?)
>

A think a plural is better here.

>
> >    trunkableaddresses (as indicated by the same value of the so_major_id
>
nit: s/trunkableaddresses/trunkable addresses/

Fixed.

>
> >    field of the eir_server_owner field returned in response to
> >    EXCHANGE_ID), trunked access is allowed even though there might not
> >    be any location entries specfically indicating the use of trunking
> >    for that file system.
> >
> >    This situation was recognized by [62], even though that document made
> >    no explicit mention of trunking and treated the situation as one of
> >    simultaneous use of two distinct file system instances, even though,
> >    in the explanatory framework now used to describe the situstion, the
> >    case is one in which a single file system is accessed by two
> >    different trunked addresses.
> >
> > In addition I anticipate rewriting the first paragraph of Section
> 11.10.1 to
> > read as follows:
> >
> >    The fs_locations_info attribute (described in Section 11.17) may
> >    indicate that two replicas may be used simultaneously, although some
> >    situations in which such simultaneous access is permitted are more
> >    appropritaely described as instances of trunking (see
>
> nit: s/appropritaely/appropriately/
>

Fixed.

>
> >    Section 11.5.4.1).  Although situations in which multiple replicas
> >    may be accessed simultaneously are somewhat similar to those in which
> >    a single replica is accessed by multiple network addresses, there are
> >    important differences, since locking state is not shared among
> >    multiple replicas.
>
> Thanks; this looks really good (nits aside).
>
> >
> > > This is somewhat awkward since the limited nature of the update results
> > > in my not having the full context of the rest of the document; with
> that
> > > limitation in my understanding in mind, I'd like to confirm that we're
> > > comfortable with the use of "network address" in the context of
> > > trunking/migration, specifically the extent to which we do not discuss
> > > port numbers.
> >
> > Now that you point this out, I guess I shouldn't have been as
> comfortable as
> > I had been.
> >
> > > The relevant XDR types do allow for optional port numbers
> > > to be included, with a default to be used when not specified,
> >
> > The xdr types you refer to are not relevant here, since we have, in the
> > interest of allowing hostname as well as IP addresses,specfied these
> > IP adresses, when IP addresses are used in the form of ASCII text.
> > Nevertheless, given that port numbers can be given, optionally after a
> > colon, the essence of the issue is as you described.
> >
> > > but in
> > > this document we do have a new note that different ports may be used
> for
> > > different connection types to the same logical server, and also that
> > > different ports "is not the essence of the distinction between the two
> > > endpoints".  I think there might be cases where the port is relevant
> for
> > > a distinction, but the main ones I can think of are of questionable
> > > relevance (essentially, roughly equivalent to multiple userspace NFS
> > > servers on a single host but in different trust/privilege domains) --
> > > I'd like another opinion or several.
> >
> > I agree that your major case is of limited use, but that doesn't mean
> that
> > it should be ignored.
> >
> > What I'm proposing doing is adding the following as a new paragraph
> > associated with the second bullet in the second set of bullets in Section
> > 11.1.2 (i.e. after <vspace> in xml2rfc v2), to read as follows:
> >
> >       The network addresses used in file system location entries
> >       typically appear without port number indications and are used to
> >       designate a server at one of the standard ports for NFS access,
> >       i.e. 2049 or 20049 for use with RPC-over-RDMA.  Port numbers may
> >       be used in file system location entries to designate servers
> >       (typically user-level ones) accesed using other port numbers.  In
> >       the case if network addresses indicating trunking relationships,
>
> nit: should this be s/if/of/?  Though, I might go with "the case where
> network addresses indicate", myself.
>

I think that's better.  Will use.

>
> >       use of explcit port number is inappropriate since trunking is a
>
> nit: s/explcit/explicit/
>

Will fix.

>
> >       relationsip btetween network addresses.  See Section 11.5.2 for
>
> nit: s/btetween/between/
>

Will fix: "relationship between".

>
> >       details.
> >
> > I also anticipate revising the second bullet in section 11.5.2 to read as
> > foollows:
> >
> >      The client may fetch the file system location attribute for the
> >       file system.  This will provide either the name of the server
> >       (which can be turned into a set of network addresses using DNS),
> >       or a set of server-trunkable location entries.  Using the latter
> >       alternative, the server can provide addresses it regards as
> >       desirable to use to access the file system in question.  Although
> >       these enties can contain port numbers, these port numbers are not
> >       used in determining trunking relationships.  Once the cadidate
> >       addresses have been determined and EXCHANGE_ID done to the proper
> >       server, only the value of the so_major field returned by the
> >       serrvers in question determines whether a trunking relationship
>
> nit: s/serrvers/servers/
>

Will fix.

>
> >       actually exists.
>
> Thanks; that seems a pretty elegant way to address the topic without
> causing churn throughout large swathes of text.
>
> Though now I do find myself wondering whether there's anything to keep a
> rogue (i.e., attacker-controlled) server from falsely claiming the so_major
> field that matches an honest server and tricking a client into thinking
> that the attacker's server is trunkable with the honest one.  The GSS
> identity check would, or course, work if available, but AFAICT the non-GSS
> case remains about as bad as, well, the non-GSS case in general.  So I
> don't feel a need to hold up this document while we figure out more to say
> than we already do in (e.g.) Appendix C.
>

Right.  In future, I hope we will be able to refer to this as "the cases in
which the
server is not authenticated".

>
> >
> > > In a similar "discuss discuss" vein, Section 11.10.8 describes a
> > > scenario that does not give much clarity, at a protocol level, into
> what
> > > degree of replication synchronization a client can expect from a given
> > > file system that advertises multiple replicas.  I recognize that this
> is
> > > de facto just stating the deployed reality, but it's also hard to feel
> > > good about having this level of ambiguity in a propsed standard,
> >
> > It gets easier over time.  Sigh!  This would be the fourth Proposed
> > Standard with this issue :-(
> >
> > Still I think we can do a bit better by relyimg on the three special
> cases
> > listed at
> > the end of that section and adding something like the following after the
> > list:
> >
> >    When none of these special situations apply, there is no basis,
> >    within the protool, for the client making assumptions about the
> >    contents of a replica file ststem and its relationship to previous
> >    file sytem instances.  This may mean that switching between file
> >    system that are not read-only is not available, where either the
>
> Hmm, do we want to add a descriptor like "nominally identical" to "file
> systems" (note: plural)
>

Will do.

>
> >    client does not use or the server does not support the
> >    fs_locations_info atribute.
>
> That helps some.  We might consider a note in the first paragraph (near
> "The namespace will typically be constructed") about the details of that
> being configured in an out-of-band manner.
>

I don't think details can be provided since this is pretty much up to the
server although we can make it clear that the server constructs this on
the basis of what it knows about the file system and their characteristics,
i.e information not really part of the protocol.

How about:.

   The server is responsible for contructing an appropriate namespace.
   Typically, it is contructed so that applications can choose an
appropriate
   level of support, so that in one position in  the namespace a varied set
   of replicas might  be listed, while in another only those that are
up-to-date
   would be considered replicas.

I'll think about doing that.

>
> To be clear, I'll drop the discuss point regardless; I'm happy with the
> exchange we've had here and the direction that your new paragraph is going.
>
> > > and the (unchanged) text in Section 11.5.5 seems to impose a stricter
> > > consistency requirement, at least on potential migration targets.  (A
> > > bit more detail in the COMMENT section.)
> >
> > That is a different and has been since the multi-server
> > namespace features were first added to NFSv4 decades ago.
> > It is inherently easier to synchronize A and B when you know that A
> > is over befoe B begins.
>
> Okay.  (I hope you can understand that trying to review just the diff left
> some of the other portions a little fuzzy to me.)
>
>
> -Ben
>
> > On Wed, Dec 18, 2019 at 3:32 AM Benjamin Kaduk via Datatracker <
> > [email-masked]> wrote:
> >
> > > Benjamin Kaduk has entered the following ballot position for
> > > draft-ietf-nfsv4-rfc5661sesqui-msns-03: Discuss
> > >
> > > When responding, please keep the subject line intact and reply to all
> > > email addresses included in the To and CC lines. (Feel free to cut this
> > > introductory paragraph, however.)
> > >
> > >
> > > Please refer to
> https://www.ietf.org/iesg/statement/discuss-criteria.html
> > > for more information about IESG DISCUSS and COMMENT positions.
> > >
> > >
> > > The document, along with other ballot positions, can be found here:
> > > https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5661sesqui-msns/
> > >
> > >
> > >
> > > ----------------------------------------------------------------------
> > > DISCUSS:
> > > ----------------------------------------------------------------------
> > >
> > > Thank you for this document (and its predecessor); it's important to
> get
> > > these points clarified, and sooner rather than later.  I expect that
> the
> > > following few issues should be quickly resolvable.
> > >
> > > Section 11.10.1 includes a reference to "Section 11.7.2.1 of RFC5661",
> > > but this document is obsoleting that document.  It seems internally
> > > inconsistent to both obsolete and depend on the same source -- if we
> rely
> > > on that content, it should be included in this document.
> > >
> > > This is somewhat awkward since the limited nature of the update results
> > > in my not having the full context of the rest of the document; with
> that
> > > limitation in my understanding in mind, I'd like to confirm that we're
> > > comfortable with the use of "network address" in the context of
> > > trunking/migration, specifically the extent to which we do not discuss
> > > port numbers.  The relevant XDR types do allow for optional port
> numbers
> > > to be included, with a default to be used when not specified, but in
> > > this document we do have a new note that different ports may be used
> for
> > > different connection types to the same logical server, and also that
> > > different ports "is not the essence of the distinction between the two
> > > endpoints".  I think there might be cases where the port is relevant
> for
> > > a distinction, but the main ones I can think of are of questionable
> > > relevance (essentially, roughly equivalent to multiple userspace NFS
> > > servers on a single host but in different trust/privilege domains) --
> > > I'd like another opinion or several.
> > >
> > > In a similar "discuss discuss" vein, Section 11.10.8 describes a
> > > scenario that does not give much clarity, at a protocol level, into
> what
> > > degree of replication synchronization a client can expect from a given
> > > file system that advertises multiple replicas.  I recognize that this
> is
> > > de facto just stating the deployed reality, but it's also hard to feel
> > > good about having this level of ambiguity in a propsed standard, and
> the
> > > (unchanged) text in Section 11.5.5 seems to impose a stricter
> > > consistency requirement, at least on potential migration targets.  (A
> > > bit more detail in the COMMENT section.)
> > >
> > > Section 11.13.2 mentions that "[i]ssues connected with a client
> > > impersonating another by presenting another client's id string are
> > > discussed in Section 21", but I failed to find this discussion in
> > > Section 21.  (The discuss-level issue is just the internal
> > > inconsistency; there's a decent argument that this is covered by
> > > Appendix C's "not written in accord with RFC3552".  Though if the text
> > > was already written for draft-ietf-nfsv4-mv1-msns-update, not including
> > > it here seems a little silly.)
> > >
> > >
> > > ----------------------------------------------------------------------
> > > COMMENT:
> > > ----------------------------------------------------------------------
> > >
> > > I think I may have mistakenly commented on some sections that are
> > > actually just moved text, since my lookahead window in the diff was too
> > > small.  I expect it's most appropriate to defer those to for the full
> > > -bis, so sorry to have them lumped in here.
> > >
> > > Thank you for all the effort put in to get the diff against RFC 5661 to
> > > be minimal!  I know that the current default output formatting is
> rather
> > > different than what is done in RFC 5661, but this diff is pretty easy
> to
> > > read.
> > >
> > > Thank you also for the detailed discussion in Appendix C; I do not
> think
> > > I could add anything more!  While the security posture of the current
> > > deployed state of NFSv4 is not great (though, arguably, somewhat
> > > understandable given the path we took to get there), this is the right
> > > start to making any sort of improvement.
> > >
> > > Since the "Updates:" header is part of the immutable RFC text (though
> > > "Updated by:" is mutable), we should probably explicitly state that
> "the
> > > updates that RFCs 8178 and 8434 made to RFC 5661 apply equally to this
> > > document".
> > >
> > > I note inline (in what is probably too many places; please don't reply
> > > at all of them!) some question about how clear the text is that a file
> > > system migration is something done at a per-file-system granularity,
> and
> > > that migrating a client at a time is not possible.  As was the case for
> > > my Discuss point about addresses/port-numbers, I'm missing the context
> > > of the rest of the document, so perhaps this is a non-issue, but the
> > > consequences of getting it wrong seem severe enough that I wanted to
> > > check.
> > >
> > > Does a client have any way to know in advance that two addresses will
> be
> > > session-trunkable other than the one listed in Section 11.1.1 that
> "when
> > > two connections of different connection types are made to the same
> > > network address and are based on a single file system location entry
> > > they are always session-trunkable"?  It seems like mostly we're
> defining
> > > the property by saying that the client has to try it and see if it
> > > works; I'd love to be wrong about that.
> > >
> > > Section 1.1
> > >
> > >    The revised description of the NFS version 4 minor version 1
> > >    (NFSv4.1) protocol presented in this update is necessary to enable
> > >    full use of trunking in connection with multi-server namespace
> > >    features and to enable the use of transparent state migration in
> > >    connection with NFSv4.1.  [...]
> > >
> > > nit: do we expect all readers to know what is meant by "trunking" with
> > > no other lead-in?
> > >
> > >    This limited scope update is applied to the main NFSv4.1 RFC with
> the
> > >
> > > nit: hyphenate "limited-scope"
> > >
> > >    scope as could expected by a full update of the protocol.  Below are
> > >    some areas which are known to need addressing in a future update of
> > >    the protocol.
> > >    [...]
> > >
> > > side note: I'd be interested in better understanding the preference for
> > > the subjunctive verb tense for most of these points ("work would have
> to
> > > be done"); my naive expectation would be that since there are plans to
> > > undertake the work, just "work needs to be done" or "work will be done"
> > > might suffice.
> > >
> > >    o  Work would have to be done with regard to RFC8178 [63] which
> > >       establishes NFSv4-wide versioning rules.  As RFC5661 is curretly
> > >       inconsistent with this document, changes are needed in order to
> > >       arrive at a situation in which there would be no need for RFC8178
> > >       to update the NFSv4.1 specfication.
> > >
> > > nit: s/this document/that document/ -- "this document" is
> > > draft-ietf-nfsv4-rfc5661sesqui-msns.
> > >
> > >    o  Work would have to be done with regard to RFC8434 [66], which
> > >       establishes the requirements for pNFS layout types, which are not
> > >       clearly defined in RFC5661.  When that work is done and the
> > >       resulting documents approved, the new NFSv4.1 specfication
> > >       document will provide a clear set of requirements for layout
> types
> > >       and a description of the file layout type that conforms to those
> > >       requirements.  Other layout types will have their own
> specfication
> > >       documents that conforms to those requirements as well.
> > >
> > > It's not entirely clear to me that the other layout types need to get
> > > mentioned in this document; how do they relate to the formal status of
> > > the "current NFSv4.1 core protocol specification document"?
> > >
> > >    o  Work would have to be done to address many erratas relevant to
> RFC
> > >       5661, other than errata 2006 [60], which is addressed in this
> > >       document.  That errata was not deferrable because of the
> > >       interaction of the changes suggested in that errata and handling
> > >       of state and session migration.  The erratas that have been
> > >       deferred include changes originally suggested by a particular
> > >       errata, which change consensus decisions made in RFC 5661, which
> > >       need to be changed to ensure compatibility with existing
> > >       implementations that do not follow the handling delineated in RFC
> > >       5661.  Note that it is expected that such erratas will remain
> > >
> > > This sentence is pretty long and hard to follow; maybe it could be
> split
> > > after "change consensus decisions made in RFC 5661" and the second half
> > > start with a more declarative statement about existing implementations?
> > > (E.g., "Existing implementations did not perform handling as
> delineated in
> > > RFC
> > > 5661 since the procedures therein were not workable, and in order to
> > > have the specification accurately reflect the existing deployment base,
> > > changes are needed [...]")
> > >
> > >       relevant to implementers and the authors of an eventual
> > >       rfc5661bis, despite the fact that this document, when approved,
> > >       will obsolete RFC 5661.
> > >
> > > (I assume the RFC Editor can tweak this line to reflect what actually
> > > happens; my understanding is that the errata reports will get cloned to
> > > this-RFC.)
> > > [rant about "errata" vs. "erratum" elided]
> > >
> > > Section 2.10.4
> > >
> > >    Servers each specify a server scope value in the form of an opaque
> > >    string eir_server_scope returned as part of the results of an
> > >    EXCHANGE_ID operation.  The purpose of the server scope is to allow
> a
> > >    group of servers to indicate to clients that a set of servers
> sharing
> > >    the same server scope value has arranged to use compatible values of
> > >    otherwise opaque identifiers.  Thus, the identifiers generated by
> two
> > >    servers within that set can be assumed compatible so that, in some
> > >    cases, identifiers generated by one server in that set may be
> > >    presented to another server of the same scope.
> > >
> > > Is there more that we can say than "in some cases"?  The previous text
> > > implies a higher level of reliability than just "some cases", to me.
> > >
> > > Section 2.10.4
> > >
> > > I see the list of identifier types for which same-scope compatibility
> > > applies got reduced from RFC 5661 to this document, by removing session
> > > ID, client ID, and state ID values.  For at least one of those I can
> see
> > > this making sense as only being workable when the server really is "the
> > > same server", inline with the improved discussion of migration vs.
> > > trunking that is a main focus of this document.  Does that
> > > justification apply to all of them, or are there more reasons involved?
> > >
> > > We also remove the text about a client needing to compare server scope
> > > values during a potential migration event, to determine whether the
> > > migration preserved state or a reclaim is needed.  I thought this
> > > scenario would still be possible (and thus still need to be listed),
> > > though perhaps we are claiming that it is so under-specified so as to
> be
> > > never workable in practice?
> > >
> > > Section 2.10.5
> > >
> > >    o  When eir_server_scope changes, the client has no assurance that
> > >       any id's it obtained previously (e.g. file handles, state ids,
> > >       client ids) can be validly used on the new server, and, even if
> > >
> > > It's interesting to see file handles, state ids, and client ids listed
> > > together here (nit: also with lowercase "id"), when in the previous
> > > section we have removed state IDs and client IDs from a list that
> > > includes all three in RFC 5661.
> > >
> > >    o  When eir_server_scope remains the same and
> > >       eir_server_owner.so_major_id changes, the client can use the
> > >       filehandles it has, consider its locking state lost, and attempt
> > >       to reclaim or otherwise re-obtain its locks.  It may find that
> its
> > >       file handle IS now stale but if NFS4ERR_STALE is not received, it
> > >       can proceed to reclaim or otherwise re-obtain its open locking
> > >       state.
> > >
> > > nit(?): this bit about "It may find that its file handle IS now stale
> > > but if NFS4ERR_STALE is not received" seems to assume some familiarity
> > > by the reader as to what actions would be performed that would get
> > > NFS4ERR_STALE back.
> > >
> > > Section 2.10.5.1
> > >
> > >    When the server responds using two different connections claim
> > >    matching or partially matching eir_server_owner, eir_server_scope,
> > >
> > > nit: The grammar got wonky here; maybe s/claim/claiming/?
> > >
> > > Section 11.1.1
> > >
> > >       In the case of NFS version 4.1 and later minor versions, the
> means
> > >       of trunking detection are as described in this document and are
> > >       available to every client.  Two network addresses connected to
> the
> > >       same server are always server-trunkable but cannot necessarily be
> > >       used together to access a single session.
> > >
> > > nit: we haven't defined "server-trunkable" yet, so it may be worth a
> > > hint that the definition is coming soon.
> > >
> > >    The combination of a server network address and a particular
> > >    connection type to be used by a connection is referred to as a
> > >    "server endpoint".  Although using different connection types may
> > >    result in different ports being used, the use of different ports by
> > >    multiple connections to the same network address is not the essence
> > >    of the distinction between the two endpoints used.
> > >
> > > There's perhaps a fine line to walk here, as the port can still have
> > > significant relevance, in general, and we are frequently in the IETF
> > > told to make no assumption about what is behind specific port values at
> > > a given network address.  (Consider, for example, a hypothetical
> virtual
> > > hosting service that provides "DS-as-a-service" where customers run
> > > their own MDS that point to configured DSes for actual storage.
> > > Different ports on that cloud provider would represent entirely
> > > different customers/servers!)  [This became a discuss point but it
> > > didn't end up including all the discussion here, so I left it as an
> > > informational thing; discussion should happen in the Discuss section]
> > >
> > > Section 11.1.2
> > >
> > >    o  In some cases, a server will have a namespace more extensive than
> > >       its local namespace by using features associated with attributes
> > >       that provide file system location information.  These features,
> > >       which allow construction of a multi-server namespace are all
> > >
> > > nit: comma after "multi-server namespace".
> > >
> > >    o  A file system present in a server's pseudo-fs may have multiple
> > >       file system instances on different servers associated with it.
> > >       All such instances are considered replicas of one another.
> > >
> > > [Some readers might take this as requiring live read/write replication
> > > such that all writes to any instance are immediately visible on all
> > > other instances.  The rest of the document ought to disabuse them of
> > > that notion, and yet...]
> > >
> > >    o  File system location entries provide the individual file system
> > >       locations within the file system location attributes.  Each such
> > >       entry specifies a server, in the form of a host name or IP an
> > >       address, and an fs name, which designates the location of the
> file
> > >
> > > nit: s/IP an/an IP/.
> > >
> > >       client may establish connections.  There may be multiple
> endpoints
> > >       because a host name may map to multiple network addresses and
> > >       because multiple connection types may be used to communicate with
> > >       a single network address.  However, all such endpoints MUST
> > >       provide a way of connecting to a single server.  The exact form
> of
> > >
> > > nit: "MUST provide" feels strange here, since it implies in some sense
> > > an extra layer of indirection ("A lists X, and X among other things
> > > provides Y"); would a different word like "indicate" work?
> > >
> > >       element derives from a corresponding location entry.  When a
> > >       location entry specifies an IP address there is only a single
> > >       corresponding location element.  File system location entries
> that
> > >       contain a host name are resolved using DNS, and may result in one
> > >       or more location elements.  All location elements consist of a
> > >       location address which is the IP address of an interface to a
> > >       server and an fs name which is the location of the file system
> > >       within the server's local namespace.  The fs name can be empty if
> > >
> > > I can't decide whether both instances of "IP address" are pedantically
> > > correct, in the presence of the potential for port information to be
> > > included/available.  The former is probably okay, but the latter might
> > > need some clarification.
> > >
> > > Section 11.2
> > >
> > >    The fs_locations attribute defined in NFSv4.0 is also a part of
> > >    NFSv4.1.  This attribute only allows specification of the file
> system
> > >    locations where the data corresponding to a given file system may be
> > >    found.  Servers should make this attribute available whenever
> > >    fs_locations_info is supported, but client use of fs_locations_info
> > >    is preferable, as it provides more information.
> > >
> > > I think this was probably okay as "SHOULD make this attribute
> available"
> > > (as it was in 5661), but don't object to the lowercase version either.
> > >
> > > Section 11.5
> > >
> > >    Where a file system had been absent, specification of file system
> > >
> > > I guess I'm probably in the rough on this one (since 5661 had my
> > > more-preferred language), but it still feels like "had been absent"
> > > implies that it is no longer absent, i.e., that it is now present or
> has
> > > otherwise changed.  What's going on here with referrals is more like a
> > > "was never present" case, though using "never" is of course problematic
> > > as it's more absolute than is appropriate.
> > >
> > >
> > > If we're going to talk about "pure referral"s, do we want to make
> > > mention of or otherwise differentiate/characterize "non-pure"
> > > ("impure"?) referrals?
> > >
> > > Section 11.5.1
> > >
> > >    In order to simplify client handling and allow the best choice of
> > >    replicas to access, the server should adhere to the following
> > >    guidelines.
> > >
> > > Just to check: these are just informal "guidelines" and not something
> > > that a server SHOULD or even MUST adhere to?
> > >
> > > Section 11.5.2
> > >
> > >    Locations entries used to discover candidate addresses for use in
> > >
> > > nit(?): is this supposed to just be "Location" singular?
> > >
> > > Section 11.5.3
> > >
> > >    Irrespective of the particular attribute used, when there is no
> > >    indication that a step-up operation can be performed, a client
> > >    supporting RDMA operation can establish a new RDMA connection and it
> > >    can be bound to the session already established by the TCP
> > >    connection, allowing the TCP connection to be dropped and the
> session
> > >    converted to further use in RDMA node.
> > >
> > > Should we say something to make this contingent on the server also
> > > supporting RDMA?
> > >
> > > Section 11.5.5
> > >
> > >    will typically use the first one provided.  If that is inaccessible
> > >    for some reason, later ones can be used.  In such cases the client
> > >    might consider that the transition to the new replica as a migration
> > >    event, even though some of the servers involved might not be aware
> of
> > >    the use of the server which was inaccessible.  In such a case, a
> > >
> > > nit: the grammar here got wonky; maybe s/as a/is a/?
> > >
> > > Section ??
> > >
> > > The old (RFC 5661) Section 11.5 mentioned several things, and I'd like
> > > to check that we have either covered or disavowed all of them.
> > > My current understanding is that:
> > >
> > > The first paragraph basically talked about trunking detection, and is
> > > covered elsewhere.
> > >
> > > The second paragraph talks about something that I would call "implicit
> > > replication" with the 5661 definition of "replica", but in the new
> model
> > > is essentially definitionally true, since we consider all addresses for
> > > the same server to be ... part of the same server, so of course that
> > > server's namespaces match up.  Though perhaps the discussion about not
> > > all of the cartesian product of (addresses-for-server, local path)
> being
> > > listed is still worth having?
> > >
> > > The third paragraph basically talks about the need for trunking
> > > detection, and includes some guidance to clients about assuming server
> > > misconfiguration that seems of questionable merit.
> > >
> > > Section 11.5.7
> > >
> > >    o  Deletions from the list of network addresses for the current file
> > >       system instance need not be acted on immediately, although the
> > >       client might need to be prepared for a shift in access whenever
> > >       the server indicates that a network access path is not usable to
> > >       access the current file system, by returning NFS4ERR_MOVED.
> > >
> > > I think this should be wordsmithed a bit more, as (IIUC) the idea here
> > > is that if a client notices in a location response that the address the
> > > client is currently using for a filesystem has disappeared from the
> > > list, the client should be prepared for imminent changes in server
> > > behavior relating to the presumed-move.  Those imminent changes would
> > > most likely be reflected in the form of the server returning
> > > NFS4ERR_MOVED, but there is no NFS4ERR_MOVED involved in the actual
> > > deletion from the list of network instances of the current system
> > > instance.
> > >
> > > Section 11.6
> > >
> > >    corresponding attribute is interrogated subsequently.  In the case
> of
> > >    a multi-server namespace, that same promise applies even if server
> > >    boundaries have been crossed.  Similarly, when the owner attribute
> of
> > >    a file is derived from the securiy principal which created the file,
> > >    that attribute should have the same value even if the interrogation
> > >    occurs on a different server from the file creation.
> > >
> > > I can see how the interrogation would be on a different server from
> file
> > > creation for "simple" replication scenarios, but I'm not sure I'm
> seeing
> > > how non-replication cases would arise, paritulcarly that cross server
> > > boundaries in a multi-server (hierarchical?) namespace.  Am I missing
> > > something obvious?
> > > nit: s/securiy/security/
> > >
> > >    o  All servers support a common set of domains which includes all of
> > >       the domains clients use and expect to see returned as the domain
> > >       portion of an owner or group in the form "id@domain".  Note that
> > >       although this set most ofen consists of a single domain, it is
> > >       possible for mutiple domains to be supported.
> > >
> > > I a little bit wonder if the "most often" still holds when client
> > > principals come from an AD forest.
> > >
> > >    o  All servers recognize the same set of security principals, and
> > >       each principal, the same credential are required, independent of
> > >       the server being accessed.  In addition, the group membership for
> > >
> > > nit: I think there's a missing word here, maybe "and for each
> > > principal"?
> > >
> > >    Note that there is no requirment that the users corresponding to
> > >
> > > nit: "requirement"
> > >
> > >    o  The "local" representation of all owners and groups must be the
> > >       same on all servers.  The word "local" is used here since that is
> > >       the way that numeric user and group ids are described in
> > >       Section 5.9.  However, when AUTH_SYS or stringified owners or
> > >       group are used, these identifiers are not truly local, since they
> > >       are known tothe clients as well as the server.
> > >
> > > I am trying to find a way to note that the AUTH_SYS case mentioned here
> > > is precisely because of the requirement being imposed by this bullet
> > > point, while acknowledging that the "stringified owners or group" case
> > > is separate, but not having much luck.
> > > Also, nit: "to the"
> > >
> > > Section 11.9
> > >
> > >    o  When use of a particular address is to cease and there is also
> one
> > >       currently in use which is server-trunkable with it, requests that
> > >       would have been issued on the address whose use is to be
> > >       discontinued can be issued on the remaining address(es).  When an
> > >       address is not a session-trunkable one, the request might need to
> > >       be modified to reflect the fact that a different session will be
> > >       used.
> > >
> > > I suggest writing this as "when an address is server-trunkable but not
> > > session-trunkable,".
> > >
> > >    o  When use of a particular connection is to cease, as indicated by
> > >       receiving NFS4ERR_MOVED when using that connection but that
> > >       address is still indicated as accessible according to the
> > >       appropriate file system location entries, it is likely that
> > >       requests can be issued on a new connection of a different
> > >       connection type, once that connection is established.  Since any
> > >       two server endpoints that share a network address are inherently
> > >       session-trunkable, the client can use BIND_CONN_TO_SESSION to
> > >       access the existing session using the new connection and proceed
> > >       to access the file system using the new connection.
> > >
> > > I'm not entirely sure how "inherent" this is (in the vein of my Discuss
> > > point, and what we mean by "network address").
> > >
> > >    o  When there are no potential replacement addresses in use but
> there
> > >
> > > What is a "replacement address"?
> > >
> > >       are valid addresses session-trunkable with the one whose use is
> to
> > >       be discontinued, the client can use BIND_CONN_TO_SESSION to
> access
> > >       the existing session using the new address.  Although the target
> > >       session will generally be accessible, there may be cases in which
> > >       that session is no longer accessible.  In this case, the client
> > >       can create a new session to enable continued access to the
> > >       existing instance and provide for use of existing filehandles,
> > >       stateids, and client ids while providing continuity of locking
> > >       state.
> > >
> > > I'm not sure I understand this last sentence.  On its own, the "new
> > > session to enable continued access to the existing instance" sounds
> like
> > > the continued access would be on the address whose use is to cease, and
> > > thus the new session would be there.  But why make a new session when
> > > the old one is still good, especially when we just said in the previous
> > > sentence that the old session can't be moved to the new
> > > connection/address?
> > > Perhaps a forward reference down to Section 11.12.{4,5} for this and
> the
> > > next bullet point would help as well as rewording?
> > >
> > > Section 11.10.6
> > >
> > >    In a file system transition, the two file systems might be clustered
> > >    in the handling of unstably written data.  When this is the case,
> and
> > >
> > > What does "clustered in the handling of unstably written data" mean?
> > >
> > >    the two file systems belong to the same write-verifier class, write
> > >
> > > How is the client supposed to determine "when this is the case"?
> > >
> > > Section 11.10.7
> > >
> > >    In a file system transition, the two file systems might be
> consistent
> > >    in their handling of READDIR cookies and verifiers.  When this is
> the
> > >    case, and the two file systems belong to the same readdir class,
> > >
> > > As above, how is the client supposed to determine "when this is the
> > > case"?
> > >
> > >    READDIR cookies and verifiers from one system may be recognized by
> > >    the other and READDIR operations started on one server may be
> validly
> > >    continued on the other, simply by presenting the cookie and verifier
> > >    returned by a READDIR operation done on the first file system to the
> > >    second.
> > >
> > > Are these "may be"s supposed to admit the possibility that the
> > > destination server can just decide to not honor them arbitrarily?
> > >
> > > Section 11.10.8
> > >
> > >    the degree indicated by the fs_locations_info attribute).  However,
> > >    when multiple file systems are presented as replicas of one another,
> > >    the precise relationship between the data of one and the data of
> > >    another is not, as a general matter, specified by the NFSv4.1
> > >    protocol.  It is quite possible to present as replicas file systems
> > >    where the data of those file systems is sufficiently different that
> > >    some applications have problems dealing with the transition between
> > >    replicas.  The namespace will typically be constructed so that
> > >    applications can choose an appropriate level of support, so that in
> > >    one position in the namespace a varied set of replicas will be
> > >    listed, while in another only those that are up-to-date may be
> > >    considered replicas.  [...]
> > >
> > > This seems quite wishy-washy for a standards-track protocol!  We give
> no
> > > hard bounds on how "different" replicas may be, no protocol element to
> > > convey even a qualitative sense of where on the spectrum of replication
> > > fidelity a replica may lie, and no indication as to how the namespace
> > > might be constructed to indicate a level of support.
> > >
> > >                          The protocol does define three special cases
> of
> > >    the relationship among replicas to be specified by the server and
> > >    relied upon by clients:
> > >
> > > I'd like to hear from the rest of the IESG, but we may need to consider
> > > limiting "replication" to just these special cases until we can be more
> > > precise about the other cases.
> > >
> > >    o  When multiple replicas exist and are used simultaneously by a
> > >       client (see the FSLIB4_CLSIMUL definition within
> > >       fs_locations_info), they must designate the same data.  Where
> file
> > >       systems are writable, a change made on one instance must be
> > >       visible on all instances, immediately upon the earlier of the
> > >       return of the modifying requester or the visibility of that
> change
> > >       on any of the associated replicas.  This allows a client to use
> > >
> > > Hmm, how would this "earlier of [...]" work when there are three
> > > nominally equivalent machines?  Assume the RPC is made to A, and the
> > > other two are B and C.  If the update first goes visible on B, it must
> > > also be visible on C, instilling what is apparently a hard requirement
> > > for exact synchronization between B an C, perhaps by some sort of
> > > negotiated "make visible at timestamp X" mechanism.  But if the RPC
> > > returns from A first, then the change still has to be visible on B and
> C
> > > at the same time.  Does this phrasing give any weaker a requirement
> than
> > > "must be visible on all machines at the same time", in practice?
> (There
> > > are, of course, various distributed-consensus protocols that can do
> > > this, as could a scenario where all NFS servers are connected to a
> > > common file store backend.)
> > >
> > > Section 11.10.9
> > >
> > >    When access is transferred between replicas, clients need to be
> > >    assured that the actions disallowed by holding these locks cannot
> > >
> > > To check my understanding: this "access is transferred" means *all*
> > > clients' access (not just one particular client)?  Otherwise I'm not
> > > sure how the destination would know to enforce the grace period.
> > >
> > > Section 11.11.1
> > >
> > > I think the last two paragraphs might be duplicating some things
> > > mentioned earlier in the section, but the repetition is probably not
> > > harmful.
> > >
> > > Section 11.12.1
> > >
> > >    Because of the absence of NFSV4ERR_LEASE_MOVED, it is possible for
> > >    file systems whose access path has not changed to be successfully
> > >
> > > It might be worth phrasing this as "SEQ4_STATUS_LEASE_MOVED is not an
> > > error condition".
> > >
> > > Section 11.12.2
> > >
> > >    o  No action needs to be taken for such indications received by the
> > >       those performing migration discovery, since continuation of that
> > >       work will address the issue.
> > >
> > > nit: "by the those" is not right, but the proper fix eludes me, as this
> > > bullet point needs to be more specific somehow than the next one.
> > >
> > >    o  If the fs_status attribute indicates that the file system is a
> > >       migrated one (i.e. fss_absent is true and fss_type !=
> > >       STATUS4_REFERRAL) and thus that it is likely that the fetch of
> the
> > >       file system location attribute has cleared one the file systems
> > >       contributing to the lease-migrated indication.
> > >
> > > This looks like a sentence fragment -- it's of the form "If X, and thus
> > > Y." with no concluding clause.
> > >
> > > Section 11.12.4
> > >
> > >    Once the client has determined the initial migration status, and
> > >    determined that there was a shift to a new server, it needs to re-
> > >    establish its locking state, if possible.  To enable this to happen
> > >    without loss of the guarantees normally provided by locking, the
> > >    destination server needs to implement a per-fs grace period in all
> > >    cases in which lock state was lost, including those in which
> > >    Transparent State Migration was not implemented.
> > >
> > > Similarly to above, does this imply that the migration has to happen
> for
> > > all clients concurrently, as opposed to clients getting migrated in
> > > sequence?
> > >
> > > Section 11.3.1
> > >
> > >    In this case, destination server need have no knowledge of the locks
> > >
> > > nit: singular/plural mismatch "destination server"/"need"
> > >
> > > Section 11.13.3
> > >
> > >    o  Not responding with NFS4ERR_SEQ_MISORDERED for the initial
> request
> > >       on a slot within a transferred session, since the destination
> > >
> > > Does this then translate to "process as usual in the absence of
> > > migration"?  "Don't return error X" tells me what not to do, but
> doesn't
> > > really tell me what to do instead.
> > >
> > > Section 11.16.1
> > >
> > >    With the exception of the transport-flag field (at offset
> > >    FSLI4BX_TFLAGS with the fls_info array), all of this data applies to
> > >    the replica specified by the entry, rather that the specific network
> > >    path used to access it.
> > >
> > > Is it clear that this applies only to the fields defined by this
> > > specification (since, as mentioned later, future extensions must
> specify
> > > whether they apply to the replica or the entry)?
> > >
> > > Section 15.1.1.3
> > >
> > >    o  When NFS4ERR_DELAY is returned on an operation other than the
> > >       first within a request and there has been a non-idempotent
> > >       operation processed before the NFS4ERR_DELAY was returned, the
> > >       reissued request should avoid the non-idempotent operation.  The
> > >       request still must use a SEQUENCE operation with either a
> > >       different slot id or sequence value from the SEQUENCE in the
> > >       original request.  Because this is done, there is no way the
> > >       replier could avoid spuriously re-executing the non-idempotent
> > >       operation since the different SEQUENCE parameters prevent the
> > >       requester from recognizing that the non-idempotent operation is
> > >       being retried.
> > >
> > > I don't think that this is very clear about the counterfactual scenario
> > > in which the replier is trying to avoid spuriously re-executing the
> > > non-idempotent operation.  Is it supposed to be explaining why the
> > > client has to use a different slot or sequence value, because the
> > > replier would reexecute the non-idempotent operation otherwise?
> > >
> > > Section 18.35.3
> > >
> > > I a little bit wonder if we want to reaffirm that co_verifier remains
> > > fixed when the client is establishing multiple connections for trunking
> > > usage -- the "incarnation of the client" language here could make a
> > > reader wonder, though I think the discussion of its use elsewhere as
> > > relating to "client restart" is sufficiently clear.
> > >
> > >    The eia_clientowner field is composed of a co_verifier field and a
> > >    co_ownerid string.  As noted in s Section 2.4, the co_ownerid
> > >
> > > s/s //
> > >
> > > Section 18.51.4
> > >
> > >    o  When a server might become the destination for a file system
> being
> > >       migrated, inappropriate use of per-fs RECLAIM_COMPLETE is more
> > >       concerning.  In the case in which the file system designated is
> > >       not within a per-fs grace period, the per-fs RECLAIM_COMPLETE
> > >       SHOULD be ignored, with the negative consequences of accepting it
> > >       being limited, as in the case in which migration is not
> supported.
> > >       However, if the server encounters a file system undergoing
> > >       migration, the operation cannot be accepted as if it were a
> global
> > >       RECLAIM_COMPLETE without invalidating its intended use.
> > >
> > > This seems to be the only place where we acknowledge that the "misuse"
> > > in question was to "treat rca_one_fs of TRUE as if it was FALSE", which
> > > is probably not so great for clarity.
> > >
> > > Section 21
> > >
> > > Some other topics at least somewhat related to trunking and migration
> > > that we could potentially justify including in the current,
> > > limited-scope, update (as opposed to deferring for a full -bis)
> include:
> > >
> > > - clients that lie about reclaimed locks during a post-migration grace
> > >   period
> > > - how attacker capabilities compare by using a compromised server to
> > >   give bogus referrals/etc. as opposed to just giving bogus data/etc.
> > > - an attacker in the network trying to shift client traffic (in terms
> of
> > >   what endpoints/connections they use) to overload a server
> > > - how asynchronous replication can cause clients to repeat
> > >   non-idempotent actions
> > > - the potential for state skew and/or data loss if migration events
> > >   happen in close succession and the client "misses a notification"
> > > - cases where a filesystem moves and there's no longer anything running
> > >   at the old network endpoint to return NFS4ERR_MOVED
> > > - what can happen when non-idempotent requests are in a COMPOUND before
> > >   a request that gets NFS4ERR_MOVED
> > > - how bad it is if the client messes up at Transparent State Migration
> > >   discovery, most notably in the case when some lock state is lost
> > > - the interactions between cached replies and migration(-like) events,
> > >   though a lot of this is discussed in section 11.13.X and 15.1.1.3
> > >   already
> > >
> > > but I defer to the WG as to what to cover now vs. later.
> > >
> > > In light of the ongoing work on draft-ietf-nfsv4-rpc-tls, it might be
> > > reasonable to just talk about "integrity protection" as an abstract
> > > thing without the specific focus on RPCSEC_GSS's integrity protection
> > > (or authentication)
> > >
> > >       being returned.  These include cases in which the client is
> > >       directed a server under the control of an attacker, who might get
> > >
> > > nit: "directed to"
> > >
> > >    o  Despite the fact that it is a requirement that "implementations"
> > >       provide "support" for use of RPCSEC_GSS, it cannot be assumed
> that
> > >       use of RPCSEC_GSS is always available between any particular
> > >       client-server pair.
> > >
> > > side note: scare-quotes around "support" makes sense to me, but not
> > > around "implementations".
> > >
> > >    the destination.  Even when RPCSEC_GSS authentication is available
> on
> > >    the destination, the server might validly represent itself as the
> > >    server to which the client was erroneously directed.  Without a way
> > >
> > > Something about the wording here tickles me funny; at first I thought
> it
> > > was the "validly", but now I think it's "represent itself", perhaps
> > > because that phrasing can have connotations of "falsely represent".
> > > ("Valid" is fine -- the attack here is the misdirection, and the target
> > > of the misdirection doesn't have to misbehave at all for it to be a
> > > damaging attack.)  The best remedy I can come up with is a somewhat
> > > drastic change, and thus questionable: "Even when [...], the server
> > > might still properly authenticate as the server to which the client was
> > > erroneously directed."
> > >
> > >
> > > I'd also consider adding a third bullet point to the final list ("to
> > > summarize considerations regarding the use of RPCSEC_GSS"):
> > >
> > > % o The integrity protection afforded to results by RPCSEC_GSS protects
> > > %   only a given request/response transaction; RPCSEC_GSS does not
> > > %   protect the binding from one server to another as part of a
> referral
> > > %   or migration event.  The source server must be trusted to provide
> > > %   the correct information, based on whatever factors are available to
> > > %   the client.
> > >
> > > Section 22.1
> > >
> > > Thank you for thinking about how the IANA considerations should be
> > > presented in the post-update document.  (I think I've had to place at
> > > least two Discuss positions on bis documents that did not...)
> > >
> > > Section 23.2
> > >
> > > I'm not sure that all of the moves from Normative to Informative should
> > > stick; e.g., HMAC (which went from [11] to [59]) is needed for SSV
> > > calculation.  Hmm, actually, maybe that's the only one.
> > >
> > > Appendix B
> > >
> > > I have mixed feelings about whether to keep this content for the final
> > > RFC.  (Appendix A seems clearly useful; the specific details of the
> > > reorganization are less clear, as to some extent they can be deduced
> > > from the changes themselves.  But only to some extent...)
> > >
> > > Appendix B.1.2
> > >
> > >    o  The new Sections 11.8 and 11.9 have resulted in existing sections
> > >       wit these numbers to be renumbered.
> > >
> > > s/wit/with/
> > >
> > > Section B.2.1
> > >
> > >    The new treatment can be found in Section 18.35 below.  It is
> > >
> > > s/below/above/
> > >
> > >    intended to supersede the treatment in Section 18.35 of RFC5661
> [62].
> > >    Publishing a complete replacement for Section 18.35 allows the
> > >    corrected definition to be read as a whole, in place of the one in
> > >    RFC5661 [62].
> > >
> > > This seems like it was more appropriate in the scope of
> > > draft-ietf-nfsv4-mv1-msns-update but could be obsolete here.
> > >
> > > Section B.4
> > >
> > >    o  The discussion of trunking which appeared in Section 2.10.5 of
> > >       RFC5661 [62] needed to be revised, to more clearly explain the
> > >       multiple types of trunking supporting and how the client can be
> > >       made aware of the existing trunking configuration.  In addition
> > >       the last paragraph (exclusive of sub-sections) of that section,
> > >       dealing with server_owner changes, is literally true, it has been
> > >       a source of confusion.  [...]
> > >
> > > nit: the grammar here is weird; I think there's a missing "while" or
> > > similar.
> > >
> > >
> > >
>

---

From: David Noveck <[email-masked]>
Date: Thu, 2 Jan 2020 10:09:02 -0500

On Wed, Dec 18, 2019 at 3:32 AM Benjamin Kaduk via Datatracker <
[email-masked]> wrote:

> Responded to these on 12/20.

> small.

No harm, no foul.

> "Updated by:" is mutable), we should probably explicitly state that "the

I think we could update the last paragraph of Section 1.1 to be more
explicit about
this.  Perhaps it could read:

   Until the above work is done, there will not be a consistent set of
   documents providing a description of the NFSv4.1 protocol and any
   full description would involve documents updating other documents
   within the specification.   The updates applied by RFC 8434 [66] and
   RFC 8178 [63] to RFC5661 also apply to this specification, and
   will apply to any subsequent v4.1 specification until that work is done.

> system migration is something done at a per-file-system granularity, and
> that migrating a client at a time is not possible.

It might be possible but doing so is not a goal of this specfication.

I'm not sure how to address your concern.   I don't know why anyone would
assume that migrating entire clients is a goal of this specification.   As
far as
I can see, when the word "migration" is used it is always in connection with
migrating a file system.   Is there some specific place where you think
this
issue is likely to arise?

As was the case for
> consequences of getting it wrong seem severe enough that I wanted to

I'm not seeing any severe consequences.   Am I missing something?

>
> Does a client have any way to know in advance that two addresses will be
> session-trunkable other than the one listed in Section 11.1.1 that "when
> they are always session-trunkable"?

No.

> It seems like mostly we're defining

We could add an extension to provide easier access to this information, but
the goal
of this document is to clarify what is possible with the current protocol.

Section 1.1
> nit: do we expect all readers to know what is meant by "trunking" with

Good point.  perhaps it could be addressed by rewriting the material in the
first paragraph of  Section 1.1 to read as follows;.

   Two important features previously defined in minor version 0 but
   never fully addressed in minor version 1 are trunking, the use of
   multiple connections between a client and server potentially to
   different network addresses, and transparent state migration, which
   allows a file system to be transferred betwwen servers in a way that
   provides for the client to maintain its existing locking state accross
   the transfer.

   The revised description of the NFS version 4 minor version 1
   (NFSv4.1) protocol presented in this update is necessary to enable
   full use of these features with other multi-server namespace features
   This document is in the form of an updated description of the NFS 4.1
   protocol previously defined in RFC5661 [62].  RFC5661 is obsoleted by
   this document.  However, the update has a limited scope and is focused
   on enabling full use of trunkinng and transparent state migration.  The
   need for these changes is discussed in Appendix A.  Appendix B describes
   the specific changes made to arrive at the current text.

>    This limited scope update is applied to the main NFSv4.1 RFC with the
>

Will fix.

> the subjunctive verb tense for most of these points ("work would have to

I think that when these words were written, the wg had not fully embraced
the idea that rfc5661bis had to be done.   The difficulty/length of rfc7530
was
a focus of attention.   Since that time, it has become generally accepted
that
this work is unavoidable and I intend to switch to the future indicative in
the next
revision.

> draft-ietf-nfsv4-

rfc5661sesqui-msns.

Will fix.

>       document will provide a clear set of requirements for layout types
>       requirements.  Other layout types will have their own specfication

Other layout types are not specfically mentioned but it is clear that they
will exist
and that this document needs to make provision for them, as rfc5661 did.

Such documents refer normatively to rfc5661, but rfc5661 (and this document)
would only refer to them informatively..

There is some text in Section 22.3 about setting up a layout type registry
and it
refers (informatively) to the defining documents.

It may be that incorporation of RFC8434 in rfc5661bis will involve further
references
to layout type specifications but I expect that all such references would be
 informative.

>
>    o  Work would have to be done to address many erratas relevant to RFC
> This sentence is pretty long and hard to follow; maybe it could be split
> (E.g., "Existing implementations did not perform handling as delineated in
> RFC

I will clean this bullet up.  See below for a proposed replcement.

> happens; my understanding is that the errata reports will get cloned to

I understand that Magnus has already got that issue addressed.  I'll
discuss the appropriate text with him.

> [rant about "errata" vs. "erratum" elided]

This is annoying but there is no way we are going to get people to use
"erratum".   What I've tried to do in my propsed replacement text
is to refer to "errata report(s)", which is more accurate and allows
people who speak English to use English singulars and plurals, without
having to worry about Latin grammar.

Here's my proposed replacement for the troubled bullet:

   o  Work needs to be done to address many errata reports relevant to
      RFC 5661, other than errata report 2006 [60], which is addressed
      in this document.  Addressing of that report was not deferrable
      because of the interaction of the changes suggested there and the
      newly described handling of state and session migration.

      The errata reports that have been deferred and that will need to
      be addressed in a later document include reports currently
      assigned a range of statuses in the errata reporting system
      including reports marked Accepted and those marked Held Over
      because the change was too minor to address immediately.

      In addition, there is a set of other reports, including at least
      one in state Rejected, which will need to be addressed in a later
      document.  This will involve making changes to consensus decisions
      reflected in RFC 5661, in situations in which the working group has
      already decided that the treatment in RFC 5661 is incorrect, and
needs
      to be revised to reflect the working group's new consensus and ensure
      compatibility with existing implementations that do not follow the
      handling described in in RFC 5661.

      Note that it is expected that such all errata reports will remain
      relevant to implementers and the authors of an eventual
      rfc5661bis, despite the fact that this document, when approved,
      will obsolete RFC 5661 [62].

>    EXCHANGE_ID operation.  The purpose of the server scope is to allow a
>    group of servers to indicate to clients that a set of servers sharing
>    otherwise opaque identifiers.  Thus, the identifiers generated by two
> Is there more that we can say than "in some cases"?

Not really.  In general, when a server sends you an id, it comes with an
implied promise to recognize it when you present it subsequently to the
same server.

The fact that two servers have decided to co-operate in their Id assignment
does not change that.

The previous text
> implies a higher level of reliability than just "some cases", to me.

I think I need to change the text, perhaps by replacing "use compatible
values of otherwise
opaque identifiers" by "use distinct values of otherwise opaque identifiers
so that the two
servers never assign the same value to two distinct objects".

I anticipate the following replacement for the first two paragraphs of
Section 2.10.4:

   Servers each specify a server scope value in the form of an opaque
   string eir_server_scope returned as part of the results of an
   EXCHANGE_ID operation.  The purpose of the server scope is to allow a
   group of servers to indicate to clients that a set of servers sharing
   the same server scope value has arranged to use distinct values of
   opaque identifiers so that the two servers never assign the same
   value to two distinct object.  Thus, the identifiers generated by two
   servers within that set can be assumed compatible so that, in certain
   important cases, identifiers generated by one server in that set may
   be presented to another server of the same scope.

   The use of such compatible values does not imply that a value
   generated by one server will always be accepted by another.  In most
   cases, it will not.  However, a server will not accept a value
   generated by another inadvertently.  When it does accept it, it will
   be because it is recognized as valid and carrying the same meaning as
   on another server of the same scope.

As an illustration of the (limited) value of this information, consider the
case of client recovery from a server reboot.  The client has to reclaim
his locks using file handles returned by the previous server instance.  If
the server scopes are the same (they almost always are), the client is not
sure he will get his locks back (e.g. the file might have been deleted),
but he does know that, if the lock reclaim succeeds, it is for the same
file.  If the server scopes are not the same, he has no such assurance.

> ID, client ID, and state ID values.  For at least one of those I can see

That was involved but overall we wound up deciding to have the list reflect
actual
server practice, as opposed to rfc5661, which focused on what it thought
servers
should do.

> though perhaps we are claiming that it is so under-specified so as to be

It's not workable in practice.

> includes all three in RFC 5661.

They shouldn't be. Given the potential interaction with migration, will be
fixing
this document.

>       to reclaim or otherwise re-obtain its locks.  It may find that its

No actions needed.  I think the last sentence would be better as two
sentences:

It might find out that its file handle is now stale. However, if
NFS4ERR_STALE is not returned, the client can proceed to reclaim or
otherwise re-obtain its open locking state.

> nit: The grammar got wonky here; maybe s/claim/claiming/?

Will fix

>
>
>
> Section 11.1.1
>
>       In the case of NFS version 4.1 and later minor versions, the means
>       available to every client.  Two network addresses connected to the

I would prefer not using "server-trunkable before it is defined.   I
anticipate
rewriting this paragraph as follows:

      In the case of NFS version 4.1 and later minor versions, the means
      of trunking detection are as described in this document and are
      available to every client.  Two network addresses connected to the
      same server can always be used to together to access a particular
      server but cannot necessarily be used together to access a single
      session.  See below for definitions of the terms "server-
      trunkable" and "session-trunkable"

> significant relevance, in general,

I'd prefer not to keep walking a fine line if that can be avoided and
intend to
allow port specification as indicated in my response to your DISCUSS
points.
Given that I've done that I need to make clear that, in this case, I am
only
referring to differences in ports due to different connection types, i.e.
2049 vs 20049.

 I anticipate revising this paragraph to read:

   The combination of a server network address and a particular
   connection type to be used by a connection is referred to as a
   "server endpoint".  Although using different connection types may
   result in different ports being used, the use of different ports by
   multiple connections to the same network address in such cases is not
   the essence of the distinction between the two endpoints used.  This
   is in contrast to the case in which the explicit specification of
   port numbers within network addresses is used to allow a single
   server node to suppport multiple NFS servers.

> and we are frequently in the IETF
> a given network address.

That may be, that there is a lot of client code out there that assumes that
you
will find an NFS server at port 2049.

> (Consider, for example, a hypothetical virtual

DS's at different ports, although interesting, are not addressed by any of
this text.   Network addresses for DS's are specified in layouts, which do
use
XDR structures to define network addresses.  I assume port numbers are
appropriately dealt with.  If they aren't, then this would be an issue for
rfc566bis,
in the case of pNFS files and other documents in the case of other layout
types
(e.g. flex files)

Will fix.

> other instances.

I expect that such readers would have led exceptionally sheltered lives :-)

> The rest of the document ought to disabuse them of
> that notion, and yet...]
>

It is reasonable to give them more warning.

I intend to address this by adding the following to the paragraph.

      Whether such replicas can be used simultaneously is discussed in
      Section 11.11.1, while the level of co-ordination betwenn them
      (important when switching between them) is discussed in Sections
      11.11.2 through 11.11.8 below.

>       address, and an fs name, which designates the location of the file
>
> nit: s/IP an/an IP/.
>

Will fix

>       client may establish connections.  There may be multiple endpoints
>       provide a way of connecting to a single server.  The exact form of
>

I think "designate"would work.   Will use that in next revision.

>       corresponding location element.  File system location entries that

I think you can switch from."IP address" to "network address containing an
IP address"
or from "is an IP address" to "includes an IP address".  Will fix.

>    NFSv4.1.  This attribute only allows specification of the file system
> I think this was probably okay as "SHOULD make this attribute available"

I think "SHOULD" is right. Will fix.

>    Where a file system had been absent, specification of file system

I expect to change this to say "Where a file system is currently absent".

> implies that it is no longer absent, i.e., that it is now present or has

I did later use "never previously present" but will switch that to "not
previously present".

> mention of or otherwise differentiate/characterize "non-pure"
I'd prefer not to do that.

> that a server SHOULD or even MUST adhere to?

They are informal.   It woluld be nice to go farther but we can't as it
raises
compatibility isses to suddenly tell a sever that what it had been doing is
now illegal/strongly disfavored.

> nit(?): is this supposed to just be "Location" singular
>

Yes.
 Will fix

>    connection, allowing the TCP connection to be dropped and the session

I will add ", if the server supports that" to the sentence.

>    event, even though some of the servers involved might not be aware of

How about s/as a/to be a/ ?

> to check that we have either covered or disavowed all of them.

No disavowal, as far as I know.

Yes.

> replication" with the 5661 definition of "replica", but in the new model

> server's namespaces match up.

We've disavowed the description although not the reality described.

> Though perhaps the discussion about not
> all of the cartesian product of (addresses-for-server, local path) being

I think this is already discussed (in two places in the next revision).

> misconfiguration that seems of questionable merit.

Agree.   I think the treatment of misconfiguration is primarily in 2.10.5.
Not sure
what else to add.

> deletion from the list of network instances of the current system

Yes.   If there is a deletion, there need not be any MOVED error.
 However, if you
get a MOVED error, then there has to be a deletion of your address from the
list.

I anticipate replacing this bullet by the following:

   o  Deletions from the list of network addresses for the current file
      system instance do not need to be acted on immediately by ceasing
      use of existing access paths although new connections are not to
      be esablished on addresses that have been deleted.  However,
      clients can choose to act on such deletions by making preparations
      for an eventual shift in access which would become unavoidable as
      soon as the server indicates that a particular network access path
      is not usable to access the current file system, by returning
      NFS4ERR_MOVED

> Section 11.6
>
>    corresponding attribute is interrogated subsequently.  In the case of
>    boundaries have been crossed.  Similarly, when the owner attribute of
> I can see how the interrogation would be on a different server from file
> creation for "simple" replication scenarios, but I'm not sure I'm seeing

I suppose  non-replication cases cannot arise.

nit: s/securiy/security/

Will fix.

> principals come from an AD forest.

"Most often" describes the reality that exists.   The protocol allows
multiple domains, but the big obstacle to multi-domain support is the
on-disk file systems that support only 32-bit uids and gids, as well
as the vfs's which have the same restriction. :-(

> nit: I think there's a missing word here, maybe "and for each

Yes, but I needed to rewrite the bullet as follows:

   o  All servers recognize the same set of security principals.  For
      each principal, the same credential is required, independent of
      the server being accessed.  In addition, the group membership for
      each such prinicipal is to be the same, independent of the server
      accessed.

Will fix.

> point,

Not sure what you mean by that.  I think the requirement is to allow the
client
to be able to use AUTH_SYS, without the contortions that would be required
if
different fs's had the same uid's meaning different things.

while acknowledging that the "stringified owners or group" case
> is separate, but not having much luck.

My attempt to revise this area is below:

   Note that there is no requirement in general that the users
   corresponding to particular security principals have the same local
   representation on each server, even though it is most often the case
   that this is so.

   When AUTH_SYS is used, the following additional requirements must be
   met:

   o  Only a single NFSv4 domain can be supported through use of
      AUTH_SYS.

   o  The "local" representation of all owners and groups must be the
      same on all servers.  The word "local" is used here since that is
      the way that numeric user and group ids are described in
      Section 5.9.  However, when AUTH_SYS or stringified numeric owners
      or groups are used, these identifiers are not truly local, since
      they are known to the clients as well as the server.

   Similarly, when strigified numeric user and group ids are used, the
   "local" representation of all owners and groups must be the same on
   all servers, even when AUTH_SYS is not used.

Also, nit: "to the"

Fixed.

> Section 11.9
>
>    o  When use of a particular address is to cease and there is also one
> session-trunkable,

OK. Will fix.

> point, and what we mean by "network address").

Will need to address for specified ports, given that these are now allowed.

Part of this will be new text in the definition of "server endpoints":

   The combination of a server network address and a particular
   connection type to be used by a connection is referred to as a
   "server endpoint".  Although using different connection types may
   result in different ports being used, the use of different ports by
   multiple connections to the same network address in such cases is not
   the essence of the distinction between the two endpoints used.  This
   is in contrast to the case of port-specific endpoints, in which the
   explicit specification of port numbers within network addresses is
   used to allow a single server node to suppport multiple NFS servers.

Also need to to revise, as follows, the paragraph containing 'inherently".

   o  When use of a particular connection is to cease, as indicated by
      receiving NFS4ERR_MOVED when using that connection but that
      address is still indicated as accessible according to the
      appropriate file system location entries, it is likely that
      requests can be issued on a new connection of a different
      connection type, once that connection is established.  Since any
      two, non-port-specific server endpoints that share a network
      address are inherently session-trunkable, the client can use
      BIND_CONN_TO_SESSION to access the existing session using the new
      connection and proceed to access the file system using the new
      connection.

>
>    o  When there are no potential replacement addresses in use but there
>

I've explained that in some new text added before these bullets, as a new
second
paragraph of this section:

   The appropriate action depends on the set of replacement addresses
   (i.e. server endpoints which are server-trunkable with one previously
   being used) which are available for use.

>
>       are valid addresses session-trunkable with the one whose use is to
>       be discontinued, the client can use BIND_CONN_TO_SESSION to access
> session to enable continued access to the existing instance" sounds like
> thus the new session would be there.

That is not the intention.  Will need to clarify.

> But why make a new session when
> the old one is still good,

It isn't usable on the new connection.

> especially when we just said in the previous

Because we can't use it on the new connection, we have to create a
new session to access  the client.

Perhaps a forward reference down to Section 11.12.{4,5} for this and the
> next bullet point would help as well as rewording?

It rurns out these would add confusion since they deal with migration
situations
and deciding wheher transparent stte miugration has occurred in the switch
between
replicas.  In the cases we are dealing with, ther is only a  single
replicas/fs and no
migration..

Here is my proposed replacement text for the two bullets in question:

   o  When there are no potential replacement addresses in use but there
      are valid addresses session-trunkable with the one whose use is to
      be discontinued, the client can use BIND_CONN_TO_SESSION to access
      the existing session using the new address.  Although the target
      session will generally be accessible, there may be rare situations
      in which that session is no longer accessible, when an attempt is
      made tto bind the new conntectin to it.  In this case, the client
      can create a new session to enable continued access to the
      existing instance and provide for use of existing filehandles,
      stateids, and client ids while providing continuity of locking
      state.

   o  When there is no potential replacement address in use and there
      are no valid addresses session-trunkable with the one whose use is
      to be discontinued, other server-trunkable addresses may be used
      to provide continued access.  Although use of CREATE_SESSION is
      available to provide continued access to the existing instance,
      servers have the option of providing continued access to the
      existing session through the new network access path in a fashion
      similar to that provided by session migration (see Section 11.12).
      To take advantage of this possibility, clients can perform an
      initial BIND_CONN_TO_SESSION, as in the previous case, and use
      CREATE_SESSION only if that fails.

>    in the handling of unstably written data.  When this is the case, and
>

Here's a prpoed replcment for this pargraph:

   In a file system transition, the two file systems might be
   cooperating in the handling of unstably written data.  Clients can
   ditermine if this is the case, by seeing if the two file systems
   belong to the same write-verifier class.  When this is the case,
   write verifiers returned from one system may be compared to those
   returned by the other and superfluous writes avoided.

> Section 11.10.7
>
>    In a file system transition, the two file systems might be consistent
>    in their handling of READDIR cookies and verifiers.  When this is the

>    the other and READDIR operations started on one server may be validly

No. They are intended to indicate that the client might or might not use
the capability

Here is proposed replacement text for the paragraph:

   In a file system transition, the two file systems might be consistent
   in their handling of READDIR cookies and verifiers.  Clients can
   determine if this is the case, by seeing if the two file systems
   belong to the same readdit class.  When this is the case, readdir
   class, READDIR cookies and verifiers from one system will be
   recognized by the other and READDIR operations started on one server
   can be validly continued on the other, simply by presenting the
   cookie and verifier returned

> This seems quite wishy-washy for a standards-track protocol!  We give no

I agree but feel that fixing this situation gets us beyond rfc5661sesqui or
even rfc5661bis,
but it would be possible to extend the fs_location_info attribute to
accommodate
this sort of information.  This would take the form of an extension to
NFsv4.2.

>
>                          The protocol does define three special cases of

The troublesome problem is that this terminology (and the associated wishy-
washiness has been used for multiple minor versions. That makes it hard
to change now.

>       fs_locations_info), they must designate the same data.  Where file
>       return of the modifying requester or the visibility of that change
> returns from A first, then the change still has to be visible on B and C
> at the same time.  Does this phrasing give any weaker a requirement than
> "must be visible on all machines at the same time", in practice?  (There

Actually it leads to a stronger requirement and I'm not sure that's right.
 The
current text would not allow A to store  the new data in non-volatile
memory,
but delay propagation until one of  A,B,C is askedf or the data.   Will
switch to
use your suggetion.

I anticipate using the following text for this bullet:

   o  When multiple replicas exist and are used simultaneously by a
      client (see the FSLIB4_CLSIMUL definition within
      fs_locations_info), they must designate the same data.  Where file
      systems are writable, a change made on one instance must be
      visible on all instances at the same time, regardless of whether
      the interrogated instance is the one on which the modification was
      done.  This allows a client to use these replicas simultaneously
      without any special adaptation to the fact that there are multiple
      replicas, beyond adapting to the fact that locks obtained on one
      replica are maintained separately (i.e. under a different client
      ID).  In this case, locks (whether share reservations or byte-
      range locks) and delegations obtained on one replica are
      immediately reflected on all replicas, in the sense that access
      from all other servers is prevented regardless of the replica
      used.  However, because the servers are not required to treat two
      associated client IDs as representing the same client, it is best
      to access each file using only a single client ID.

> sure how the destination would know to enforce the grace period.

It doesn't imply that.   Even if all clients of the source fs were
transferred, there will be clients of the dstination fs who have not been
transferred>

As far as grace periods go, co-operating servers could transfer locks
transparently or limit a grace period to a single transferred client

The following anticipated text for Section 11.11.9 may be helpful:

   When access is transferred between replicas, clients need to be
   assured that the actions disallowed by holding these locks cannot
   have occurred during the transition.  This can be ensured by the
   methods below.  Unless at least one of these is implemented, clients
   will not be assured of continuity of lock possession across a
   migration event.

   o  Providing the client an opportunity to re-obtain his locks via a
      per-fs grace period on the destination server, denying all clients
      using the destination filesysten the opportunity to obtain new
      locks that conflict which those held by the transferred client as
      long as that client has not completed its per-fs grace period.
      Because the lock reclaim mechanism was originally defined to
      support server reboot, it implicitly assumes that file handles
      will, upon reclaim, will be the same as those at open.  In the
      case of migration, this requires that source and destination
      servers use the same filehandles, as evidenced by using the same
      server scope (see Section 2.10.4) or by showing this agreement
      using fs_locations_info (see Section 11.11.2 above).

      Note that such a grace period can be implemented without
      interfering with the ability of non-transferred clients to obtain
      new locks while it is going on.  As long as the destination server
      is aware of the transferred locks, it can distinguish requests to
      obtain new locks that contrast with existing locks from those that
      do not, allowing it to treat such client requests without
      reference to the ongoing grace period.

Will do.

> bullet point needs to be more specific somehow than the next one.

I expect to use the following replacement:

   o  No action needs to be taken for such indications received by any
      threads performing migration discovery, since continuation of that
      work will address the issue.

>       STATUS4_REFERRAL) and thus that it is likely that the fetch of the

It is.  Will fix by using the following replacement:

   o  If the fs_status attribute indicates that the file system is a
      migrated one (i.e. fss_absent is true and fss_type !=
      STATUS4_REFERRAL) then a migrated file system has been found.  In
      this situation, it is likely that the fetch of the file system
      location attribute has cleared one the file systems contributing
      to the lease-migrated indication.

> Similarly to above, does this imply that the migration has to happen for

No, it doesn't.  The following replacement text should help make this clear:

   Once the client has determined the initial migration status, and
   determined that there was a shift to a new server, it needs to re-
   establish its locking state, if possible.  To enable this to happen
   without loss of the guarantees normally provided by locking, the
   destination server needs to implement a per-fs grace period in all
   cases in which lock state was lost, including those in which
   Transparent State Migration was not implemented.  Each client for
   which there was a shift of locking state to the new server will have
   the length of the grace period to reclaim its locks, from the time
   its locks were transferred.

> nit: singular/plural mismatch "destination server"/"need"

"needs have" would not work as "need" was being used as an (undeclinble)
auxiliary verb.

Went with "does not need any knowledge".

Also clarified stuff about grace period with the following replacement text:

   In this case, destination server does not need any knowledge of the
   locks held on the source server, but relies on the clients to
   accurately report (via reclaim operations) the locks previously held,
   not allowing new locks to be granted on migrated file system until
   the grace period expires.  Note that the diallowing of new locks
   applies to all clients accessing the file system, while grace period
   expiration occurs for each migrated client independently.

> Section 11.13.3
>
>    o  Not responding with NFS4ERR_SEQ_MISORDERED for the initial request
> migration"?  "Don't return error X" tells me what not to do, but doesn't

Intend to use the following replacement text:

   o  Not responding with NFS4ERR_SEQ_MISORDERED for the initial request
      on a slot within a transferred session, since the destination
      server cannot be aware of requests made by the client after the
      server handoff but before the client became aware of the shift.
      In cases in which NFS4ERR_SEQ_MISORDERED would normally have been
      reported, the request is to be processed normally, as a new
      request.

> specification (since, as mentioned later, future extensions must specify

Intend to use the following replacement text:

   With the exception of the transport-flag field (at offset
   FSLI4BX_TFLAGS with the fls_info array), all of this data defuined in
   this specification applies to the replica specified by the entry,
   rather that the specific network path used to access it.  The
   classification of data in extensions to this data is discussed
   below

>       requester from recognizing that the non-idempotent operation us
>       being retried.
>

Expect to use the following replacement text:

   o  When NFS4ERR_DELAY is returned on an operation other than the
      first within a request and there has been a non-idempotent
      operation processed before the NFS4ERR_DELAY was returned,
      reissuing the request as is normally done would incorrectly cause
      the re-execution of the non-idempotent operation.

      To avoid this, the reissued request should avoid the non-
      idempotent operation.  The request still must use a SEQUENCE
      operation with either a different slot id or sequence value from
      the SEQUENCE in the original request.  Because this is done, there
      is no way the replier could avoid spuriously re-executing the non-
      idempotent operation since the different SEQUENCE parameters
      prevent the requester from recognizing that the non-idempotent
      operation is being retried.

> relating to "client restart" is sufficiently clear.

This should be made clearer but the clarification needs to be done multiple
places.

Possible replacement text for eighth non-code paragraph of section 2.4:

   The first field, co_verifier, is a client incarnation verifier,
   allowing the server to distingish successive incarnations (e.g.
   reboots) of the same client.  The server will start the process of
   canceling the client's leased state if co_verifier is different than
   what the server has previously recorded for the identified client (as
   specified in the co_ownerid field).

Likely replacement text for the seventh paragraph of this section:

   The eia_clientowner field is composed of a co_verifier field and a
   co_ownerid string.  As noted in Section 2.4, the co_ownerid describes
   the client, and the co_verifier is the incarnation of the client.  An
   EXCHANGE_ID sent with a new incarnation of the client will lead to
   the server removing lock state of the old incarnation.  Whereas an
   EXCHANGE_ID sent with the current incarnation and co_ownerid will
   result in an error, an update of the client ID's properties,
   depending on the arguments to EXCHANGE_ID, or the return of
   information about the existing client_id as might happen when this
   opration is done to the same seerver using different network
   addresses as part of creating trunked connections.

Will fix.

> Section 18.51.4
>
>    o  When a server might become the destination for a file system being
>       being limited, as in the case in which migration is not supported.
>       migration, the operation cannot be accepted as if it were a global

Addressed in the following revised pargraph in SEction 18.51.4:

   Because previous descriptions of RECLAIM_COMPLETE were not
   sufficiently explicit about the circumstances in which use of
   RECLAIM_COMPLETE with rca_one_fs set to TRUE was appropriate, there
   have been cases which it has been misused by clients who have issued
   RECLAIM_COMPLETE with rca_one_fs set to TRUE when it should have not
   been.  There have also been cases in which servers have, in various
   ways, not responded to such misuse as described above, either
   ignoring the rca_one_fs setting (treating the operation as a global
   RECLAIM_COMPLETE) or ignoring the entire operation.

> limited-scope, update (as opposed to deferring for a full -bis) include:
>

Some of these are related to multi-server namespace but not related to
security, as far as I can see.

> - clients that lie about reclaimed locks during a post-migration grace

Will address in a number of places:

First of all, I inted to add a new paragraph to Section 21, to be placed as
the
sixth non-bulleted paragraph and to read as follows:

   Security consideration for lock reclaim differ between the state
   reclaim done after server failure (discussed in Section 8.4.2.1.1 and
   the per-fs state reclaim done in support of migration/replication
   (discussed in Section 11.11.9.1).

Next is a new proposed new section to appear as Section 11.11.9.1:

11.11.9.1.  Security Consideration Related to Reclaiming Lock State
            after File System Transitions

   Although it is possible for a client reclaiming state to misrepresent
   its state, in the same fashion as described in Section 8.4.2.1.1,
   most implementations providing for such reclamation in the case of
   file system transitions will have the ability to detect such
   misreprsentations.  this limits the ability of unauthenicatd clients
   to execute denial-of-service attacks in these cirsumstances.
   Nevertheless, the rules stated in Section 8.4.2.1.1, regarding
   prinipal verification for reclaim requests, apply in this situation
   as well.

   Typically,implementations support file system transitions will have
   extensive information about the locks to be transferred.  This is
   because:

   o  Since failure is not involved, there is no need store to locking
      information in persistent storage.

   o  There is no need, as there is in the failure case, to update
      multiple repositories containg locking state to keep them in sync.
      Instead, there is a one-time communication of locking state from
      the source to the destination server.

   o  Providing this information avoids potential interference with
      existing clients using the destination file system, by denying
      them the ability to obtain new locks during the grace period.

   When such detailed locking infornation, not necessarily including the
   associated stateid,s is available,

   o  It is possible to detect reclaim requests that attempt to reclsim
      locks that did not exist before the transfer, rejecting them with
      NFS4ERR_RECLAIM_BAD (Section 15.1.9.4).

   o  It is possible when dealing with non-reclaim requests, to
      determine whether they conflict with existing locks, eliminating
      the need to return NFS4ERR_GRACE ((Section 15.1.9.2) on non-
      reclaim requests.

   It is possible for implementations of grace periods in connection
   with file system transitions not to have detailed locking information
   available at the destination server, in which case the security
   situation is exactly as described in Section 8.4.2.1.1.

I think I should also draw your attention to a revised Section 15.1.9.
 These
includes some revisions originally done for
draft-ietf-nfsv4-rfc5661-msns-update,
which somehow got dropped as a few that turned up as necessary in writing
11.11.9.1:

15.1.9.  Reclaim Errors

   These errors relate to the process of reclaiming locks after a server
   restart.

15.1.9.1.  NFS4ERR_COMPLETE_ALREADY (Error Code 10054)

   The client previously sent a successful RECLAIM_COMPLETE operation
   specifying the same scope, whether that scope is global or for the
   same file system in the case of a per-fs RECLAIM_COMPLETE.  An
   additional RECLAIM_COMPLETE operation is not necessary and results in
   this error.

15.1.9.2.  NFS4ERR_GRACE (Error Code 10013)

   This error is returned when the server was in its recovery or grace
   period. with regard to the file system object for which the lock was
   requested resulting in a situation in which a non-reclaim locking
   request could not be granted.  This can occur because either

   o  The server does not have sufficiuent information about locks that
      might be poentially reclaimed to determine whether the lock could
      validly be granted.

   o  The request is made by a client responsible for reclaiming its
      locks that has not yet done the appropriate RECLAIM_COMPLETE
      operation, allowing it to proceed to obtain new locks.

   It should be noted that, in the case of a per-fs grace period, there
   may be clients, i.e. those currently using the destination file
   system who might be unaware of the circumstances resulting in the
   intiation of the grace period.  Such clients need to periodically
   retry the request until the grace period is over, just as other
   clients do.

15.1.9.3.  NFS4ERR_NO_GRACE (Error Code 10033)

   A reclaim of client state was attempted in circumstances in which the
   server cannot guarantee that conflicting state has not been provided
   to another client.  This occurs if there is no active grace period
   appliying to the file system object for which the request was made,if
   the client making the request has no current role in reclaining
   locks, or because previous operations have created a situation in
   which the server is not able to determine that a reclaim-interfering
   edge condition does not exist.

15.1.9.4.  NFS4ERR_RECLAIM_BAD (Error Code 10034)

   The server has determined that a reclaim attempted by the client is
   not valid, i.e. the lock specified as being reclaimed could not
   possibly have existed before the server restart or file system
   migration event.  A server is not obliged to make this determination
   and will typically rely on the client to only reclaim locks that the
   client was granted prior to restart.  However, when a server does
   have reliable information to enable it make this determination, this
   error indicates that the reclaim has been rejected as invalid.  This
   is as opposed to the error NFS4ERR_RECLAIM_CONFLICT (see
   Section 15.1.9.5) where the server can only determine that there has
   been an invalid reclaim, but cannot determine which request is
   invalid.

15.1.9.5.  NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)

   The reclaim attempted by the client has encountered a conflict and
   cannot be satisfied.  This potentially indicates a misbehaving
   client, although not necessarily the one receiving the error.  The
   misbehavior might be on the part of the client that established the
   lock with which this client conflicted.  See also Section 15.1.9.4
   for the related error, NFS4ERR_RECLAIM_BAD

>   give bogus referrals/etc. as opposed to just giving bogus data/etc.

Will address. See the paragraphs to be added to the end of Section 21.

> - an attacker in the network trying to shift client traffic (in terms of

Will address. See the paragraphs to be added to the end of Section 21.

> - how asynchronous replication can cause clients to repeat

Not sure what you are referring to.

>   happen in close succession and the client "misses a notification"

Is there a specfic problem that needs to be addressed?

>   at the old network endpoint to return NFS4ERR_MOVED

This seems to me just a recognition that sometimes system fail.  Not sure
specifically what to address.

>   a request that gets NFS4ERR_MOVED

Intend to address in Section 15.1.2.4:

   The file system that contains the current filehandle object is not
   present at the server, or is not accessible using the network
   addressed.  It may have been made accessible on a different ser of
   network addresses, relocated or migrated to another server, or it may
   have never been present.  The client may obtain the new file system
   location by obtaining the "fs_locations" or "fs_locations_info"
   attribute for the current filehandle.  For further discussion, refer
   to Section 11.3.

   As with the case of NFS4ERR_DELAY, it is possible that one or more
   non-idempotent operation may have been successfully executed within a
   COMPOUND before NFS4ERR_MOVED is returned.  Because of this, once the
   new location is determined, the original request which received the
   NFS4ERR_MOVED should not be re-executed in full.  Instead, a new
   COMPOUND, with any successfully executed non-idempotent operation
   removed should be executed.  This new request should have different
   slot id or sequence in those cases in which the same session is used
   for the new request (i.e. transparent session migration or an
   endpoint transition to a new address session-trunkable with the
   original one).

>   discovery, most notably in the case when some lock state is lost

Propose to address this by adding the following paragraph to the end of
Section 11.13.2:

   Lease discovery needs to be provided as described above, in order to
   ensure that migrations are discovered soon enough to ensure that
   leases moved to new servers are discovred in time to make sure that
   leases are renewed early enough to avoid lease expiration, leading to
   loss of locking state.  While the consequences of such loss can be
   ameliorated through implementations of courtesy locks, servers are
   under no obligation to do, and a conflicting lock request may means
   that a lock is revoked unexpectedly.  Clients should be aware of this
   possibility.

>   though a lot of this is discussed in section 11.13.X and 15.1.1.3

Will address any specfics that you feel aren't adequately addressed.

> thing without the specific focus on RPCSEC_GSS's integrity protection
I was initially leery of this, but when I looked at the text, I was able to
avoid referring to RPCSEC_GSS in most cases in which integrity was
mentioned:-).  The same does not seem posible for authentication :-(

Fixed.

>       provide "support" for use of RPCSEC_GSS, it cannot be assumed that

Fixed.

>
>    the destination.  Even when RPCSEC_GSS authentication is available on
> Something about the wording here tickles me funny; at first I thought it

I think it's a good remedy and I intend to adopt it.

> summarize considerations regarding the use of RPCSEC_GSS"):

Actually it is to summarise considertions regarding its use in fetching
location information

> %   only a given request/response transaction;

True but that is in the nature of fetching location information.

> RPCSEC_GSS does not
> %   protect the binding from one server to another as part of a referral

These are both situations for which RPCSEC_GSS has no solution, but neither
is there another one.   It is probably best to just say that without
reference
to integrity protection.

I have added new paragraphs after these bullets that may address some of
the
issues you were concerned about.

   Even if such requests are not interfered with in flight, it is
   possible for a compromised server to direct the client to use
   inappropriate servers, such as those under the control of the
   attacker.  It is not clear that being directed to such servers
   represents a greater threat to the client than the damage that could
   be done by the comprromised server itself.  However, it is possible
   that some sorts of transient server compromises might be taken
   advantage of to direct a client to a server capable of doing greater
   damage over a longer time.  One useful step to guard against this
   possibility is to issue requests to fetch location data using
   RPCSEC_GSS, even if no mapping to an RPCSEC_GSS principal is
   available.  In this case, RPCSEC_GSS would not be used, as it
   typically is, to identify the client principal to the server, but
   rather to make sure (via RPCSEC_GSS mutual authentication) that the
   server being contacted is the one intended.

   Similar considrations apply if the threat to be avoided is the
   direction of client traffic to inappropriate (i.e. poorly performing)
   servers.  In both cases, there is no reason for the information
   returned to depend on the identity of the client principal requesting
   it, while the validity of the server information, which has the
   capability to affect all client principals, is of considerable
   importance.

> calculation.  Hmm, actually, maybe that's the only one.

Moved it back to being Normative.

>    intended to supersede the treatment in Section 18.35 of RFC5661 [62].

 Anticipate using the following replacement text:

  o  The discussion of trunking which appeared in Section 2.10.5 of
      RFC5661 [62] needed to be revised, to more clearly explain the
      multiple types of trunking supporting and how the client can be
      made aware of the existing trunking configuration.  In addition,
      while the last paragraph (exclusive of sub-sections) of that
      section, dealing with server_owner changes, is literally true, it
      has been a source of confusion.  Since the existing paragraph can
      be read as suggesting that such changes be dealt with non-
      disruptively, the issue needs to be clarified in the revised
      section, which appears in Section 2.10.5.

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 13 Jan 2020 10:27:16 -0800

Hi David,

Lots of good stuff trimmed (and thank you for them!).  Just one comment for
this thread...

[...]
> I'll think about doing that.

I'm not so much concerned about how the server constructs the namespace (as
the current practice leaves that entirely up to the server), but rather how
the client, or some human associated with the client, learns what the
server did.  That is, having a fancy namespace with extra semantics doesn't
do you any good if no one else knows what those semantics are :)
I was just hoping that this document could be more explicit about "this
protocol doesn't give you a way to communicate the extra semantics from
server to client, but you probably want to do it somehow".  I don't mind
giving more details about the types of semantics a server might encode in
the namespace as well, but that's not the direction I was trying to push
in.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 13 Jan 2020 14:54:11 -0800

Hi David,

Trimming lots of good stuff here as well...

>    will apply to any subsequent v4.1 specification until that work is done.

Sounds good.

> issue is likely to arise?

I think I garbled my point; my apologies.
To give a semi-concrete example, suppose I have clients A and B that are
accessing filesystem F on server X, and filesystem F is also available on
server Y.  If X decides that it needs to migrate access to F away from X
(e.g., for maintenance), then the "file system migration event" involves
telling both A and B to look to Y for access to F, at basically the same
time.  If X tries to tell only A but not B to access F via Y but lets B
continue to access F at X, then I think there can be some subtle
consistency issues.

In some sense, this is easy to consider as a dichotomy between "migration
is for server maintenance" vs. "migration is for load balancing".  Assuming
I understand correctly (not a trivial assumption!), there was never any
intent to use these mechanisms for load balancing, and if we can explicitly
disclaim such usage, then we don't have to try to reason through any
potential subtle consistency issues.

[...]
>    provides for the client to maintain its existing locking state accross

Maybe "the simultaneous use of multiple connections"?
nit: s/betwwen/between/

>    the specific changes made to arrive at the current text.

This looks good, thanks.

[...]
> having to worry about Latin grammar.

That's what I try to do as well :)

>       including reports marked Accepted and those marked Held Over

nit: it's "Hold For Document Update"

>       Note that it is expected that such all errata reports will remain

nit: s/such all/all such/

>       will obsolete RFC 5661 [62].

This looks really good!

>    generated by another inadvertently.  When it does accept it, it will

nit: I think it flows better to put "invertently" as "will not
inadvertently accept".

> file.  If the server scopes are not the same, he has no such assurance.

Thanks, the new text (and explanation here) is very clear about what's
going on.

[...]
> How about s/as a/to be a/ ?

That works if you drop the earlier "that", for "the client might consider
the transition to the new replica to be a migration event".

[...]
>    all servers, even when AUTH_SYS is not used.

I really like this rewriting; thank you for undertaking it.
I think that what I was trying to say here is roughly that we need
scare-quotes for "local" because of things like AUTH_SYS (or stringified
user/group ids) that involve sending local representations over the
network.  So your rewrite did in fact address my concern, even though I
didn't manage to say it very well the first time :)

[...]
>       made tto bind the new conntectin to it.  In this case, the client

nits: s/tto/to/, s/conntectin/connection/

>       stateids, and client ids while providing continuity of locking

Just to check: this sounds like even in the case where the client creates
a new session, the filehandle, stateid, clientid, and locking state
(values) are in effect "transparently preserved" by the server, so the
client has no need to do any reclamation of locking state.  I think that's
what's intended, but holler if I'm wrong about that.

>    belong to the same readdit class.  When this is the case, readdir

nit: s/readdit/readdirt

>    cookie and verifier returned

Ah, this formulation (for both write-verifier and readdir) is very helpful.

[...]
>    FSLI4BX_TFLAGS with the fls_info array), all of this data defuined in

nit: s/defuined/defined/

[...]
>    addresses as part of creating trunked connections.

I think I get the general sense of what is going on here (i.e., the last
sentence) but am still uncertain on the specifics.  Namely, "most of the
time" (TM), sending EXCHANGE_ID with current incarnation/ownerid will be an
error, since it's a client bug to try to register the same way twice in a
row.  However, some times we might have to do that in order to update
properties of the client or get some new information that a server has
associated to a given client ID.  I *think* (but am not sure) that the
error case is exactly when the (same-incarnation/ownerid) EXCHANGE_ID is
done to the same *server and address* as the original EXCHANGE_ID, and that
the "update properties or get new information back" case is exactly when
the EXCHANGE_ID is done to a different server/address combination.

If I'm right about that, then I'd suggest:

%    the server removing lock state of the old incarnation.  Whereas an
%    EXCHANGE_ID sent with the current incarnation and co_ownerid will
%    result in an error when sent to a given server at a given address for
%    a second time, it is not an error to send EXCHANGE_ID with current
%    incarnation and co_ownerid to a different server (e.g., as part of a
%    migration event).  In such cases, the EXCHANGE_ID can allow for an
%    update of the client ID's properties, depending on the arguments to
%    EXCHANGE_ID, or the return of (potentially updated) information about
%    the existing client_id, as might happen when this opration is done to
%    the same server using different network addresses as part of creating
%    trunked connections.

> security, as far as I can see.

It does look like it; in some sense I was going through a brainstorming
exercise to make this list, and appreciate the sanity checks.  (To be
clear, I am not insisting that any of them get covered in specifically the
sesqui update, just mentioning topics for potential consideration.)
>    misreprsentations.  this limits the ability of unauthenicatd clients

typos: "misrepresentations", "This", "unauthenticated"

>    to execute denial-of-service attacks in these cirsumstances.

"circumstances"

>    Typically,implementations support file system transitions will have

nits: space after comma, and "that" for "that support"

>    associated stateid,s is available,

nits: "information", s/stateid,s/stateids,/
>    o  It is possible to detect reclaim requests that attempt to reclsim

nit: s/reclsim/reclaim/

> which somehow got dropped as a few that turned up as necessary in writing

(no full stop)

>    system who might be unaware of the circumstances resulting in the

nit: comma after "file system"

>    for the related error, NFS4ERR_RECLAIM_BAD

Thanks for remembering to fetch these updates from the full bis WIP!

> Not sure what you are referring to.

I don't have something fully fleshed out here, but it's in the general
space when there are multiple replicas that get updates at (varying)
delays from the underlying write.  A contrived situation would be if you
have a pool of worker machines that use NFS for state management (I know, a
pretty questionable idea), and try to do compare-and-set on a state file.
If one worker tries to assert that it owns the state but other NFS replicas
see delayed updates, additional worker machines could also try to claim the
state and perform whatever operation the state file is controlling.

Basically, the point here is that if you as an NFS consumer are using NFS
with relaxed replication semantics, you have to think through how your
workflow will behave in the presence of such relaxed updates.  Which ought
to be obvious, when I say it like that, but perhaps is not always actually
obvious.

> Is there a specfic problem that needs to be addressed?

I don't have a concrete scenario that's specific to NFS, no; this is just a
generic possibility for any scheme that involves discrete updates (e.g.,
file-modifying RPCs) and the potential for asynchronous replication.

> specifically what to address.

Okay.

>    for the new request (i.e. transparent session migration or an

nit: comma after "i.e.".

>    loss of locking state.  While the consequences of such loss can be

nit: the double "are discovered {soon enough,in time}" is a little awkward
of a construction; how about "Lease discovery needs to be provided as
described above, in order to ensure that migrations are discovered soon
enough that leases moved to new servers can successfully be renewed before
they expire, avoiding loss of locking state"?

>    under no obligation to do, and a conflicting lock request may means

nit: s/means/mean/

> Will address any specfics that you feel aren't adequately addressed.

I don't remember any particular specifics, so we should probably just let
this go for now.

> mentioned:-).  The same does not seem posible for authentication :-(

We'll take the easy wins and try to not fret too much about the other
stuff.

> is there another one.   It is probably best to just say that without

True.

>    capability to affect all client principals, is of considerable

These do address some of the issues I mentioned; thank you.  I do have a
couple further comments:

- I'm not sure what "no mapping to an RPCSEC_GSS principal is available"
  means (but maybe that's just because I've not read the RPCSEC_GSS RFCs
  recently enough)
- w.r.t. "there is no reason for the information returned to depnd on the
  identity of the client principal", I could perhaps imagine some setup
  that uses information from a corporate contacts database to determine the
  current office/location of a given user and provide a referral to a
  spatially-local replica.  So "no reason" may be too absolute (but I don't
  have a proposed alternative and don't object to using this text).

[...]
>       multiple types of trunking supporting and how the client can be

nit: just "trunking support" (not "-ing")?

>       section, which appears in Section 2.10.5.

Thanks again for going through my giant pile of comments; I hope that you
think the improvements to the document are worth the time spent.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 13 Jan 2020 14:59:50 -0800

Replying to myself: I wrote this in multiple sittings and somehow thought
the point I wanted to revise was in the other fork of the thread.

> potential subtle consistency issues.

Some of your later replies in the "comment-section" thread make me think
that my understanding, quoted above, is incorrect.  That is, that it's okay
for X to tell A to migrate to Y for filesystem F while X continues to serve
F to B.  In particular, the updated text about server reclaim behavior (and
knowing what specific clientids might be reclaiming) seems to address the
main "subtle consistency issues" that I can think of right now.

Sorry about that.

-Ben

---

From: David Noveck <[email-masked]>
Date: Sat, 18 Jan 2020 08:30:18 -0500

On Mon, Jan 13, 2020 at 5:54 PM Benjamin Kaduk <[email-masked]> wrote:

> time.

This clarifies things for me.   When you were speaking of "migrating a
client"
i ssumed you  worried anout consistency of fs's F, G,, H for a particular
client.
Now it appears the issue is consistency among clients A, B, c, all
accessing a
common F.

If X tries to tell only A but not B to access F via Y but lets B
> continue to access F at X, then I think there can be some subtle

Or worse, some decidely unsubltle ones :-(

> is for server maintenance" vs. "migration is for load balancing".

That categorization helps.

> Assuming
> intent to use these mechanisms for load balancing,

Well "Never" covers a lot.    There are cases which you do want to do
load balancing.   For example, if you are dealing with multiple network
access path to the same replica, there is no issue with the load balancing
approach.   In the case of multiple replicas where data consistency applies
between them, then you might lod balance but it is the server's
resposibility to
provide the consistency, meaning that he needs to be warned of the
possibility
of issues that might arise if clients   modifying the same dara are placed
on
different replicas. In the case in which you don't guarantee data
consistency among
replicas, you might as well say about doing load balacing that "there be
dragons".

and if we can explicitly
> potential subtle consistency issues.

I think we can disclaim the really problematic part.  I think the new text
will be needed in the migration section.  Issues with replication are
different and do not involve any server choice.

I anticipate revising section 11.5.5 to read as follows:

   When a file system is present and becomes inacessible using the
   current access path, the NFSv4.1 protocol provides a means by which
   clients can be given the opportunity to have continued access to
   their data.  This may involve use of a different access path to the
   existing replica or by providing a path to a different replica.  The
   new access path or the location of the new replica is specified by a
   file system location attribute.  The ensuing migration of acceess
   includes the ability to retain locks across the transition.
   Depending on circumstances, this can involve:

   o  The continued use of the existing clientid when accessing the
      current replica using a new access path.

   o  Use of lock reclaim, taking advantage of a per-fs grace period.

   o  Use of Tranparent State Migration.

   Typically, a client will be accessing the file system in question,
   get an NFS4ERR_MOVED error, and then use a file system location
   attribute to determine new the access path for the data.  When
   fs_locations_info is used, additional information will be available
   that will define the nature of the client's handling of the
   transition to a new server.

   In most instances clients will choose to migrate all clients using a
   particular file system to a successor replica at the same time to
   avoid cases in which different clients are updating diufferent
   replicas.  However migration of individual client can be helpful in
   providing load balancing, as long as the replicas in question are
   such that they represent the same data as described in
   Section 11.11.8.

   o  In the case in which there is no transition between replicas
      (i.e., only a change in access path), there are no special
      difficulties in using of this mechanism to effect load balancing.

   o  In the case in which the two replicas are sufficiently co-
      ordinated as to allow coherent simultaneous access to both by a
      single client, there is, in general, no obstacle to use of
      migration of particular clients to effect load balancing.
      Generally, such simultaneous use involves co-operation between
      servers to ennsure that locks granted on two co-ordinated replica
      cannot conflict and can remain effective when transferred to a
      common replica.

   o  In the case in which a large set of clients are accessing a file
      system in a read-only fashion, in can be helpful to migrate all
      clients with writable access simultaneously, while using load
      balancing on the set of read-only copies, as long as the rules
      appearing in Section 11.11.8, designed to prevent data reversion
      are adhered to.

   In other cases, the client might not have sufficient guarantees of
   data similarity/coherence to function prperly (e.g. the data in the
   two replicas is similar but not identical), and the possibility that
   different clients are updating different replicas can exacerbate the
   difficulties, making use of load balancing in such situations a
   perilous enterprise.

   The protocol does not specify how the file system will be moved
   between servers or how updates to multiple replicas will be co-
   ordinated.  It is anticipated that a number of different server-to-
   server co-ordination mechanisms might be used with the choice left to
   the server implementer.  The NFSv4.1 protocol specifies the method
   used to communicate the migration event between client and server.

   The new location may be, in the case of various forms of server
   clustering, another server providing access to the same physical file
   system.  The client's responsibilities in dealing with this
   transition will depend on whether a switch between replicas has
   occurred and the means the server has chosen to provide continuity of
   locking state.  These issues will be discussed in detail below.

   Although a single successor location is typical, multiple locations
   may be provided.  When multiple locations are provided, the client
   will typically use the first one provided.  If that is inaccessible
   for some reason, later ones can be used.  In such cases the client
   might consider the transition to the new replica to be a migration
   event, even though some of the servers involved might not be aware of
   the use of the server which was inaccessible.  In such a case, a
   client might lose access to locking state as a result of the access
   transfer.

   When an alternate location is designated as the target for migration,
   it must designate the same data (with metadata being the same to the
   degree indicated by the fs_locations_info attribute).  Where file
   systems are writable, a change made on the original file system must
   be visible on all migration targets.  Where a file system is not
   writable but represents a read-only copy (possibly periodically
   updated) of a writable file system, similar requirements apply to the
   propagation of updates.  Any change visible in the original file
   system must already be effected on all migration targets, to avoid
   any possibility that a client, in effecting a transition to the
   migration target, will see any reversion in file system state.

> > I'm not seeing any severe consequences.   Am I missing something?

This is clearer now. I think we can avoid any severe consequences.

> Maybe "the simultaneous use of multiple connections"?

Will add.

Fixed.

> >    the specific changes made to arrive at the current text.

:-)

> Fixed

> nit: I think it flows better to put "invertently" as "will not

OK.  Fixed.

> Did that.

Fixed.

> what's intended, but holler if I'm wrong about that.
Ok.
I'll holler that *you're right about that.*

Fixed.

 Fixed.

> >    addresses as part of creating trunked connections.

Not sure what error that error text was  referring to above.   Think it
added to the
confusion..

> row.

No it isn't.   This is case 2 on page 508, " Non-Update on Existing Client
ID".
Given  retries and possible communication difficulties, it is just too hard
to
make this case an error.

However, some times we might have to do that in order to update
> %    the same server using different network addresses as part of creating

I think I have to revise the paragraph above to be clearer.   I anticipate
replacing the seventh paragraph of section 18.35.3 by the following
replacement:

   The eia_clientowner field is composed of a co_verifier field and a
   co_ownerid string.  As noted in Section 2.4, the co_ownerid
   identifies the client, and the co_verifier specfies a particular
   incarnation of that client.  An EXCHANGE_ID sent with a new
   incarnation of the client will lead to the server removing lock state
   of the old incarnation.  On the other hand, an EXCHANGE_ID sent with
   the current incarnation and co_ownerid will, when it does not result
   in an unrelated error, porentially update an existing client ID's
   properties, or simply return of information about the existing
   client_id.  That latter would happen when this operation is done to
   the same server using different network addresses as part of creating
   trunked connections.

Fixed.

Fixed.

Fixed.

Fixed.

 Fixed.

> nit: comma after "file system"

This phrase is now within parentheses.

> file-modifying RPCs) and the potential for asynchronous replication.

I think that the necessary discussion can be folded into some clarification
of replication discussed in
another thread.

Fixed.

> they expire, avoiding loss of locking state"?

Went with the following:

        Lease discovery needs to be provided as described above, in order to
        ensure that migrations are discovered soon enough to enable
        leases moved to new servers to be  appropriately renewed in order to
        avoid lease expiration, leading to loss of locking state.

Fixed.

>   means (but maybe that's just because I've not read the RPCSEC_GSS RFCs

This is partly because " even if no mapping to an RPCSEC_GSS principal is
available"
is misleading.   It would have been better to say "even if no mapping  is
to an
RPCSEC_GSS principal is available for the user currently obtaining the
information".
The issue is not within RPCSEC_GSS itself but relates to how it is used by
NFSv4.
Servers are required to support RPCSEG_GSS but to use it, you need to
translate
a uid to an RPCSEC_GSS principal,   Where that mapping is not available, as
it often
is not, you cannot use RPCSEC_GSS for that user.
>    available

Fixed.

> think the improvements to the document are worth the time spent.

I do.

---

From: David Noveck <[email-masked]>
Date: Sat, 18 Jan 2020 08:35:57 -0500

> the namespace as well, but that's not the direction I was trying to push

OK.  I think I can address your concerns but it needs to be pointed out
that
in many cases of the  "extra" semanntics we are talking about, the semantics
are clear.  This is true of trunking, referrals and migration.   In the
case of
"replication", we do have a lot of existing uncertainty and I will try to
put in the
kind of text you suggest.

This and related issues have been mentioned in a number of sub-threads and
I intend to
consolidate text addressing  them here.   I'm going to propose that we
address all
these issues by some additions to Section 11.5.4.  Specfically I propose
revsing the
the second paragrapgh and adding a new tthird pragraph so that the last two
paragraphs
read as follows;

   In the event that the occurrence of server failures, communications
   problems, or other difficulties make continued access to the current
   file system impossible or otherwise impractical, the client can use
   the alternate locations as a way to get continued access to its data.

   The alternate locations may be physical replicas of the (typically
   read-only) file system data supplemneted by possible asynchronous
   propagation of updates.  Alternatively, they may provide for the use
   of various forms of server clustering in which multiple servers
   provide alternate ways of accessing the same physical file system.
   How the difference between replicas affects file system transitions
   can be represented within the fs_locations and fs_locations_info
   attributes and how the client deals with file system transition
   issues will be discussed in detail in later sections.

   Although the location attributes provide some information about the
   nature of the inter-replica transition, many aspects of the semantics
   of posible asynchronous updates are not currently described by the
   protocol, making it necessary that clients using replication to
   switch among replicas undergoing change familiarize themselves with
   the semantics of the update approach used.  Because of this lack of
   specificity, many applications may find use of migration more
   appropiate, since, in that case, the server, when effecting the
   transition, has established a point in time such that all updates
   made before that can propagated to the new replica as part of the
   migration event.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 21 Jan 2020 19:16:50 -0800

Not attempting to trim anything, so sparsely inline...

>    In most instances clients will choose to migrate all clients using a

(I assume s/clients/servers/ (just the first time))

>    the same server using different network addresses as part of creating

Ah, I think I get it now.  Thanks.

> of replication discussed in

Sure.

>         avoid lease expiration, leading to loss of locking state.

I think maybe we should leave any further tweaks to the RFC Editor; my only
concern is whether it can be misread as saying that the renewal leads to
loss of locking state (which is, admittedly, a nonsensical interpretation).

> is not, you cannot use RPCSEC_GSS for that user.

Ah, that was enough to jostle the neurons into place (and no change to the
text needed).

Thanks again,

Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 21 Jan 2020 19:25:35 -0800

>    read-only) file system data supplemneted by possible asynchronous

(nit: "supplemented", though I made the same typo myself just now)

>    made before that can propagated to the new replica as part of the

Thanks, this covers the key point I was concerned about.

-Ben

---

From: David Noveck <[email-masked]>
Date: Wed, 22 Jan 2020 04:10:50 -0500

I should be able to submit a -04 soon, including any last-minute nits.

On Tue, Jan 21, 2020, 10:16 PM Benjamin Kaduk <[email-masked]> wrote:

> > > > I can see, when the word "migration" is used it is always in
> connection
> > > with
> > > > Good point.  perhaps it could be addressed by rewriting the material
> in
> > > the
> > > > As an illustration of the (limited) value of this information,
> consider
> > > the
> > > > his locks using file handles returned by the previous server
> instance.
> > > If
> > > > the server scopes are the same (they almost always are), the client
> is
> > > not
> clustered
> describes
> > > > First of all, I inted to add a new paragraph to Section 21, to be
> placed
> > > as
> > > > the
> > > > I was initially leery of this, but when I looked at the text, I was
> able
> > > to